DMVR and BDOF-based inter-prediction method and apparatus

Abstract

A method and apparatus for increasing video coding efficiency is provided. [Solution] The video encoding method according to this document includes deriving L0 and L1 motion vectors for a current block, applying DMVR to the current block based on DMVR flag information to derive refined L0 and L1 motion vectors based on the L0 and L1 motion vectors, deriving an L0 prediction sample for the current block based on the refined L0 motion vector and an L1 prediction sample for the current block based on the refined L1 motion vector, applying BDOF to the current block based on BDOF flag information to derive a prediction sample for the current block based on the L0 and L1 prediction samples, and generating a reconstructed sample for the current block based on the prediction sample, wherein the DMVR flag information is derived based on predetermined DMVR application conditions, and the BDOF flag information is derived based on predetermined BDOF application conditions.

Description

[Technical Field]

[0001] This document relates to video coding techniques, and more particularly to methods and apparatus for performing inter prediction based on Decoder-side Motion Vector Refinement (DMVR) and / or Bi-Directional Optical Flow (BDOF). [Background technology]

[0002] Recently, demand for high-resolution, high-quality images / videos, such as 4K or 8K or higher UHD (Ultra High Definition) images / videos, is increasing in various fields. As the resolution and quality of image / video data increases, the amount of information or bits to be transmitted increases relatively compared to conventional image / video data. Therefore, when transmitting image data using a medium such as a conventional wired or wireless broadband line or storing image / video data using a conventional storage medium, the transmission cost and storage cost increase.

[0003] In addition, interest and demand for immersive media such as VR (Virtual Reality), AR (Artificial Reality) content, and holograms has been increasing recently, and the broadcast of images / videos with different visual characteristics from real images, such as game images, is increasing.

[0004] Therefore, a highly efficient image / video compression technology is required to effectively compress, transfer, store, and play back high-resolution, high-quality image / video information having the above-mentioned various characteristics. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical problem of this document is to provide a method and apparatus for increasing video coding efficiency.

[0006] Another technical problem of this document is to provide an efficient inter-prediction method and apparatus.

[0007] Yet another technical problem of this document is to provide a method and apparatus for performing inter prediction based on Decoder-side Motion Vector Refinement (DMVR).

[0008] Yet another technical problem of this document is to provide a method and apparatus for performing inter prediction based on BDOF (Bi-Directional Optical Flow).

[0009] Yet another technical problem of this document is to provide a method and apparatus for improving prediction performance by providing conditions for determining whether to apply DMVR and / or conditions for determining whether to apply BDOF to improve video coding efficiency. [Means for solving the problem]

[0010] According to one embodiment of the present document, there is provided a video decoding method performed by a decoding device, the method including the steps of: deriving an L0 motion vector and an L1 motion vector of a current block; applying a decoder-side motion vector refinement (DMVR) to the current block based on DMVR flag information indicating whether DMVR is applied to the current block, and deriving a refined L0 motion vector and a refined L1 motion vector based on the L0 motion vector; deriving an L0 prediction sample of the current block based on the refined L0 motion vector and an L1 prediction sample of the current block based on the refined L1 motion vector; and performing bidirectional optical flow (Bi-Directional Optical Flow) on the current block. the method includes a step of applying BDOF to a current block based on BDOF flag information indicating whether or not to apply BDOF (Boundary Decomposition Flow; BDOF), and deriving a predicted sample for the current block based on an L0 predicted sample and an L1 predicted sample; and a step of generating a reconstructed sample for the current block based on the predicted sample, wherein the DMVR flag information is derived (induced) based on a preset DMVR application condition, and the BDOF flag information is derived based on the preset BDOF application condition.

[0011] According to another embodiment of the present document, there is provided a video encoding method performed by an encoding device, the method including the steps of: deriving an L0 motion vector and an L1 motion vector of a current block; applying a decoder-side motion vector refinement (DMVR) to the current block based on DMVR flag information indicating whether DMVR is applied to the current block, and deriving a refined L0 motion vector and a refined L1 motion vector based on the L0 motion vector; deriving an L0 prediction sample of the current block based on the refined L0 motion vector and an L1 prediction sample of the current block based on the refined L1 motion vector; and performing bidirectional optical flow (Bi-Directional Optical Flow) on the current block. the step of applying BDOF to a current block based on BDOF flag information indicating whether or not to apply BDOF (Boundary Decomposition Flow; BDOF) to the current block, and deriving a prediction sample for the current block based on an L0 prediction sample and an L1 prediction sample; the step of deriving a residual sample based on the prediction sample; and the step of encoding video information having information on the residual sample, wherein the DMVR flag information is derived based on a preset DMVR application condition, and the BDOF flag information is derived based on the preset BDOF application condition. [Effects of the Invention]

[0012] According to this document, it is possible to improve the overall image / video compression efficiency.

[0013] According to this document, efficient inter-prediction can reduce computational complexity and improve overall coding efficiency.

[0014] According to this document, when applying DMVR and / or BDOF to refine motion information during the motion compensation process, various application conditions are proposed, thereby improving efficiency in terms of complexity and performance. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 illustrates a schematic diagram of an example of a video / image coding system to which embodiments of the present document can be applied. [Figure 2] 1 is a diagram illustrating the configuration of a video / image encoding device that can be applied to embodiments of the present document; [Figure 3] FIG. 1 is a diagram illustrating the configuration of a video / image decoding device that can be applied to embodiments of the present document. [Figure 4] FIG. 1 illustrates an example of a video / image encoding method based on inter prediction. [Figure 5] FIG. 1 is a diagram illustrating an example of a schematic diagram of an inter prediction unit in an encoding device. [Figure 6] FIG. 1 illustrates an example of a video / picture decoding method based on inter prediction. [Figure 7] FIG. 1 is a diagram illustrating an example of a schematic diagram of an inter-prediction unit in a decoding device. [Figure 8] FIG. 1 illustrates an embodiment of a process for performing Decoder-side Motion Vector Refinement (DMVR) in true paired prediction. [Figure 9] FIG. 1 is a diagram illustrating an embodiment of a process for performing decoder-side motion vector refinement (DMVR) using the sum of absolute differences (SAD). [Figure 10] FIG. 10 is a diagram showing an example of a method for checking the application conditions of DMVR and BDOF and performing a decoding process. [Figure 11]FIG. 10 is a diagram showing another example of a method for checking the application conditions of DMVR and BDOF and performing a decoding process. [Figure 12] FIG. 10 is a diagram showing another example of a method for checking the application conditions of DMVR and BDOF and performing a decoding process. [Figure 13] FIG. 1 is a diagram illustrating the concept of BDOF. [Figure 14] 1 is a flow chart that schematically illustrates an encoding method that can be performed by an encoding device according to an embodiment of the present document; [Figure 15] 1 is a flow chart that schematically illustrates a decoding method that can be performed by a decoding device according to an embodiment of the present document; [Figure 16] FIG. 1 illustrates an example of a content streaming system to which embodiments disclosed herein may be applied. [Figure 17] FIG. 1 is a diagram illustrating an example of a service system including a digital device. [Figure 18] FIG. 1 is a configuration block diagram shown to explain an embodiment of a digital device. [Figure 19] FIG. 10 is a configuration block diagram shown to explain another embodiment of the digital device. [Figure 20] 1 illustrates a schematic diagram of an architecture for providing 3D image / video services in which embodiment(s) of the present document may be utilized. [Figure 21] 1 is a diagram illustrating the configuration of a first digital device to which the present method can be applied; [Figure 22] FIG. 2 is a diagram illustrating the configuration of a second digital device to which the method can be applied. DETAILED DESCRIPTION OF THE INVENTION

[0016] Because this document may be modified in various ways and may have various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit this document to the specific embodiment. Common terms used in this document are used to describe specific embodiments and are not intended to limit the technical ideas of this document. The singular includes the plural unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the presence or possibility of addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0017] Meanwhile, the components in the drawings described in this document are shown independently for the convenience of describing their distinct characteristic functions, and do not imply that the components are realized by separate hardware or software. For example, two or more of the components may be combined to form a single component, or a single component may be divided into multiple components. Embodiments in which the components are integrated and / or separated are also within the scope of this document as long as they do not deviate from the essence of this document.

[0018] In this document, the terms " / " and "," are interpreted as "and / or." For example, "A / B" is parsed as "A and / or B," and "A, B" is parsed as "A and / or B." Additionally, "A / B / C" means "at least one of A, B, and / or C." Also, "A, B, C" means "at least one of A, B, and / or C." (In this document, the terms " / " and "," should be interpreted to indicate "and / or." For instance, the expression "A / B" may mean "A and / or B." Further, "A, B" may mean "A and / or B." Further, "A / B / C" may mean "at least one of A, B, and / or C." Also, "A / B / C" may mean "at least one of A, B, and / or C.")

[0019] Additionally, in this document, the term "or" should be interpreted to mean "and / or." For example, "A or B" can mean 1) only "A," 2) only "B," or 3) "A and B." Further, in the document, the term "or" should be interpreted to indicate "and / or." For instance, the expression "A or B" may comprise 1) only A, 2) only B, and / or 3) both A and B. In other words, the term "or" in this document should be interpreted to indicate "additionally or alternatively."

[0020] This document relates to video / image coding. For example, the methods / embodiments disclosed in this document can be applied to methods disclosed in the Versatile Video Coding (VVC) standard, the Essential Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd Generation of Audio Video coding Standard (AVS2) or next-generation video / image coding standards (e.g., H.267 or H.268).

[0021] This document presents various embodiments relating to video / image coding, which may be implemented in combination with one another unless otherwise stated.

[0022] In this document, video can refer to a collection of a series of images over time. A picture generally refers to a unit that represents one image at a specific time period, and a slice / tile is a unit that constitutes part of a picture in coding. A slice / tile can contain one or more Coding Tree Units (CTUs). A picture can be composed of one or more slices / tiles. A picture can be composed of one or more tile groups. A tile group can contain one or more tiles. A brick can represent a rectangular region of CTU rows within a tile in a picture. A tile can be partitioned into multiple bricks, and each brick can consist of one or more CTU rows within the tile. A tile that is not partitioned into multiple bricks may also be referred to as a brick.A brick scan can represent a specific sequential ordering of CTUs partitioning a picture, where the CTUs can be ordered consecutively in a CTU raster scan within a brick, the bricks within a tile can be ordered consecutively in a raster scan of the bricks of the tile, and the tiles within a picture can be ordered consecutively in a raster scan of the tiles of the picture. A tile is a rectangular region of CTUs within a particular tile column and a particular tile row in a picture. The tile column is a rectangular region of CTUs having a height equal to the height of the picture and a width specified by syntax elements in the picture parameter set.The tile row is a rectangular region of CTUs having a height specified by syntax elements in the picture parameter set and a width equal to the height of the picture. A tile scan can represent a specific sequential ordering of CTUs partitioning a picture, where the CTUs can be ordered consecutively in a CTU raster scan in a tile, and tiles in a picture can be ordered consecutively in a raster scan of the tiles of the picture. A slice includes an integer number of bricks of a picture that may be exclusively contained in a single NAL unit.A slice may consist of either a number of complete tiles or only a consecutive sequence of complete bricks of one tile. In this document, the terms tile group and slice may be used interchangeably. For example, in this document, tile group / tile group header may be referred to as slice / slice header.

[0023] A pixel or a pel can refer to the smallest unit constituting a picture (or image). The term "sample" can also be used as a term corresponding to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only a pixel / pixel value of a luma component, or can represent only a pixel / pixel value of a chroma component. Alternatively, a sample can refer to a pixel value in the spatial domain, or, when such a pixel value is transformed into the frequency domain, can refer to a transform coefficient in the frequency domain.

[0024] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. The term unit may be used interchangeably with terms such as block or area. In general, an MxN block may include a set (or array) of samples or transform coefficients consisting of M columns and N rows.

[0025] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Hereinafter, the same components in the drawings will be designated by the same reference numerals, and duplicated descriptions of the same components will be omitted.

[0026] FIG. 1 illustrates schematically an example of a video / image coding system to which embodiments of this document can be applied.

[0027] 1, a video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image information or data to the receiving device in the form of a file or streaming data via a digital storage medium or a network.

[0028] The source device may include a video source, an encoding device, and a transfer unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may be called a video / video encoding device, and the decoding device may be called a video / video decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may also include a display unit, which may be configured as a separate device or an external component.

[0029] A video source can acquire video / video through a video / video capture, synthesis, or generation process. A video source can include a video / video capture device and / or a video / video generation device. A video / video capture device can include, for example, one or more cameras, a video / video archive containing previously captured video / video, etc. A video / video generation device can include, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate video / video. For example, a virtual (pseudonym) video / video can be generated via a computer, etc., in which case the video / video capture process can be replaced by a process in which the associated data is generated.

[0030] An encoding device can encode input video / images. The encoding device can perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bit stream.

[0031] The transmitting unit can transmit the encoded video / image information or data output in the form of a bitstream to a receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit can include elements for generating a media file in a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0032] The decoding device can decode the video / image by performing a series of steps such as inverse quantization, inverse transform, and prediction that correspond to the operations of the encoding device.

[0033] The renderer can render the decoded video / image, which can then be displayed via the display unit.

[0034] 2 is a diagram for schematically illustrating the configuration of a video / image encoding device that can be applied to an embodiment of this document. Hereinafter, the term "video encoding device" may include an image encoding device.

[0035] Referring to FIG. 2, the encoding apparatus 200 may include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter predictor 221 and an intra predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may further include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. The above-described image dividing unit 210, predicting unit 220, residual processing unit 230, entropy encoding unit 240, adding unit 250, and filtering unit 260 may be configured by one or more hardware components (e.g., an encoder chipset or a processor) depending on the embodiment. Also, the memory 270 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The above hardware components may further include the memory 270 as an internal / external component.

[0036] The image division unit 210 may divide an input image (or picture, frame) input to the encoding device 200 into one or more processing units. For example, the processing units may be called coding units (CUs). In this case, the coding units may be recursively divided from coding tree units (CTUs) or largest coding units (LCUs) according to a quad-tree, binary-tree, and ternary-tree (QTBTTT) structure. For example, one coding unit may be divided into multiple coding units of deeper depths based on a quad-tree structure, a binary tree structure, and / or a ternary tree structure. In this case, for example, the quad-tree structure may be applied first, and then the binary tree structure and / or the ternary tree structure may be applied later. Alternatively, the binary tree structure may be applied first. The coding procedure according to this document may be performed based on the final coding unit that is not further divided. In this case, the largest coding unit may be directly used as the final coding unit based on coding efficiency according to video characteristics, or, if necessary, the coding unit may be recursively divided into coding units of lower depths, and a coding unit of an optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit.The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0037] The term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or pixel value, or can represent only a pixel / pixel value of a luma component, or can represent only a pixel / pixel value of a chroma component. A sample can also be used as a term that corresponds to a pixel or pel in one picture (or image).

[0038] The encoding apparatus 200 may subtract a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 221 or the intra prediction unit 222 from an input video signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as shown in the figure, a unit in the encoder 200 that subtracts a prediction signal (predicted block, prediction sample array) from an input video signal (original block, original sample array) may be referred to as a subtraction unit 231. The prediction unit may predict a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied for each current block or CU. As will be described later in the description of each prediction mode, the prediction unit may generate various information related to prediction, such as prediction mode information, and transmit the information to the entropy encoding unit 240. The prediction information may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0039] The intra prediction unit 222 can predict the current block by referring to samples in the current picture. The referenced samples can be located in the neighborhood of the current block or can be located far away, depending on the prediction mode. In intra prediction, prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC mode and planar mode. Directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the granularity of the prediction direction. However, this is for illustrative purposes only, and more or fewer directional prediction modes may be used depending on the settings. The intra prediction unit 222 can also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0040] The inter prediction unit 221 may derive a predicted block for a current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information regarding an inter prediction direction (such as L0 prediction, L1 prediction, or Bi prediction). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring blocks may be the same or different. The temporal surrounding blocks may be referred to as collocated reference blocks, collocated CUs (colCUs), etc., and a reference picture including the temporal surrounding blocks may be referred to as a collocated picture (colPic). For example, the inter predictor 221 may construct a motion information candidate list based on the surrounding blocks and generate information indicating which candidates are used to derive a motion vector and / or a reference picture index for the current block. Inter prediction may be performed based on various prediction modes. For example, in the case of a skip mode or a merge mode, the inter predictor 221 may use motion information of surrounding blocks as motion information for the current block. In the case of the skip mode, unlike the merge mode, a residual signal may not be transmitted.In the case of the Motion Vector Prediction (MVP) mode, the motion vector of the current block can be indicated by using the motion vector of the surrounding block as a motion vector predictor and signaling the motion vector difference.

[0041] The prediction unit 220 can generate a prediction signal based on various prediction methods, which will be described later. For example, the prediction unit can apply intra prediction or inter prediction for predicting a block, or can simultaneously apply intra prediction and inter prediction. This can be called combined inter and intra prediction (CIIP). The prediction unit can also use an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode can be used for content image / video coding, such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be similar to inter prediction in that it derives a reference block within the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document. The palette mode can be considered an example of intra coding or intra prediction. When the palette mode is applied, sample values ​​within a picture can be signaled based on information about a palette table and a palette index.

[0042] The prediction signal generated by the prediction unit (including the inter prediction unit 221 and / or the intra prediction unit 222) may be used to generate a reconstructed signal or a residual signal. The transform unit 232 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, GBT refers to a transform obtained from a graph representing inter-pixel relationship information. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. The transform process may be applied to pixel blocks having the same rectangular size, or to non-rectangular blocks of variable size.

[0043] The quantization unit 233 quantizes the transform coefficients and transmits them to the entropy encoding unit 240. The entropy encoding unit 240 encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. The information about the quantized transform coefficients may be referred to as residual information. The quantization unit 233 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit 240 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoding unit 240 may encode information required for video / image restoration (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / video information) may be transferred or stored in the form of a bitstream in Network Abstraction Layer (NAL) units. The video / video information may further include information on various parameter sets, such as an Adaptation Parameter Set (APS), a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), or a Video Parameter Set (VPS). The video / video information may also include general constraint information. Information and / or syntax elements transmitted / signaled from an encoding device to a decoding device in this document may be included in the video / video information. The video / video information may be encoded through the above-described encoding procedure and included in the bitstream.The bitstream can be transferred via a network or stored in a digital storage medium. Here, the network can include a broadcasting network and / or a communication network, and the digital storage medium can include various storage media such as a USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transfer unit (not shown) that transfers and / or a storage unit (not shown) that stores the signal output from the entropy encoding unit 240 can be configured as an internal / external element of the encoding device 200, or the transfer unit can be included in the entropy encoding unit 240.

[0044] The quantized transform coefficients output from the quantizer 233 may be used to generate a prediction signal. For example, a residual signal (residual block or residual samples) may be reconstructed by applying inverse quantization and inverse transform to the quantized transform coefficients via the inverse quantizer 234 and the inverse transformer 235. The adder 155 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter predictor 221 or the intra predictor 222. When there is no residual for the current block, such as when a skip mode is applied, a predicted block may be used as the reconstructed block. The adder 250 may be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next current block in the current picture, or may be used for inter prediction of the next picture after filtering, as described below.

[0045] Meanwhile, Luma Mapping with Chroma Scaling (LMCS) may be applied during picture encoding and / or restoration.

[0046] The filtering unit 260 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 260 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. The various filtering methods may include, for example, deblock filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc. The filtering unit 260 may generate various information related to filtering and transmit it to the entropy encoding unit 240, as will be described later in the description of each filtering method. The filtering information may be encoded in the entropy encoding unit 240 and output in the form of a bitstream.

[0047] The modified reconstructed picture transferred to the memory 270 can be used as a reference picture in the inter prediction unit 221. When inter prediction is applied through this, the encoding device can avoid prediction mismatch between the encoding device 100 and the decoding device, and can also improve coding efficiency.

[0048] The memory 270DPB can store the modified reconstructed picture for use as a reference picture in the inter predictor 221. The memory 270 can store motion information of blocks from which motion information in the current picture is derived (or encoded) and / or motion information of blocks in already reconstructed pictures. The stored motion information can be transmitted to the inter predictor 221 to be used as motion information of spatially or temporally surrounding blocks. The memory 270 can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra predictor 222.

[0049] FIG. 3 is a diagram illustrating the configuration of a video / image decoding device that can be applied to the embodiments of this document.

[0050] Referring to FIG. 3, the decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter predictor 331 and an intra predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 322. Depending on the embodiment, the entropy decoder 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 may be implemented as a single hardware component (e.g., a decoder chipset or a processor). The memory 360 may include a decoded picture buffer (DPB) or may be implemented as a digital storage medium. The above hardware components may further include a memory 360 as an internal / external component.

[0051] When a bitstream including video / image information is input, the decoding device 300 can reconstruct an image corresponding to the process by which the video / image information was processed in the encoding device of FIG. 2. For example, the decoding device 300 can derive units / blocks based on block division-related information acquired from the bitstream. The decoding device 300 can perform decoding using the processing units applied in the encoding device. Therefore, the processing unit for decoding can be, for example, a coding unit, and the coding unit can be divided from a coding tree unit or a maximal coding unit according to a quadtree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units can be derived from the coding unit. The reconstructed video signal decoded and output by the decoding device 300 can then be played back via a playback device.

[0052] The decoding device 300 may receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal may be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 may parse the bitstream to derive information (e.g., video / video information) necessary for video restoration (or picture restoration). The video / video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). The video / video information may also include general constraint information. The decoding device may decode pictures based on information on the parameter sets and / or the general constraint information. Signaled / received information and / or syntax elements, which will be described later in this document, may be decoded via the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 may decode information in a bitstream based on a coding method such as Exponential-Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for video reconstruction and quantized values ​​of transform coefficients related to the residual. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using information about the syntax element to be decoded, decoded information about neighboring and current blocks, or information about symbols / bins decoded in previous steps, predicts the occurrence probability of the bins according to the determined context model, and performs arithmetic decoding of the bins to generate symbols corresponding to the values ​​of each syntax element. After determining the context model, the CABAC entropy decoding method may update the context model using information about the decoded symbols / bins for the context model of the next symbol / bin.Among the information decoded by the entropy decoding unit 310, information related to prediction is provided to a prediction unit (inter prediction unit 332 and intra prediction unit 331), and residual values ​​entropy decoded by the entropy decoding unit 310, i.e., quantized transform coefficients and related parameter information, may be input to a residual processing unit 320. The residual processing unit 320 may derive a residual signal (residual block, residual sample, residual sample array). Furthermore, among the information decoded by the entropy decoding unit 310, information related to filtering may be provided to a filtering unit 350. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding device may be further configured as an internal / external element of the decoding device 300, or the receiving unit may be a component of the entropy decoding unit 310. Meanwhile, the decoding device according to this document may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit 310, and the sample decoder may include at least one of the inverse quantization unit 321, the inverse transform unit 322, the addition unit 340, the filtering unit 350, the memory 360, the inter prediction unit 332, and the intra prediction unit 331.

[0053] The inverse quantization unit 321 can inverse quantize the quantized transform coefficients and output transform coefficients. The inverse quantization unit 321 can rearrange the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit 321 can obtain transform coefficients by inverse quantizing the quantized transform coefficients using a quantization parameter (e.g., quantization step size information).

[0054] The inverse transform unit 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0055] The prediction unit may perform prediction on the current block and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoding unit 310, and may determine a specific intra / inter prediction mode.

[0056] The prediction unit 320 may generate a prediction signal based on various prediction methods, which will be described later. For example, the prediction unit may apply not only intra prediction or inter prediction for predicting a block, but also simultaneous intra prediction and inter prediction. This may be referred to as combined inter and intra prediction (CIIP). The prediction unit may also use an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content image / video coding, such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but may be similar to inter prediction in deriving a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described herein. The palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, information regarding a palette table and a palette index may be included in the video / picture information and signaled.

[0057] The intra prediction unit 331 can predict the current block by referring to samples in the current picture. The referenced samples can be located in the neighborhood of the current block or far away from it depending on the prediction mode. Prediction modes in intra prediction can include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 331 can also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0058] The inter prediction unit 332 may derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. To reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information regarding the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter prediction unit 332 may construct a motion information candidate list based on the neighboring blocks and derive a motion vector and / or a reference picture index for the current block based on received candidate selection information. Inter prediction may be performed based on various prediction modes, and the prediction information may include information indicating the inter prediction mode for the current block.

[0059] The adder 340 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the obtained residual signal to a prediction signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 332 and / or the intra prediction unit 331). When there is no residual for the current block, such as when a skip mode is applied, the predicted block may be used as the reconstructed block.

[0060] The adder 340 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of a next block to be processed in the current picture, may be output after filtering as described below, or may be used for inter prediction of a next picture.

[0061] Meanwhile, Luma Mapping with Chroma Scaling (LMCS) can be applied in the picture decoding process.

[0062] The filtering unit 350 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 350 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may transfer the modified reconstructed picture to the memory 360, specifically, to the DPB of the memory 360. The various filtering methods may include, for example, deblock filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc.

[0063] The (modified) reconstructed picture stored in the DPB of the memory 360 can be used as a reference picture in the inter prediction unit 332. The memory 360 can store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of a block in an already reconstructed picture. The stored motion information can be transmitted to the inter prediction unit 332 to be used as motion information of a spatially surrounding block or a temporally surrounding block. The memory 360 can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 331.

[0064] In this specification, the embodiments described for the filtering unit 260, inter prediction unit 221 and intra prediction unit 222 of the encoding device 100 can also be applied identically or correspondingly to the filtering unit 350, inter prediction unit 332 and intra prediction unit 331 of the decoding device 300, respectively.

[0065] As described above, prediction is performed to improve compression efficiency during video coding. This allows for the generation of a predicted block including predicted samples for a current block, which is a block to be coded. Here, the predicted block includes predicted samples in the spatial domain (or pixel domain). The predicted block is derived in the same way by both an encoding device and a decoding device. The encoding device can improve video coding efficiency by signaling to the decoding device information (residual information) regarding the residual between the original block and the predicted block, rather than the original sample values ​​of the original block themselves. The decoding device can derive a residual block including residual samples based on the residual information, combine the residual block with the predicted block, and generate a reconstructed block including reconstructed samples, thereby generating a reconstructed picture including the reconstructed block.

[0066] The residual information may be generated through a transform and quantization procedure. For example, an encoding device may derive a residual block between the original block and the predicted block, perform a transform procedure on residual samples (residual sample array) included in the residual block to derive transform coefficients, perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, and signal the related residual information to a decoding device (via a bitstream). Here, the residual information may include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding device may perform an inverse quantization / inverse transform procedure based on the residual information to derive residual samples (or a residual block). The decoding device may generate a reconstructed picture based on the predicted block and the residual block. The encoding device may also inverse quantize / inverse transform the quantized transform coefficients to derive a residual block for reference for inter-prediction of a subsequent picture, and generate a reconstructed picture based on the residual block.

[0067] Meanwhile, as described above, intra prediction or inter prediction can be applied to predict the current block. Hereinafter, a case where inter prediction is applied to the current block will be described.

[0068] A prediction unit (more specifically, an inter prediction unit) of an encoding / decoding device may perform inter prediction on a block-by-block basis to derive predicted samples. Inter prediction may refer to prediction derived in a manner dependent on data elements (e.g., sample values, motion information, etc.) of picture(s) other than the current picture. When inter prediction is applied to a current block, a predicted block (prediction sample array) for the current block may be derived based on a reference block (reference sample array) identified by a motion vector in a reference picture indicated by a reference picture index. In this case, to reduce the amount of motion information transmitted in the inter prediction mode, motion information may be predicted on a block, sub-block, or sample-by-block basis based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). When inter prediction is applied, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal surrounding block may be the same or different. The temporal surrounding block may be called a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal surrounding block may also be called a collocated picture (colPic). For example, a motion information candidate list may be constructed based on the surrounding blocks of the current block, and flags or index information indicating which candidates are selected (used) to derive the motion vector and / or reference picture index of the current block may be signaled.Inter prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the motion information of the current block may be the same as that of a selected neighboring block. In skip mode, unlike merge mode, a residual signal may not be transmitted. In motion vector prediction (MVP) mode, the motion vector of a selected neighboring block is used as a motion vector predictor, and a motion vector difference may be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0069] The motion information may include L0 motion information and / or L1 motion information depending on the inter prediction type (such as L0 prediction, L1 prediction, or Bi prediction). A motion vector in the L0 direction may be referred to as an L0 motion vector or MVL0, and a motion vector in the L1 direction may be referred to as an L1 motion vector or MVL1. Prediction based on an L0 motion vector may be referred to as L0 prediction, prediction based on an L1 motion vector may be referred to as L1 prediction, and prediction based on both an L0 motion vector and an L1 motion vector may be referred to as pairwise (Bi) prediction. Here, the L0 motion vector may represent a motion vector associated with a reference picture list L0 (L0), and the L1 motion vector may represent a motion vector associated with a reference picture list L1 (L1). The reference picture list L0 may include, as reference pictures, pictures that are earlier in output order than the current picture, and the reference picture list L1 may include pictures that are later in output order than the current picture. The earlier pictures may be referred to as forward (reference) pictures, and the later pictures may be referred to as backward (reference) pictures. The reference picture list L0 may further include, as reference pictures, pictures that are later in output order than the current picture. In this case, the earlier picture may be indexed first in the reference picture list L0, followed by the later picture. The reference picture list L1 may further include, as reference pictures, pictures that are earlier in output order than the current picture. In this case, the later picture may be indexed first in the reference picture list L1, followed by the earlier picture. Here, the output order may correspond to the order of the Picture Order Count (POC).

[0070] Information indicating whether the above-mentioned list0 (L0) prediction, list1 (L1) prediction, or paired prediction (bi-prediction) is used for the current block (current coding unit) may be signaled. This information may be referred to as motion prediction direction information, inter-prediction direction information, or inter-prediction indication information, and may be configured / encoded / signaled in the form of, for example, an inter_pred_idc syntax element. That is, the inter_pred_idc syntax element may indicate whether the above-mentioned list0 (L0) prediction, list1 (L1) prediction, or paired prediction (bi-prediction) is used for the current block (current coding unit). In this document, for convenience of explanation, the inter-prediction type (L0 prediction, L1 prediction, or BI prediction) indicated by the inter_pred_idc syntax element may be represented as a motion prediction direction. L0 prediction may be represented as pred_L0, L1 prediction as pred_L1, and paired prediction as pred_BI. For example, the prediction types shown in Table 1 below can be expressed depending on the value of the inter_pred_idc syntax element.

[0071] [Table 1]

[0072] In addition, various inter prediction modes may be used when applying inter prediction to the current block. For example, various modes may be used, such as merge mode, skip mode, motion vector prediction (MVP) mode, affine mode, and historical motion vector prediction (HMVP) mode. Decoder-side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, and bi-directional optical flow (BDOF) may also be used as additional modes. The affine mode may be referred to as an affine motion prediction mode. The MVP mode may also be referred to as an advanced motion vector prediction (AMVP) mode. In this document, motion information candidates derived according to some modes and / or some modes may be included as one of the motion information-related candidates of other modes.

[0073] Prediction mode information indicating the inter prediction mode of the current block may be signaled from the encoding device to the decoding device. In this case, the prediction mode information may be included in a bitstream and received by the decoding device. The prediction mode information may include index information indicating one of multiple candidate modes. Alternatively, the inter prediction mode may be indicated through hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags. For example, a skip flag may be signaled to indicate whether the skip mode is to be applied, and if the skip mode is not applied, a merge flag may be signaled to indicate whether the merge mode is to be applied, and if the merge mode is not applied, an MVP mode may be applied, or a flag for additional classification may be further signaled. The affine mode may be signaled as an independent mode or as a mode dependent on the merge mode or MVP mode. For example, the affine mode may include an affine merge mode and an affine MVP mode.

[0074] Furthermore, motion information of the current block can be used when applying inter prediction to the current block. The encoding device can derive optimal motion information for the current block through a motion estimation procedure. For example, the encoding device can use an original block in an original picture for the current block to search for a highly correlated similar reference block within a predetermined search range in the reference picture in fractional pixel units, thereby deriving motion information. Block similarity can be derived based on differences in phase-based sample values. For example, block similarity can be calculated based on the sum of absolute differences (SAD) between the current block (or the template of the current block) and the reference block (or the template of the reference block). In this case, motion information can be derived based on the reference block with the smallest SAD within the search space. The derived motion information can be signaled to the decoding device in various ways based on the inter prediction mode.

[0075] As described above, a predicted block for a current block may be derived based on motion information derived according to the inter prediction mode. The predicted block may include a predicted sample (prediction sample array) of the current block. If the motion vector (MV) of the current block points to a fractional sample unit, an interpolation procedure may be performed, thereby deriving a predicted sample of the current block based on a reference sample in a fractional sample unit within a reference picture. If affine inter prediction is applied to the current block, a predicted sample may be generated based on a sample / sub-block-based MV. If pairwise prediction is applied, a predicted sample derived through a weighted sum or weighted average (according to the phase) of a predicted sample derived based on L0 prediction (i.e., prediction using a reference picture in the reference picture list L0 and MVL0) and a predicted sample derived based on L1 prediction (i.e., prediction using a reference picture in the reference picture list L1 and MVL1) may be used as a predicted sample of the current block. When paired prediction is applied, if the reference picture used for L0 prediction and the reference picture used for L1 prediction are located in different temporal directions relative to the current picture (i.e., paired prediction and bidirectional prediction), this can be called true paired prediction.

[0076] As described above, reconstructed samples and reconstructed pictures can be generated based on the derived predicted samples, and then procedures such as in-loop filtering can be performed.

[0077] Fig. 4 illustrates an example of a video / image encoding method based on inter prediction, and Fig. 5 illustrates an example of a schematic diagram of an inter prediction unit in an encoding device. The inter prediction unit in the encoding device of Fig. 5 may be applied identically or correspondingly to the inter prediction unit 221 of the encoding device 200 of Fig. 2 described above.

[0078] 4 and 5, an encoding apparatus performs inter prediction on a current block (S400). The encoding apparatus derives an inter prediction mode and motion information for the current block and generates predicted samples for the current block. Here, the steps of determining the inter prediction mode, deriving the motion information, and generating predicted samples can be performed simultaneously, or one step can be performed before the other steps.

[0079] For example, the inter prediction unit 221 of the encoding apparatus may include a prediction mode determination unit 221_1, a motion information derivation unit 221_2, and a prediction sample derivation unit 221_3. The prediction mode determination unit 221_1 may determine a prediction mode for the current block, the motion information derivation unit 221_2 may derive motion information of the current block, and the prediction sample derivation unit 221_3 may derive a prediction sample for the current block. For example, the inter prediction unit 221 of the encoding apparatus may search for blocks similar to the current block within a certain region (search region) of a reference picture through motion estimation and derive a reference block whose difference from the current block is minimum or equal to or less than a certain criterion. Based on this, the inter prediction unit 221 may derive a reference picture index indicating the reference picture in which the reference block is located, and derive a motion vector based on the positional difference between the reference block and the current block. The encoding apparatus may determine a mode to be applied to the current block from various prediction modes. The encoding apparatus may compare RD costs for various prediction modes to determine an optimal prediction mode for the current block.

[0080] For example, when a skip mode or a merge mode is applied to a current block, the encoding device may construct a merge candidate list and derive a reference block, among reference blocks indicated by merge candidates included in the merge candidate list, whose difference from the current block is minimum or equal to or less than a certain criterion. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to a decoding device. Motion information of the current block may be derived using motion information of the selected merge candidate.

[0081] As another example, when the (A)MVP mode is applied to the current block, the encoding apparatus may construct an (A)MVP candidate list and use the motion vector of a selected MVP (motion vector predictor) candidate from among the MVP candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, a motion vector pointing to a reference block derived by the above-described motion estimation may be used as the motion vector of the current block, and the MVP candidate having the smallest difference from the motion vector of the current block may be selected as the MVP candidate. A motion vector difference (MVD), which is the difference obtained by subtracting the MVP from the motion vector of the current block, may be derived. In this case, information regarding the MVD may be signaled to the decoding apparatus. Furthermore, when the (A)MVP mode is applied, the value of the reference picture index may be configured as reference picture index information and separately signaled to the decoding apparatus.

[0082] The encoding device may derive residual samples based on the predicted samples (S410). The encoding device may derive residual samples by comparing the original samples of the current block with the predicted samples.

[0083] The encoding apparatus encodes video information including prediction information and residual information (S420). The encoding apparatus can output the encoded video information in the form of a bitstream. The prediction information can include prediction mode information (e.g., a skip flag, a merge flag, or a mode index) and information about motion information, which are information related to a prediction procedure. The information about the motion information can include candidate selection information (e.g., a merge index, an MVP flag, or an MVP index), which is information for deriving a motion vector. The information about the motion information can also include information about the above-mentioned MVD and / or reference picture index information. The information about the motion information can also include information indicating whether L0 prediction, L1 prediction, or bi-prediction is applied. The residual information is information about residual samples. The residual information can include information about quantized transform coefficients for the residual samples.

[0084] The output bitstream can be stored on a (digital) storage medium and transmitted to the decoding device, or can be transmitted to the decoding device via a network.

[0085] As described above, the encoding apparatus can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on reference samples and residual samples. This is because the encoding apparatus can derive the same prediction result as that performed in the decoding apparatus, thereby improving coding efficiency. Therefore, the encoding apparatus can store the reconstructed picture (or reconstructed samples, reconstructed blocks) in memory and use it as a reference picture for inter-prediction. As described above, an in-loop filtering procedure can be further applied to the reconstructed picture.

[0086] Figure 6 shows an example of a video / image decoding method based on inter prediction, and Figure 7 shows an example of a schematic diagram of an inter prediction unit in a decoding device. The inter prediction unit in the decoding device of Figure 7 may be applied in the same or corresponding manner to the inter prediction unit 332 of the decoding device 300 of Figure 3 described above.

[0087] 6 and 7, the decoding device may perform operations corresponding to those performed in the encoding device. The decoding device may perform prediction on the current block based on the received prediction information to derive predicted samples.

[0088] Specifically, the decoding device may determine a prediction mode for a current block based on received prediction information (S600). The decoding device may determine which inter prediction mode is applied to the current block based on prediction mode information in the prediction information.

[0089] For example, it may determine whether a merge mode or an (A)MVP mode is applied to the current block based on the merge flag. Alternatively, it may select one of various inter prediction mode candidates based on the mode index. The inter prediction mode candidates may include skip mode, merge mode, and / or (A)MVP mode, or may include the various inter prediction modes described above.

[0090] The decoding apparatus derives motion information of the current block based on the determined inter prediction mode (S610). For example, when a skip mode or a merge mode is applied to the current block, the decoding apparatus may construct a merge candidate list and select one merge candidate from among the merge candidates included in the merge candidate list. Here, the selection may be made based on the above-mentioned selection information (merge index). Motion information of the selected merge candidate may be used to derive motion information of the current block. The motion information of the selected merge candidate may be used as motion information of the current block.

[0091] As another example, when the (A)MVP mode is applied to the current block, the decoding device constructs an (A)MVP candidate list and uses a motion vector of a selected MVP (motion vector predictor) candidate from among the MVP candidates included in the (A)MVP candidate list as the MVP of the current block. Here, the selection may be made based on the above-mentioned selection information (MVP flag or MVP index). In this case, the MVD of the current block may be derived based on information related to the MVD, and the motion vector of the current block may be derived based on the MVP and MVD of the current block. In addition, the reference picture index of the current block may be derived based on reference picture index information. A picture pointed to by a reference picture index in the reference picture list for the current block may be derived as a reference picture referenced for inter-prediction of the current block.

[0092] On the other hand, the motion information of the current block can be derived without constructing a candidate list, in which case the motion information of the current block can be derived according to the procedure started in the prediction mode. In this case, the candidate list construction as described above can be omitted.

[0093] The decoding device may generate predictive samples for the current block based on the motion information of the current block (S620). In this case, a reference picture may be derived based on the reference picture index of the current block, and the predictive samples of the current block may be derived using samples of the reference block pointed to by the motion vector of the current block in the reference picture. In this case, a predictive sample filtering procedure may be further performed on all or some of the predictive samples of the current block, depending on the case.

[0094] For example, the inter prediction unit 332 of the decoding device may include a prediction mode determination unit 332_1, a motion information derivation unit 332_2, and a prediction sample derivation unit 332_3, and may determine a prediction mode for a current block based on prediction mode information received in the prediction mode determination unit 332_1, derive motion information (such as a motion vector and / or a reference picture index) for the current block based on information regarding the motion information received from the motion information derivation unit 332_2, and have the prediction sample derivation unit 332_3 derive a prediction sample for the current block.

[0095] The decoding device generates residual samples for the current block based on the received residual information (S630). The decoding device generates reconstructed samples for the current block based on the predicted samples and the residual samples, and can generate a reconstructed picture based on the reconstructed samples (S640). Thereafter, as described above, an in-loop filtering procedure can be further applied to the reconstructed picture.

[0096] As described above, the inter prediction procedure may include an inter prediction mode determination step, a motion information deriving step according to the determined prediction mode, and a prediction (prediction sample generation) step based on the derived motion information. The inter prediction procedure may be performed in an encoding device and a decoding device, as described above.

[0097] On the other hand, skip mode and / or merge mode have limitations in motion prediction because they predict the motion of the current block based on the motion vectors of neighboring blocks without MVD (Motion Vector Difference). To overcome the limitations of skip mode and / or merge mode, the motion vector can be refined by applying Decoder-side Motion Vector Refinement (DMVR), Bi-Directional Optical Flow (BDOF) mode, etc. DMVR and BDOF modes can be used when true paired prediction is applied to the current block.

[0098] FIG. 8 is a diagram illustrating one embodiment of a process for performing decoder-side motion vector refinement (DMVR) in true paired prediction.

[0099] DMVR is a method for performing motion prediction by refining motion information of neighboring blocks on the decoder side. When DMVR is applied, the decoder can derive refined motion information through cost comparison based on a template generated using motion information of neighboring blocks in merge / skip mode. In this case, the accuracy of motion prediction can be increased without additional signaling information, thereby improving compression performance.

[0100] For convenience of explanation, this document will mainly describe a decoding device, but the DMVR according to the embodiment of this document can be implemented in the same way in an encoding device.

[0101] 8, the decoding device may derive prediction blocks (i.e., reference blocks) identified by initial motion vectors (or motion information) in the list0 and list1 directions (e.g., MV0 and MV1), and generate a template (or bilateral template) using a weighted sum (e.g., average) of the derived prediction blocks (step 1). Here, the initial motion vectors (MV0 and MV1) may represent motion vectors derived using motion information of neighboring blocks in merge / skip mode.

[0102] Then, the decoding apparatus can derive motion vectors (e.g., MV0' and MV1') that minimize the difference between the template and the sample area of ​​the reference picture through a template matching operation (step 2). Here, the sample area represents a surrounding area of ​​the initial prediction block in the reference picture, and the sample area may be referred to as a surrounding area, reference area, search space, search range, search space, etc. The template matching operation may include an operation of calculating a cost measurement value between the template and the sample area of ​​the reference picture. For example, the cost measurement may use SAD (Sum of Absolute Differences). As an example, a normalized SAD may be used as the cost function. In this case, the matching cost may be given by SAD(T-mean(T), 2*P[x]-2*mean(P[x])). Here, T represents the template, and P[x] represents a block in the search space. Then, the motion vector that calculates the minimum template cost for each of the two reference pictures can be considered as an updated motion vector (replacing the initial motion vector). As shown in Figure 8, the decoding device can generate a final bidirectional prediction result (i.e., a final bidirectional prediction block) using the updated motion vectors MV0' and MV1'. In one embodiment, multi-iteration for deriving an updated (or new) motion vector can be used to obtain the final bidirectional prediction result.

[0103] In one embodiment, the decoding device may invoke the DMVR process to improve the accuracy of initial motion compensation prediction (i.e., motion compensation prediction via conventional merge / skip mode). For example, the decoding device may perform the DMVR process when the prediction mode of the current block is merge mode or skip mode and bidirectional pair prediction, in which both reference pictures are in opposite directions relative to the current picture in display order, is applied to the current block.

[0104] FIG. 9 is a diagram illustrating an embodiment of a process for performing Decoder-side Motion Vector Refinement (DMVR) using Sum of Absolute Differences (SAD).

[0105] As described above, the decoding device can measure the matching cost using SAD when performing DMVR. As an embodiment, FIG. 9 illustrates a method for refining a motion vector by calculating the Mean Sum of Absolute Difference (MRSAD) between prediction samples in two reference pictures without generating a template. That is, the method of FIG. 9 illustrates an embodiment of bilateral matching using MRSAD.

[0106] 9, the decoding device can derive, on the L0 reference picture, neighboring pixels of a pixel (sample) indicated by a motion vector (MV0) in the list0 (L0) direction, and derive, on the L1 reference picture, neighboring pixels of a pixel (sample) indicated by a motion vector (MV1) in the list1 (L1) direction. Then, the decoding device can measure the matching cost by calculating the MRSAD between the L0 predicted block (i.e., the L0 reference block) identified by the motion vector indicating the neighboring pixels derived on the L0 reference picture and the L1 predicted block (i.e., the L1 reference block) identified by the motion vector indicating the neighboring pixels derived on the L1 reference picture. At this time, the decoding device can select the search point with the minimum cost (i.e., the search space with the minimum SAD between the L0 predicted block and the L1 predicted block) as the refined motion vector pair. That is, the refined motion vector pair may include a refined L0 motion vector pointing to a pixel location (L0 prediction block) with the smallest cost in the L0 reference picture and a refined L1 motion vector pointing to a pixel location (L1 prediction block) with the smallest cost in the L1 reference picture.

[0107] In one embodiment, after the search space of the reference picture is set in calculating the matching cost, unidirectional prediction may be performed using a regular 8-tap DCTIF interpolation filter. Also, as an example, the MRSAD calculation may use 16-bit precision, and clipping and / or rounding operations may not be applied before the MRSAD calculation in consideration of the internal buffer.

[0108] As described above, when true pairwise prediction is applied to the current block, BDOF can be used to refine the pairwise prediction signal. When pairwise prediction is applied to the current block, BDOF (Bi-Directional Optical Flow) can be used to calculate improved motion information and generate prediction samples based on the information. For example, BDOF can be applied at a 4x4 sub-block level. That is, BDOF can be performed in units of 4x4 sub-blocks within the current block. Alternatively, BDOF can be applied only to the luma component. Alternatively, BDOF can be applied only to the chroma component, or to the luma component and the chroma component.

[0109] As the name suggests, BDOF mode is based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4x4 sub-block, motion refinement (v) is performed by minimizing the difference between the L0 and L1 predicted samples. x ,v y ) can be calculated, and motion refinement can be used to adjust pairwise predicted sample values ​​in the 4x4 sub-blocks.

[0110] More specifically, in refining the prediction signal by applying BDOF, horizontal and vertical gradients of the L0 prediction sample and the L1 prediction sample can be calculated first. Here, the horizontal and vertical gradients can be calculated based on the difference between two neighboring samples located around the prediction sample (i, j), and can be calculated as shown in Equation 1 below.

[0111] <Expression 1>

number

[0112] During the ceremony, JPEG2026035608000004.jpg1117 is horizontal gradation, JPEG2026035608000005.jpg1117 is a vertical gradation. JPEG2026035608000006.jpg816 is the predicted value at coordinate (i,j) of the predicted sample in list k (k=0,1).

[0113] Next, the auto-correlation and cross-correlation of the horizontal and vertical grayscales can be calculated as follows:

[0114] <Expression 2>

number

[0115] <Expression 3>

number

[0116] During the ceremony, JPEG2026035608000009.jpg65 is a 6x6 window around a 4x4 subblock.

[0117] Next, motion refinement (v x ,v y ) is calculated using auto-correlation and cross-correlation, and can be calculated as follows:

[0118] <Expression 4>

number

[0119] During the ceremony, JPEG2026035608000011.jpg835, JPEG2026035608000012.jpg742, JPEG2026035608000013.jpg728, JPEG2026035608000014.jpg76 is the floor function.

[0120] Next, based on the above gray level and motion refinement, b(x,y) for refinement of the BDOF prediction sample can be calculated as follows:

[0121] <Formula 5>

number

[0122] Finally, the BDOF predicted sample (i.e., the predicted sample value refined by applying the BDOF) can be calculated as follows:

[0123] <Formula 6>

number

[0124] Meanwhile, the above-mentioned DMVR and BDOF are refinement techniques with similar concepts in that they refine motion information to perform prediction when true pair prediction is applied (where true pair prediction refers to motion prediction / compensation in a reference picture in a different direction based on the picture of the current block), and assume that the movement of an object within a picture occurs at a constant speed and in a constant direction. However, when true pair prediction is performed, the conditions for applying DMVR and BDOF are different, so multiple condition checks must be performed for each technique. Therefore, this document proposes a method that can improve the efficiency in terms of decoder complexity and performance by improving the condition check process when determining the prediction mode to be applied to the current block.

[0125] Table 2 below shows the conditions for applying DMVR when using traditional true-to-true prediction. DMVR can be applied when all the conditions listed below are met.

[0126] [Table 2]

[0127] Referring to Table 2 above, 1) whether to apply DMVR can be determined based on flag information (e.g., sps_dmvr_enabled_flag) signaled in the SPS (Sequence Parameter Set) syntax. Here, the flag information (e.g., sps_dmvr_enabled_flag) can indicate whether true vs. prediction-based DMVR is enabled. For example, if sps_dmvr_enabled_flag is 1 (i.e., true vs. prediction-based DMVR is enabled), it can be determined that the DMVR enablement condition is met.

[0128] 2) Whether to apply DMVR can be determined based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode. For example, if merge_flag is 1 (i.e., if inter prediction is performed using merge mode / skip mode), it can be determined that the condition for applying merge mode / skip mode is met.

[0129] 3) Whether to apply DMVR can be determined based on flag information (e.g., mmvd_flag) indicating whether inter prediction is performed using MMVD (Merge mode with Motion Vector Difference) mode. For example, if mmvd_flag is 0 (i.e., if MMVD mode is not used), it can be determined that the conditions for applying MMVD mode are met.

[0130] 4) Whether to apply DMVR may be determined based on whether bidirectional prediction (pairwise prediction) is used. Here, bidirectional prediction may refer to inter-prediction performed based on reference pictures located in different directions relative to the current picture. For example, if predFlagL0[0][0]=1 and predFlagL0[1][1]=1, it may be determined that bidirectional prediction is applied, and it may be determined that the bidirectional prediction condition is met.

[0131] 5) Whether to apply DMVR can be determined based on whether the distances between the current picture and the reference pictures in both directions are the same in true paired prediction. That is, it can be determined whether the distance between the current picture and the L0 reference picture (i.e., the reference picture in the reference picture list L0) and the distance between the current picture and the L1 reference picture (i.e., the reference picture in the reference picture list L1) are the same. For example, if DiffPicOrderCnt(currPic,RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic,RefPicList1[refIdxL1])=0, it can be determined that the distances between the current picture and the reference pictures in both directions are the same in true paired prediction, and it can be determined that the condition for the reference picture distances in both directions being the same is met.

[0132] 6) Whether to apply DMVR can be determined based on whether the height of the current block is greater than a threshold. For example, if the height of the current block is 8 or greater, it can be determined that the size (length) condition of the current block is met.

[0133] 7) Whether or not to apply DMVR can be determined based on whether the size of the current block is greater than a threshold. For example, if the size of the current block, i.e., length (height) * width (width), is 64 or greater, it can be determined that the size (length * width) condition of the current block is met.

[0134] The decoding device can determine whether to apply DMVR depending on whether conditions 1) to 7) in Table 2 are satisfied. That is, if all of conditions 1) to 7) in Table 2 are satisfied, the decoding device can apply DMVR to perform true paired prediction, and if any of the conditions in Table 2 are not satisfied, the decoding device does not apply DMVR.

[0135] Table 3 below shows the conditions for applying BDOF when performing conventional true-to-prediction. BDOF can be applied when all of the conditions listed below are met.

[0136] [Table 3]

[0137] Referring to Table 3 above, 1) whether to apply BDOF can be determined based on flag information (e.g., sps_bdof_enabled_flag) signaled in the SPS (Sequence Parameter Set) syntax. Here, the flag information (e.g., sps_bdof_enabled_flag) can indicate whether true vs. prediction-based BDOF is enabled. For example, if sps_bdof_enabled_flag is 1 (i.e., true vs. prediction-based BDOF is enabled), it can be determined that the BDOF enabling condition is met.

[0138] 2) Whether to apply BDOF can be determined based on whether bidirectional prediction is used. Here, bidirectional prediction can refer to inter-prediction performed based on reference pictures located in different directions relative to the current picture. For example, if predFlagL0 and predFlagL1 are both 1, it can be determined that bidirectional prediction is applied, and it can be determined that the bidirectional prediction condition is met.

[0139] 3) Whether to apply BDOF can be determined based on whether true paired prediction is true. That is, whether an L0 reference picture (i.e., a reference picture in the reference picture list L0) and an L1 reference picture (i.e., a reference picture in the reference picture list L1) are located in different temporal directions relative to the current picture can be determined. For example, if DiffPicOrderCnt(currPic, refPicList0[refIdxL0]) * DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0, it can be determined that the reference pictures in both directions are located in different temporal directions relative to the current picture, and thus the condition for true paired prediction can be satisfied.

[0140] 4) Whether or not to apply BDOF can be determined based on whether or not the affine mode is used. Here, whether or not the affine mode is used can be determined by deriving MotionModelIdc. For example, if the derived MotionModelIdc is 0, it can be determined that the affine mode is not used, and in this case, it can be determined that the condition for whether or not to apply the affine mode is met.

[0141] 5) Whether to apply BDOF can be determined based on flag information (e.g., merge_subblock_flag) indicating whether inter prediction in merge mode is performed on a subblock-by-subblock basis. For example, if merge_subblock_flag is 0 (i.e., merge mode is not applied on a subblock-by-subblock basis), it can be determined that the condition for subblock-based merge mode is met.

[0142] 6) Whether or not BDOF is applied can be determined based on whether GBi exists. Here, whether GBi exists can be determined based on GBi index information (e.g., GbiIdx). For example, if GbiIdx is 0 (i.e., if GbiIdx is the default), it can be determined that the condition for GBi is met.

[0143] 7) Whether to apply BDOF can be determined based on whether the current block is a luma block including a luma component. For example, if an index (e.g., cIdx) indicating whether the current block is a luma block is 0 (i.e., the current block is a luma block), it can be determined that the condition for the luma block is met.

[0144] The decoding device can determine whether to apply BDOF depending on whether conditions 1) to 7) in Table 3 are satisfied. That is, the decoding device can apply BDOF and perform true pair prediction when all conditions 1) to 7) in Table 3 are satisfied, and does not apply BDOF when any of the conditions in Table 3 are not satisfied.

[0145] Here, the above-mentioned GBi may represent generalized bi-prediction, which allows different weights to be applied to L0 prediction and L1 prediction, and may be represented using, for example, GbiIdx. GbiIdx may be present in the case of bi-prediction and may represent a bi-prediction weight index. The motion information in this document may further include GbiIdx. For example, GbiIdx may be derived from surrounding blocks in the case of merge mode, or may be signaled from the encoding device to the decoding device via a GbiIdx syntax element (e.g., gbi_idx) in the case of MVP mode. As an example, GbiIdx may indicate a weighting w applied to L1 prediction, in which case a weighting of (1-w) may be applied to L0 prediction. As another example, GbiIdx may indicate a weighting w applied to L0 prediction, in which case a weighting of (1-w) may be applied to L1 prediction. The weighting indicated by GbiIdx can be configured in various ways, for example, as shown in Tables 4 and 5 below.

[0146] [Table 4]

[0147] [Table 5]

[0148] Referring to Tables 4 and 5 above, the weighting w1 may represent the weighting applied to L1 prediction, and the weighting w1 applied to L1 prediction may be indicated via the GbiIdx value. For example, according to the embodiment of Table 4, when the value of GbiIdx represents 0, a weighting of 1 / 2 may be applied to L1 prediction, and a weighting of 1 / 2, which is a value of (1-w1), may be applied to L0 prediction. Depending on the embodiment, the weighting w1 may also represent the weighting applied to L0 prediction, and in this case, the weighting w1 applied to L0 prediction may be indicated via the GbiIdx value.

[0149] As described above, the application conditions for DMVR and BDOF are partially the same, partially similar, or different. In conventional methods, even when the conditions are the same, the conditions are checked separately for each technology, which increases the complexity of performing paired prediction. Therefore, this paper proposes efficient conditions for applying DMVR and BDOF during paired prediction.

[0150] Since the merge / skip mode has relatively low motion accuracy compared to the AMVP mode, refining motion information using the DMVR method is effective in terms of performance. However, unlike DMVR, the BDOF mode is applied not only to the merge / skip mode but also to the AMVP mode. However, when BDOF is applied in the AMVP mode, the complexity of performing BDOF may increase compared to performance. Therefore, in this embodiment, a method is proposed in which BDOF is applied to the merge / skip mode as well as the DMVR.

[0151] In this case, as one embodiment proposed in this document, the application conditions of BDOF may include the conditions presented in Table 6 below.

[0152] [Table 6]

[0153] Referring to Table 6, whether to apply BDOF can be determined based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode. For example, if merge_flag is 1 (i.e., if inter prediction is performed using merge mode / skip mode), it can be determined that the condition for applying merge mode / skip mode is met. Therefore, like DMVR, BDOF can also be applied in merge / skip mode.

[0154] That is, in this embodiment, whether to apply BDOF can be determined based on the conditions to be applied in the above-mentioned merge / skip mode, as well as the conditions to be applied when BDOF is possible, the conditions to be applied in the case of bidirectional prediction, the conditions to be applied in the case of true pair prediction, the conditions to be applied when it is not affine prediction, the conditions to be applied when it is not sub-block based merge mode, the conditions to be applied when the GBi index is default, and the conditions to be applied in the case of luma blocks.

[0155] Therefore, the decoding device determines whether all the conditions listed in Table 6 are met, and if all the conditions are met, it can apply BDOF to perform true pair prediction. If any one of the conditions listed in Table 6 is not met, the decoding device can not apply BDOF. The conditions in Table 6 can also be applied to the encoding device, and in the encoding device, (true pair prediction) can be performed in a manner corresponding to that of the decoding device.

[0156] Based on the conditions listed in Table 6 above, the method of making a true pair prediction can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0157] Meanwhile, among the conditions listed in this embodiment, when the conditions are the same as those described in Tables 2 and 3, the specific operations or meanings are similarly applied, so a detailed description of each condition will be omitted. Also, duplicated content will be omitted in the following embodiments.

[0158] Since encoding / decoding devices can be configured using various hardware, and preferences for the ratio of complexity to performance may vary, this embodiment proposes a method for refining motion information by applying DMVR not only in merge / skip mode but also in AMVP mode.

[0159] In this case, as one embodiment proposed in this document, the application conditions of the DMVR may include the conditions presented in Table 7 below.

[0160] [Table 7]

[0161] Referring to Table 7, the process of determining whether to apply DMVR based on flag information (e.g., merge_flag) indicating whether inter prediction is performed using merge mode / skip mode can be omitted. In this way, by omitting the condition of whether to apply merge mode / skip mode, DMVR can be applied not only in merge mode / skip mode but also in AMVP mode.

[0162] According to Table 7 above, it is possible to determine whether to apply DMVR based on the conditions to be applied when DMVR is possible, the conditions to be applied when MMVD mode is not used, the conditions to be applied in the case of bidirectional prediction, the conditions to be applied in the case of true pair prediction where the distance between the current picture and the bidirectional reference picture is the same, the conditions to be applied when the length of the current block is 8 or more, and the conditions to be applied when the size of the current block (length * width) is 64 or more.

[0163] That is, the decoding device determines whether all the conditions listed in Table 7 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true pair prediction. If any one of the conditions listed in Table 7 is not met, the decoding device can not apply DMVR. The conditions in Table 7 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0164] Based on the conditions listed in Table 7 above, the method of making a true pair prediction can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0165] Meanwhile, in one embodiment of this document, DMVR and BDOF can be applied to all normal merge modes. That is, DMVR and BDOF can be applied in modes other than the Advanced Temporal Motion Vector Prediction (ATMVP) mode, the affine mode, and the CPR mode. In this case, the conditions for applying DMVR include the conditions presented in Table 8 below.

[0166] [Table 8]

[0167] Referring to Table 8 above, DMVR can be applied only in the case of general merge mode by determining whether the conditions to be applied when affine mode is not used (e.g., when MotionModelIdc is 0) and the conditions to be applied when sub-block-based merge mode is not used (e.g., when merge_subblock_flag is 0) are met.

[0168] In addition, in this embodiment, whether or not DMVR is applied can be determined based on the conditions for whether the above-mentioned affine mode is possible, and whether or not the sub-block based merge mode is possible, as well as the conditions to be applied when DMVR is possible, the conditions to be applied when the merge mode / skip mode is used, the conditions to be applied when the MMVD mode is not used, the conditions to be applied when bidirectional prediction is used, the conditions to be applied when true paired prediction is used in which the distance between the current picture and the bidirectional reference picture is the same, the conditions to be applied when the length of the current block is 8 or more, and the conditions to be applied when the size of the current block (length * width) is 64 or more.

[0169] That is, the decoding device determines whether all the conditions listed in Table 8 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true paired prediction. If any one of the conditions listed in Table 8 is not met, the decoding device can not apply DMVR. The conditions in Table 8 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0170] Among the application conditions in Table 8, the condition for whether the sub-block-based merge mode is applicable (e.g., merge_subblock_flag) includes overlapping conditions among the conventional DMVR application conditions. Therefore, in one embodiment of the present document, the overlapping condition with the condition for whether the sub-block-based merge mode is applicable (e.g., merge_subblock_flag) can be removed. In this case, the corresponding condition can be removed as proposed in Table 9 below.

[0171] [Table 9]

[0172] Referring to Table 9, the sub-block-based merge mode can be applied when the size of the current block is 8x8 or more. Therefore, the condition for determining whether the sub-block-based merge mode is applicable (e.g., merge_subblock_flag=0) includes a condition related to the size of the current block, and therefore, among the conventional DMVR application conditions, the condition related to the size of the current block (e.g., CbHeight, CbHeight*CbWidth) can be excluded. For example, the conditions for whether the length of the current block is 8 or more and whether the length*width of the current block is 64 or more can be omitted, and the remaining conditions listed in Table 9 can be used to determine whether DMVR is applied.

[0173] That is, the decoding device determines whether all the conditions listed in Table 9 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true pair prediction. If any one of the conditions listed in Table 9 is not met, the decoding device can not apply DMVR. The conditions in Table 9 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0174] Based on the conditions listed in Table 8 or Table 9 above, the method of making a true pair prediction can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0175] In one embodiment of this document, in order to improve the accuracy of motion vectors at low complexity, refinement techniques such as DMVR and BDOF may not be applied when the block size is small. In conventional methods, refinement techniques are applied when the current block is larger than 8x8 or the same block, but in the case of DMVR, when the size of the current block is large, refinement is applied in 16x16 units, so DMVR may not be applied to blocks smaller than 16x16. In this case, the conditions for applying DMVR may include the conditions presented in Table 10 below.

[0176] [Table 10]

[0177] Referring to Table 10, by changing the conditions (e.g., CbHeight, CbWidth) related to the size of the current block, it is possible to prevent DMVR from being applied to blocks smaller than 16x16. For example, a condition to be applied when the length (e.g., CbHeight) of the current block is 16 or more and a condition to be applied when the width (e.g., CbWidth) of the current block is 16 or more can be used. If such an application condition related to the size of the current block is met (i.e., if the size of the current block is 16x16 or more), DMVR can be applied, and if such an application condition related to the size of the current block is not met (i.e., if the size of the current block is smaller than 16x16), DMVR can be not applied.

[0178] In addition, in this embodiment, it is possible to determine whether to apply DMVR based on the remaining application conditions listed in Table 10 above, along with the conditions related to the size of the current block (for example, CbHeight, CbWidth).

[0179] That is, the decoding device determines whether all the conditions listed in Table 10 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true paired prediction. If any one of the conditions listed in Table 10 is not met, the decoding device can not apply DMVR. The conditions in Table 10 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0180] In one embodiment of this document, if the current block is smaller than 16x16, not only DMVR but also BDOF may not be applied. In this case, the conditions for applying BDOF may include the conditions presented in Table 11 below.

[0181] [Table 11]

[0182] Referring to Table 11 above, by changing the conditions (e.g., CbHeight, CbWidth) related to the size of the current block, BDOF may not be applied to blocks smaller than 16x16. For example, a condition may be used that is applied when the length (e.g., CbHeight) of the current block is 16 or more, and a condition may be used that is applied when the width (e.g., CbWidth) of the current block is 16 or more. BDOF may be applied when such application conditions related to the size of the current block are met (i.e., when the size of the current block is 16x16 or more), and not applied when such application conditions related to the size of the current block are not met (i.e., when the size of the current block is smaller than 16x16).

[0183] Furthermore, in this embodiment, it is possible to determine whether to apply BDOF based on the remaining application conditions listed in Table 11 above, along with the conditions related to the size of the current block (for example, CbHeight, CbWidth).

[0184] That is, the decoding device determines whether all the conditions listed in Table 11 are met, and if all the conditions are met, it can apply BDOF to perform true paired prediction. If any one of the conditions listed in Table 11 is not met, the decoding device can not apply BDOF. The conditions in Table 11 can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0185] The method of making a true pair prediction based on the conditions listed in Table 10 or Table 11 above can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0186] As described above, DMVR is applied when the distances between the current picture and the reference pictures in both directions are the same, whereas BDOF is always applied when the distances between the current picture and the reference pictures in both directions are different and true pair prediction is used. Therefore, in order to improve coding efficiency, this document proposes a method in which the conditions related to the reference picture distances in both directions can be applied to DMVR and BDOF in a unified manner.

[0187] As an embodiment of this document, the application conditions of BDOF may include the conditions presented in Table 12 below.

[0188] [Table 12]

[0189] Referring to Table 12, among the BDOF application conditions, by changing a condition related to the reference picture distance (e.g., DiffPicOrderCnt), the corresponding condition can be applied to both DMVR and BDOF in the same way. For example, by determining whether DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1]) is 0, it can be determined whether the distance between the current picture and the L0 reference picture (i.e., a reference picture in the reference picture list L0) and the distance between the current picture and the L1 reference picture (i.e., a reference picture in the reference picture list L1) are the same. In other words, BDOF can be applied only when the distances between the current picture and the reference pictures in both directions are the same. In this way, by adding the conditions that it is true pairwise prediction and the reference picture distances in both directions are the same, the BDOF application range is limited, thereby reducing decoding complexity.

[0190] In addition, in this embodiment, it is possible to determine whether to apply BDOF based on the remaining application conditions listed in Table 12 above, along with the conditions related to the reference picture distance (for example, DiffPicOrderCnt).

[0191] That is, the decoding device determines whether all the conditions listed in Table 12 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 12 is not met, the decoding device can not apply BDOF. The conditions in Table 12 can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0192] In one embodiment of this document, the applicable conditions of the DMVR may include the conditions presented in Table 13 below.

[0193] [Table 13]

[0194] Referring to Table 13, by changing the condition related to the reference picture distance (e.g., DiffPicOrderCnt) among the DMVR application conditions, the corresponding condition can be applied to both DMVR and BDOF in the same way. For example, by determining whether DiffPicOrderCnt(currPic, refPicList0[refIdxL0]) * DiffPicOrderCnt(currPic, refPicList1[refIdxL1]) is less than 0, it can be determined whether the reference pictures in both directions (i.e., the L0 reference picture and the L1 reference picture) are true pair predictions located in different directions relative to the current picture. In other words, even if the distances between the current picture and the reference pictures in both directions are not the same, DMVR can always be applied if true pair predictions are true pair predictions. In this way, by applying the condition of whether true pair predictions are possible, the derived motion vector can be used without scaling, taking into account decoding complexity, even when the reference picture distances in both directions are different.

[0195] In addition, in this embodiment, it is possible to determine whether to apply DMVR based on the remaining application conditions listed in Table 13 above, along with the conditions related to the reference picture distance (for example, DiffPicOrderCnt).

[0196] That is, the decoding device determines whether all the conditions listed in Table 13 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true paired prediction. If any one of the conditions listed in Table 13 is not met, the decoding device can not apply DMVR. The conditions in Table 13 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0197] The method of making a true pair prediction based on the conditions listed in Table 12 or Table 13 above can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0198] On the other hand, each reference block may undergo motion compensation using a weighted sum due to changes in light, etc. In this case, this phenomenon can be grasped using GBi or LIC (Local Illumination Compensation), so the application conditions of DMVR and BDOF can be determined taking into account the GBi or LIC conditions.

[0199] As an embodiment of this document, we propose a method for determining whether to apply DMVR by taking into account GBi and LIC conditions. In this case, the DMVR application conditions may include the conditions presented in Table 14 below.

[0200] [Table 14]

[0201] Referring to Table 14 above, a GBi condition (e.g., GbiIdx) and a LIC condition (e.g., LICFlag) can be added to determine whether DMVR is applied. For example, if GbiIdx is 0 (i.e., GbiIdx is default), it can be determined that the condition for whether GBi is enabled is met, and if LICFlag is 0 (i.e., LIC exists), it can be determined that the condition for whether LIC is enabled is met.

[0202] Furthermore, in this embodiment, it is possible to determine whether or not to apply DMVR based on the remaining application conditions listed in Table 14 above, along with the GBi conditions (for example, GbiIdx) and LIC conditions (for example, LICFlag).

[0203] That is, the decoding device determines whether all the conditions listed in Table 14 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true paired prediction. If any one of the conditions listed in Table 14 is not met, the decoding device can not apply DMVR. The conditions in Table 14 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0204] As an embodiment of this document, we propose a method for determining whether to apply BDOF by taking into account the GBi and LIC conditions. In this case, the conditions for applying BDOF may include the conditions presented in Table 15 below.

[0205] [Table 15]

[0206] According to Table 15 above, a LIC condition (e.g., LICFlag) can be added to the conventional GBi condition (e.g., GbiIdx) to determine whether BDOF is applied. For example, if GbiIdx is 0 (i.e., GbiIdx is default), it can be determined that the condition for whether GBi is enabled is met, and if LICFlag is 0 (i.e., LIC exists), it can be determined that the condition for whether LIC is enabled is met.

[0207] Therefore, in this embodiment, it is possible to determine whether to apply BDOF based on the remaining application conditions listed in Table 15 above, along with the GBi conditions (for example, GbiIdx) and LIC conditions (for example, LICFlag).

[0208] That is, the decoding device determines whether all the conditions listed in Table 15 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 15 is not met, the decoding device can not apply BDOF. The conditions in Table 15 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0209] The method of making a true pair prediction based on the conditions listed in Table 14 or Table 15 above can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0210] On the other hand, DMVR and BDOF derive motion information through a refinement process in a decoding device, which increases decoding complexity. Therefore, this document proposes a method that can reduce decoding complexity by determining whether to apply DMVR and BDOF using a merge index. In this case, since DMVR and BDOF refine motion vectors within a limited range, the effectiveness of refinement may be reduced if the motion vector is inaccurate. Therefore, this document proposes a method that can be applied only when the value represented by the merge index is small, taking into account the efficiency of refinement.

[0211] Here, the merge index may be a syntax element signaled from the encoding device to the decoding device. For example, when a merge mode / skip mode is applied to a current block, the encoding / decoding device may construct a merge candidate list based on neighboring blocks of the current block. In this case, the encoding device may select an optimal merge candidate from among the merge candidates included in the merge candidate list based on the RD (Rate-Distortion) cost, and signal merge index information indicating the selected merge candidate to the decoding device. The decoding device may select a merge candidate to be applied to the current block based on the merge candidate list and the merge index information.

[0212] As an embodiment of this document, the method for determining whether to apply DMVR using a merge index may include the conditions presented in Table 16 below.

[0213] [Table 16]

[0214] Referring to Table 16, a merge index condition (e.g., merge_idx) can be added to determine whether to apply DMVR. For example, if the merge index (e.g., merge_idx) is less than 2, it can be determined that the merge index condition is met. Here, the merge index value (threshold) is set to 2, but this is just an example, and the corresponding value can be changed depending on coding efficiency.

[0215] Therefore, in this embodiment, it is possible to determine whether to apply a DMVR based on the merge index condition (for example, merge_idx) as well as the remaining application conditions listed in Table 16 above.

[0216] That is, the decoding device determines whether all the conditions listed in Table 16 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true pair prediction. If any one of the conditions listed in Table 16 is not met, the decoding device can not apply DMVR. The conditions in Table 165 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0217] As an embodiment of this document, the method for determining whether to apply BDOF using a merge index may include the conditions presented in Table 17 below.

[0218] [Table 17]

[0219] Referring to Table 17 above, whether to apply BDOF can be determined by adding a condition (e.g., merge_flag) for determining whether merge mode / skip mode is possible and a merge index condition (e.g., merge_idx). For example, if merge_flag is 1 (i.e., when inter prediction is performed using merge mode / skip mode) and merge_idx is not 2 or greater, it can be determined that the condition for applying BDOF is met only when the merge index value is small. In other words, if merge_flag is 1 (i.e., when inter prediction is performed using merge mode / skip mode) and merge_idx is less than 2, it can be determined that the merge index condition is met and BDOF can be applied. Here, the merge index value (threshold) is set to 2, but this is merely an example, and the corresponding value can be changed depending on coding efficiency.

[0220] That is, in this embodiment, whether to apply BDOF can be determined based on the conditions for whether merge mode / skip mode is possible (e.g., merge_flag), the merge index conditions (e.g., merge_idx), as well as the remaining application conditions listed in Table 17 above.

[0221] That is, the decoding device determines whether all the conditions listed in Table 17 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 17 is not met, the decoding device can not apply BDOF. The conditions in Table 17 can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0222] The method of making a true pair prediction based on the conditions listed in Table 16 or Table 17 above can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0223] Meanwhile, in merge / skip mode, motion information can be refined through MMVD, but in this case, since the decoder complexity increases, DMVR is not performed when MMVD is applied. However, when applied without considering MMVD, DMVR can be applied even without MMVD conditions to improve performance. In this case, according to one embodiment of this document, the DMVR application conditions can include the conditions presented in Table 18 below.

[0224] [Table 18]

[0225] Referring to Table 18, the condition of whether the MMVD mode is applicable (e.g., mmvd_flag) can be omitted from the conventional DMVR application conditions. That is, the process of determining whether mmvd_flag is 0 (i.e., when the MMVD mode is not used) can be omitted, and it can be determined whether the DMVR is applied based on the conditions listed in Table 17.

[0226] That is, the decoding device determines whether all the conditions listed in Table 18 (except for the condition for whether the MMVD mode is applicable (e.g., mmvd_flag)) are met, and if all the conditions are met, the decoding device can apply the DMVR and perform true pairwise prediction. If any one of the conditions listed in Table 18 is not met, the decoding device may not apply the DMVR. The conditions in Table 18 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0227] Also, contrary to the embodiment of Table 18 above, when refining motion information through MMVD in merge / skip mode, it is possible to determine whether to perform BDOF taking into account the decoder complexity. Therefore, this document proposes a method for determining whether to apply BDOF taking into account MMVD conditions. In this case, according to one embodiment of this document, the conditions for applying BDOF may include the conditions presented in Table 19 below.

[0228] [Table 19]

[0229] Referring to Table 19 above, by adding a condition for whether the MMVD mode is applicable (e.g., mmvd_flag), BDOF can be omitted when refining motion information through MMVD. For example, if mmvd_flag is 0 (i.e., if the MMVD mode is not used), it can be determined that the condition for whether to apply the MMVD mode is met.

[0230] Therefore, in this embodiment, it is possible to determine whether or not to apply BDOF based on the remaining application conditions listed in Table 19 above, along with the conditions for whether or not the MMVD mode is applicable (for example, mmvd_flag).

[0231] That is, the decoding device determines whether all the conditions listed in Table 19 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 19 is not met, the decoding device can not apply BDOF. The conditions in Table 19 can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0232] The method of making a true pair prediction based on the conditions listed in Table 18 or Table 19 above can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0233] Meanwhile, in the AMVP mode, the adaptive motion vector resolution (AMVR) technique can be applied. In this case, if the resolution of the motion vector is high, i.e., if there is integer-pel rounding or 4-sample rounding, it may not be suitable for application of this technique to BDOF, which refines within a limited area. Therefore, this document proposes a method for determining whether to perform BDOF according to the AMVR conditions. In this case, according to one embodiment of this document, the conditions for applying BDOF may include the conditions presented in Table 20 below.

[0234] [Table 20]

[0235] Referring to Table 20 above, it is possible to determine whether to apply BDOF by adding an AMVR condition (e.g., amvr_flag). Here, amvr_flag may be information indicating the resolution of the motion vector difference (MVD). For example, if amvr_flag is 0, it indicates that the MVD resolution is derived in quarter-luma-sample units, and if amvr_flag is not 0, it indicates that the MVD resolution is derived in integer-luma-sample units or four-luma-sample units. Alternatively, the opposite may be determined. Depending on the embodiment, as shown in Table 20 above, a condition for applying BDOF when amvr_flag is not 0 may be set. In other words, if amvr_flag is 0, BDOF may be restricted from being applied.

[0236] Therefore, in this embodiment, it is possible to determine whether to apply BDOF based on the AMVR condition (eg, amvr_flag) as well as the remaining conditions listed in Table 20 above.

[0237] That is, the decoding device determines whether all the conditions listed in Table 20 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 20 is not met, the decoding device can not apply BDOF. The conditions in Table 20 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0238] Meanwhile, the above-mentioned DMVR and BDOF can be signaled in the SPS (Sequence Parameter Set) syntax. Table 21 below shows an example of syntax elements that indicate whether DMVR and BDOF are possible, which are signaled via the SPS syntax.

[0239] [Table 21]

[0240] Referring to Table 21 above, sps_dmvr_enabled_flag can be signaled in the SPS syntax, and whether true vs. prediction-based DMVR is enabled can be indicated based on this syntax element. For example, if sps_dmvr_enabled_flag is 1, it indicates that true vs. prediction-based DMVR is enabled, and if sps_dmvr_enabled_flag is 0, it indicates that true vs. prediction-based DMVR is not enabled.

[0241] In addition, sps_bdof_enabled_flag can be signaled in the SPS syntax, and whether true vs. prediction-based BDOF is enabled can be indicated based on this syntax element. For example, if sps_bdof_enabled_flag is 1, it indicates that true vs. prediction-based BDOF is enabled, and if sps_bdof_enabled_flag is 0, it indicates that true vs. prediction-based BDOF is not enabled.

[0242] As shown in Table 21 above, the application conditions of DMVR and BDOF can be checked using a syntax element indicating whether DMVR is enabled (for example, sps_dmvr_enabled_flag) and a syntax element indicating whether BDOF is enabled (for example, sps_bdof_enabled_flag).

[0243] FIG. 10 is an example showing a method for checking the application conditions of DMVR and BDOF and performing the decoding process.

[0244] The method of Figure 10 can be applied when using a syntax element (e.g., sps_dmvr_enabled_flag) indicating whether DMVR is enabled or not and a syntax element (e.g., sps_bdof_enabled_flag) indicating whether BDOF is enabled or not, as shown in Table 21 above.

[0245] Referring to FIG. 10, a decoding device may derive motion information (eg, motion vectors, reference picture indexes, etc.) for a current block (S1000).

[0246] The decoding device can check the application conditions of the DMVR (S1010). At this time, the application conditions of the DMVR can be checked based on a syntax element (e.g., sps_dmvr_enabled_flag) that indicates whether the DMVR is enabled. For example, if the DMVR is enabled (e.g., if sps_dmvr_enabled_flag is 1), the application conditions of the DMVR can be checked.

[0247] The decoding device can determine whether to apply the DMVR process depending on whether the DMVR application conditions are met (S1020).

[0248] If all the conditions for applying DMVR are met, the decoding device may apply the DMVR process to derive refined motion information (S1030). If one or more of the conditions for applying DMVR are not met, the decoding device may not apply the DMVR process.

[0249] The decoding device can derive a predicted sample of the current block based on refined motion information derived when DMVR is applied or (unrefined) motion information derived when DMVR is not applied (S1040).

[0250] Then, the decoding device can check the application conditions of BDOF (S1050). At this time, the application conditions of BDOF can be checked based on a syntax element (e.g., sps_bdof_enabled_flag) that indicates whether BDOF is possible. For example, if BDOF is possible (e.g., if sps_bdof_enabled_flag is 1), the application conditions of BDOF can be checked.

[0251] If all the conditions for applying BDOF are met, the decoding device may apply the BDOF process to refine the predicted samples (S1070). If one or more of the conditions for applying BDOF are not met, the decoding device may not apply the BDOF process.

[0252] The decoding device derives residual samples for the current block (S1080), and can derive reconstructed samples based on the residual samples and refined prediction samples derived when the BDOF is applied or (unrefined) prediction samples derived when the BDOF is not applied (S1090).

[0253] As described above, this document proposes various embodiments that can improve coding efficiency and reduce complexity by harmonizing application conditions between DMVR and BDOF when applying them. When checking the application conditions of DMVR and BDOF according to the embodiments of this document and applying them to the decoding process, each condition can be checked and applied separately, but to improve coding efficiency, the application conditions can be checked at once. That is, this document proposes a method that can check the application conditions of DMVR and BDOF together at once.

[0254] In one embodiment of this document, the DMVR / BDOF application condition check process can be performed by signaling information (e.g., sps_refinement_enabled_flag) indicating whether refinement can be applied in a decoding device in the SPS (Sequence Parameter Set) syntax. Table 22 below shows an example of a syntax element (e.g., sps_refinement_enabled_flag) indicating whether refinement can be applied in a decoding device, which is signaled via the SPS syntax.

[0255] [Table 22]

[0256] Referring to Table 22 above, sps_refinement_enabled_flag can be signaled in the SPS syntax, and whether refinement application in the decoding device is possible can be indicated based on this syntax element. For example, when sps_refinement_enabled_flag exists (i.e., sps_refinement_enabled_flag is true), it can be determined that refinement application in the decoding device is possible. In this case, the decoding device can determine the application conditions of DMVR and BDOF by acquiring the sps_dmvr_enabled_flag syntax element indicating whether DMVR is enabled and the sps_bdof_enabled_flag syntax element indicating whether BDOF is enabled.

[0257] 11 and 12 are other examples showing a method for checking the application conditions of DMVR and BDOF and performing a decoding process.

[0258] The methods of Figures 11 and 12 can be applied when using a syntax element (e.g., sps_refinement_enabled_flag) that indicates whether refinement is applicable in the decoding device, as shown in Table 22. In Figures 11 and 12, descriptions of the same content as in Figure 10 will be omitted.

[0259] 11, compared to the process of FIG. 10, it can be seen that the DMVR and BDOF application conditions are checked at the front end at once, rather than being checked separately. As an example, in step S1110 of FIG. 11, the decoding device can check the refinement application conditions. At this time, the refinement application condition check can be performed based on sps_refinement_enabled_flag as shown in Table 22. For example, if sps_refinement_enabled_flag is 1, the decoding device can obtain the sps_dmvr_enabled_flag syntax element indicating whether DMVR is enabled or the sps_bdof_enabled_flag syntax element indicating whether BDOF is enabled, and check the DMVR and BDOF application conditions based on the obtained syntax element.

[0260] 12, compared to the process of FIG. 10, the DMVR and BDOF application conditions are checked at the front end at once instead of being checked separately, but a simple check process (BDOF application conditions) can be performed for other conditions. In one embodiment, in step S1210 of FIG. 12, the decoding device can check the refinement application conditions. Thereafter, in step S1250 of FIG. 12, the decoding device can perform an additional simple check for BDOF application conditions that have conditions different from the refinement application conditions.

[0261] On the other hand, if the length (Height) or size (Height * Width) of the current block is smaller than a specific length or size, the calculation rate of multiplication / addition for motion compensation increases. Therefore, in one embodiment of this document, to reduce the worst case, the application of BDOF can be limited to blocks with small lengths or small sizes, as in the application conditions of DMVR. In this case, the application conditions of BDOF can include the conditions presented in Table 23 below.

[0262] [Table 23]

[0263] Referring to Table 23, by adding a condition (e.g., CbHeight, CbWidth) related to the size of the current block, BDOF may not be applied to blocks smaller than a certain size. For example, a condition may be used that applies when the length (e.g., CbHeight) of the current block is 8 or greater, and a condition may be used that applies when the size (e.g., CbHeight*CbWidth) of the current block is 64 or greater. BDOF may be applied when such an application condition related to the size of the current block is met (i.e., when the length of the current block is 8 or greater and the length*width of the current block is 64 or greater), but may not be applied when such an application condition related to the size of the current block is not met.

[0264] In addition, in this embodiment, it is possible to determine whether to apply BDOF based on the remaining application conditions listed in Table 23 above, along with the conditions related to the size of the current block (for example, CbHeight, CbHeight*CbWidth).

[0265] That is, the decoding device determines whether all the conditions listed in Table 23 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 23 is not met, the decoding device can not apply BDOF. The conditions in Table 23 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0266] Meanwhile, when MMVD is applied to the current block, motion information can be derived based on information about the MMVD (e.g., mmvdindex). Here, the information about the MMVD can include an index of a base MV, a distance index, a direction index, etc. In particular, the distance index (more specifically, mmvd_distance_index[xCb][yCb]) is used to represent the distance from the base MV. For example, distance indexes 0 to 7 can be represented as {1 / 4, 1 / 2, 1, 2, 4, 8, 16, 32}, etc., respectively. When determining whether to refine motion information in DMVR and BDOF, neighboring pixels (neighboring samples) are taken into consideration. In this case, if the distance between the neighboring pixels and the base MV is large, the value of the distance index also becomes large. In such a case, it is difficult to consider neighboring pixels to improve the performance of DMVR and BDOF. Therefore, this document proposes a method that can determine whether to apply DMVR and BDOF depending on the distance index (more specifically, mmvd_distance_index[xCb][yCb]) value.

[0267] In one embodiment of this document, whether to apply DMVR can be determined by considering the distance index, and in this case, the conditions for applying DMVR can include the conditions presented in Table 24 below.

[0268] [Table 24]

[0269] Referring to Table 24 above, DMVR can be applied restrictively in MMVD mode by changing the conditions related to MMVD (e.g., mmvd_flag) among the conditions for applying DMVR. For example, if mmvd_flag is 1 and mmvd_distance_index is greater than 4, it can be determined that the distance index condition of MMVD is met. Therefore, when the MMVD mode is applied, it can be determined whether to apply DMVR depending on the distance index (e.g., mmvd_distance_index[xCb][yCb]) value.

[0270] Here, the threshold for mmvd_distance_index is set to 4, but this is just an example and can be changed to various values ​​depending on performance and coding efficiency.

[0271] Therefore, in this embodiment, whether to apply DMVR can be determined based on the conditions for whether MMVD is applicable (e.g., mmvd_flag), the conditions for the MMVD distance index (e.g., mmvd_distance_index), as well as the remaining application conditions listed in Table 24 above.

[0272] That is, the decoding device determines whether all the conditions listed in Table 24 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true pair prediction. If any one of the conditions listed in Table 24 is not met, the decoding device can not apply DMVR. The conditions in Table 24 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0273] Also, as an embodiment of this document, whether to apply BDOF can be determined by considering the distance index, and in this case, the conditions for applying BDOF can include the conditions presented in Table 25 below.

[0274] [Table 25]

[0275] Referring to Table 25 above, by adding a condition for whether merge mode / skip mode is possible (e.g., merge_flag) and a condition related to MMVD (e.g., mmvd_flag, mmvd_distance_index), BDOF can be applied in a limited manner in MMVD mode. For example, if merge_flag does not exist (i.e., merge_flag is not 1), or if merge_flag is 1, mmvd_flag is 1, and mmvd_distance_index is greater than 4, it can be determined that the conditions for applying BDOF in a limited manner in MMVD mode are met. Therefore, when MMVD mode is applied, it can be determined whether BDOF is applied depending on the distance index (e.g., mmvd_distance_index[xCb][yCb]) value.

[0276] Here, the threshold for mmvd_distance_index is set to 4, but this is just an example and can be changed to various values ​​depending on performance and coding efficiency.

[0277] Therefore, in this embodiment, whether to apply BDOF can be determined based on the conditions for whether merge mode / skip mode is possible (e.g., merge_flag), conditions related to MMVD (e.g., mmvd_flag, mmvd_distance_index), as well as the remaining application conditions listed in Table 25 above.

[0278] That is, the decoding device determines whether all the conditions listed in Table 25 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 25 is not met, the decoding device can not apply BDOF. The conditions in Table 25 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0279] The method of making a true pair prediction based on the conditions listed in Table 24 or Table 25 above can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0280] Meanwhile, as described above, a combined intra-inter prediction (CIIP) mode, which simultaneously performs intra prediction and inter prediction, can be applied to a current block. In this case, a predicted block (inter block) on which inter prediction is performed is combined with an intra prediction method to generate a final predicted sample value, thereby improving prediction accuracy. However, since DMVR and BDOF are techniques for refining inter blocks, the application of the CIIP mode may not be necessary in terms of complexity versus performance. Therefore, this document proposes a method for determining whether to apply DMVR and BDOF while taking CIIP into consideration.

[0281] In one embodiment of this document, whether to apply DMVR can be determined taking into consideration CIIP, and in this case, the conditions for applying DMVR may include the conditions presented in Table 26 below.

[0282] [Table 26]

[0283] Referring to Table 26, by adding a condition for whether the CIIP mode is applicable (e.g., ciip_flag), the DMVR can be applied restrictively depending on whether the CIIP mode is applicable. For example, if ciip_flag is 0 (i.e., the CIIP mode is not applicable), it is determined that the condition for whether the CIIP mode is applicable is met, and the DMVR can be applied.

[0284] Therefore, in this embodiment, it is possible to determine whether to apply DMVR based on the remaining application conditions listed in Table 26 above, along with the condition for whether CIIP mode is applicable (for example, ciip_flag).

[0285] That is, the decoding device determines whether all the conditions listed in Table 26 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true paired prediction. If any one of the conditions listed in Table 26 is not met, the decoding device can not apply DMVR. The conditions in Table 26 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0286] In addition, as an embodiment of this document, whether to apply BDOF can be determined taking CIIP into consideration, and in this case, the conditions for applying BDOF can include the conditions presented in Table 27 below.

[0287] [Table 27]

[0288] Referring to Table 27, by adding a condition for whether CIIP mode is applicable (e.g., ciip_flag), BDOF can be applied in a limited manner depending on whether CIIP mode is applicable. For example, if ciip_flag is 0 (i.e., CIIP mode is not applicable), it is determined that the condition for whether CIIP mode is applicable is met, and BDOF can be applied.

[0289] Therefore, in this embodiment, it is possible to determine whether or not to apply BDOF based on the remaining application conditions listed in Table 27 above, along with the condition as to whether or not the CIIP mode is applicable (for example, ciip_flag).

[0290] That is, the decoding device determines whether all the conditions listed in Table 27 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 27 is not met, the decoding device can not apply BDOF. The conditions in Table 27 can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0291] The method of making a true pair prediction based on the conditions listed in Table 26 or Table 27 above can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0292] The methods listed in Tables 6 to 27 according to the embodiments of this document can be applied in combination. That is, by checking the refinement application conditions, it can be determined whether to apply DMVR and BDOF, and the conditions shown in Table 28 below can be applied.

[0293] [Table 28]

[0294] Referring to Table 28, when DMVR or BDOF is applied, the following conditions can be used: a condition to be applied in the case of bidirectional prediction, a condition to be applied in the case of true paired prediction where the distance between the current picture and the bidirectional reference picture is the same, a condition to be applied when the MMVD mode is not used, a condition to be applied when the prediction mode is not affine, a condition to be applied when the sub-block based merge mode is not used, and a condition to be applied when the GBi index is default. In other words, it can be determined whether DMVR or BDOF is applied depending on whether the above conditions are met.

[0295] In addition, as described above, conditions can be added such as determining whether it is a merge mode for DMVR or whether it is a luma block for BDOF.

[0296] The application conditions listed in Table 28 above are just an example, and it is obvious that the various conditions listed in the above-described embodiments (embodiments of Tables 6 to 27 above) can be used in combination.

[0297] Meanwhile, DMVR adopts the SAD function as the cost function instead of the MRSAD (Mean-Removed SAD) function to consider decoding complexity. However, if the GBi index is not the default (e.g., GbiIdx is 0), two reference blocks may have different weighting factors, making DMVR using SAD undesirable. Therefore, the DMVR conditions may be fixed taking the GBi index into consideration. Experimental results showed a 0.00% RD-rate change at 100% encoding and decoding runtime compared to VTM4.0 (VVC Test Model).

[0298] In the case of VVC (Versatile Video Coding), the DMVR process can be performed when all the conditions listed in Table 29 below are met.

[0299] [Table 29]

[0300] The current DMVR can search for mismatched blocks by comparing the SAD of a reference block, from which a weighted average is derived later. In this document, since two reference blocks can have different weights, the DMVR conditions can be determined taking such cases into consideration. According to one embodiment of this document, DMVR can be prevented from being performed on blocks whose GBi index is not the default. In this case, the DMVR application conditions can be as shown in Table 30 below.

[0301] [Table 30]

[0302] As shown in Table 30 above, a condition for performing DMVR can be added when the value of the GBi index (e.g., GbiIdx) is 0. In other words, when the value of the GBi index (e.g., GbiIdx) is not 0, different weights are applied to two reference blocks (i.e., a reference block referenced for L0 prediction and a reference block referenced for L1 prediction), so in this case, DMVR can be restricted not to be performed.

[0303] Meanwhile, BDOF is performed when the current block (i.e., current coding unit; current CU) satisfies the true-to-prediction condition. Considering that the optical flow equation was designed to predict the movement of objects moving at a constant velocity (i.e., momentum), the current true-to-prediction condition is not the optimal condition for applying BDOF. Therefore, the BDOF condition can be fixed by taking into account the distance of the reference picture. Experimental results showed a 0.01% RD-rate change at 100% encoding and decoding runtime compared to VTM4.0 (VVC Test Model).

[0304] FIG. 13 is a diagram for explaining the concept of BDOF.

[0305] As mentioned above, BDOF was designed to improve the performance of motion compensation using the concept of optical flow. According to BDOF, it can be assumed that an object moves at a constant speed (constant motion) and that the brightness of each pixel does not change while the object moves, as shown in Figure 13. Under such assumptions, the mathematical formula for optical flow can be expressed as the following Equation 7.

[0306] <Formula 7>

number

[0307] As described above, BDOF is performed when the current CU satisfies the true-to-prediction condition. However, this true-to-prediction condition does not mean that the object moves at a constant speed. Therefore, this document proposes a method for applying BDOF when the object has a constant motion, thereby improving the performance of motion compensation.

[0308] According to one embodiment of this document, BDOF can be applied when the distance between the current picture and the L0 reference picture (Reference 0 in FIG. 13) and the distance between the current picture and the L1 reference picture (Reference 1 in FIG. 13) are equal to each other as δt in Equation 7. In this case, the BDOF application condition can be changed as shown in Table 31 below.

[0309] [Table 31]

[0310] Referring to Table 31 above, by changing the conditions related to the reference picture distance (e.g., DiffPicOrderCnt) among the BDOF application conditions, BDOF can be applied only when a certain motion speed is present. For example, by determining whether DiffPicOrderCnt(currPic, RefPicList[0][refIdxL0]) and DiffPicOrderCnt(RefPicList[1][refIdxL1], currPic) are the same, it can be determined whether the distance between the current picture and the L0 reference picture (i.e., the reference picture in the reference picture list L0) and the distance between the current picture and the L1 reference picture (i.e., the reference picture in the reference picture list L1) are the same. In other words, BDOF can be applied only when the distance between the L0 reference picture and the L1 reference picture based on the current picture is the same. In this way, by using the condition that the reference picture distances in both directions based on the current picture are the same, it is possible to determine whether a block is true versus predicted and contains an object moving at a constant speed. By applying BDOF to blocks that satisfy such conditions, it is possible to obtain improved motion information refinement results.

[0311] In addition, in this embodiment, it is possible to determine whether to apply BDOF based on the remaining application conditions listed in Table 31 above, along with the conditions related to the reference picture distance (for example, DiffPicOrderCnt).

[0312] That is, the decoding device determines whether all the conditions listed in Table 31 are met, and if all the conditions are met, it can apply BDOF to perform true paired prediction. If any one of the conditions listed in Table 31 is not met, the decoding device can not apply BDOF. The conditions in Table 31 can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0313] Meanwhile, this document proposes a method for determining whether to apply BDOF depending on the block size. Table 32 below shows a case where a block size restriction is included as an application condition according to one embodiment of this document.

[0314] [Table 32]

[0315] Referring to Table 32 above, when applying BDOF, if the length (e.g., CbHeight) of the current block is 8 or more, and if the size (length * width) (e.g., CbHeight * CbWidth) of the current block is 64 or more, a condition can be added.

[0316] Therefore, in this embodiment, whether to apply BDOF can be determined based on the conditions that the length of the current block is 8 or more, the length * width of the current block is 64 or more, and the remaining application conditions listed in Table 32 above.

[0317] That is, the decoding device determines whether all the conditions listed in Table 32 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 32 is not met, the decoding device can not apply BDOF. The conditions in Table 32 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0318] The DMVR process described above in this document can be implemented according to the specifications shown in Table 32 below. Table 33 below shows an example of a motion vector refinement process based on SAD as an embodiment of this document.

[0319] [Table 33-1]

[0320] [Table 33-2]

[0321] Meanwhile, as mentioned above, DMVR adopts the SAD function as the cost function instead of the MRSAD (Mean-Removed SAD) function in consideration of decoding complexity. However, if the GBi index is not the default (e.g., GbiIdx is 0) and the weighting flag for explicit weighted prediction is not 0, DMVR using SAD may be undesirable. Therefore, the DMVR conditions can be fixed by considering the GBi index and the weighting flag for explicit weighted prediction. The same conditions can also be applied to BDOF. Experimental results showed a 0.00% RD-rate change at 100% encoding and decoding runtime compared to VTM4.0 (VVC Test Model).

[0322] The current DMVR can search for mismatched blocks by comparing the SADs of reference blocks whose weighted averages are derived later. In this document, since two reference blocks can have different weights, the DMVR conditions can be determined taking such cases into consideration. According to one embodiment of this document, DMVR can be prevented for blocks whose GBi index is not the default. Also, DMVR can be prevented for blocks whose weight flag by explicit weighted prediction is not 0.

[0323] In one embodiment of this document, we propose a method for determining whether to apply DMVR by considering whether weighted pair prediction is possible. In this case, the conditions for applying DMVR include the conditions presented in Table 34 below.

[0324] [Table 34]

[0325] Referring to Table 34 above, by adding conditions (e.g., luma_weight_l0_flag, luma_weight_l1_flag) that explicitly indicate whether weighting can be applied to L0 prediction and L1 prediction, and index conditions (e.g., GbiIdx) that indicate the weighting to be applied to L0 prediction and L1 prediction, it is possible to determine whether DMVR is applied depending on whether weighted pair prediction is possible.

[0326] For example, it can be determined whether explicit weighted prediction is applied to the current block based on flag information (e.g., luma_weight_l0_flag) indicating whether weighting is explicitly applied to L0 prediction and flag information (e.g., luma_weight_l1_flag) indicating whether weighting is explicitly applied to L1 prediction. That is, when the value of luma_weight_l0_flag is 0 and the value of luma_weight_l1_flag is 0, DMVR can be applied in a limited manner. In other words, when weighted prediction is not explicitly applied to L0 and L1 prediction, it can be determined that DMVR is applied.

[0327] Furthermore, it can be determined whether pairwise predictions (i.e., L0 prediction and L1 prediction) using different weights are applied to the current block based on the value of a pairwise prediction weight index (e.g., GbiIdx) representing the weights applied to L0 prediction and L1 prediction. That is, when the value of the pairwise prediction weight index (e.g., GbiIdx) is 0, this may be the default case in which different weights are not applied to L0 prediction and L1 prediction, as described in the embodiment of Table 4 above. Therefore, when the value of the pairwise prediction weight index (e.g., GbiIdx) is 0, DMVR can be applied in a limited manner.

[0328] Depending on the embodiment, if weighted prediction is not explicitly applied to L0 and L1 prediction (if the value of luma_weight_l0_flag is 0 and the value of luma_weight_l1_flag is 0), pairwise prediction weight index (e.g., GbiIdx) information can be further obtained to determine whether the value of the pairwise prediction weight index (e.g., GbiIdx) is 0.

[0329] In this embodiment, whether or not DMVR is applied can be determined based on the conditions (e.g., luma_weight_l0_flag, luma_weight_l1_flag) indicating whether weighting can be explicitly applied to L0 prediction and L1 prediction, and the index conditions (e.g., GbiIdx) indicating the weighting to be applied to L0 prediction and L1 prediction, as well as the remaining conditions listed in Table 34 above.

[0330] That is, the decoding device determines whether all the conditions listed in Table 34 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true paired prediction. If any one of the conditions listed in Table 34 is not met, the decoding device can not apply DMVR. The conditions in Table 34 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0331] On the other hand, in the case of BDOF, the GBi index is taken into account, but the weight flag of explicit weighted prediction is not. Therefore, this paper proposes a method to determine whether to apply BDOF by considering the GBi index and the weight flag of explicit weighted prediction.

[0332] In one embodiment of this document, whether to apply BDOF can be determined by taking into account whether weighted pair prediction is possible. In this case, the conditions for applying BDOF can include the conditions presented in Table 35 below.

[0333] [Table 35]

[0334] Referring to Table 35 above, by adding conditions (e.g., luma_weight_l0_flag, luma_weight_l1_flag) that explicitly indicate whether weighting can be applied to L0 prediction and L1 prediction, it is possible to determine whether BDOF is applied depending on whether weighted pair prediction is possible.

[0335] For example, it can be determined whether explicit weighted prediction is applied to the current block based on flag information (e.g., luma_weight_l0_flag) indicating whether weighting is explicitly applied to L0 prediction and flag information (e.g., luma_weight_l1_flag) indicating whether weighting is explicitly applied to L1 prediction. That is, when the value of luma_weight_l0_flag is 0 and the value of luma_weight_l1_flag is 0, BDOF can be applied in a limited manner. In other words, when weighted prediction is not explicitly applied to L0 and L1 prediction, it can be determined that BDOF is applied.

[0336] In this embodiment, whether or not to apply BDOF can be determined based on the conditions (e.g., luma_weight_l0_flag, luma_weight_l1_flag) that explicitly indicate whether weighting can be applied to L0 prediction and L1 prediction, as well as the remaining conditions listed in Table 35 above.

[0337] That is, the decoding device determines whether all the conditions listed in Table 35 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 35 is not met, the decoding device can not apply BDOF. The conditions in Table 35 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0338] Based on the conditions listed in Table 34 or Table 35 above, the method of making a true pair prediction can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0339] As described above, whether to apply DMVR and BDOF can be determined by considering not only the GBi index but also the weighting factor of explicit weighted prediction. Here, to determine whether explicit weighted prediction is applicable, luma_weight_lX_flag (where X is 0 or 1) is used to determine whether weighted prediction is applicable to the luma component. However, it is also possible to determine whether weighted prediction is applicable to the chroma component. Therefore, this document proposes a method for determining whether to apply DMVR and BDOF by considering both the weighting factor of the luma component and the weighting factor of the chroma component.

[0340] As one embodiment of this document, whether to apply DMVR can be determined by considering the weighting factors of explicit weighted prediction for the luma and chroma components of the current block, and in this case, the conditions for applying DMVR may include the conditions presented in Table 36 below.

[0341] [Table 36]

[0342] Referring to Table 36 above, by adding conditions (e.g., luma_weight_l0_flag, luma_weight_l1_flag) indicating whether weighting can be explicitly applied to the luma component of L0 prediction (luma prediction value) and the luma component of L1 prediction (luma prediction value), conditions (e.g., chroma_weight_l0_flag, chroma_weight_l1_flag) indicating whether weighting can be explicitly applied to the chroma component of L0 prediction (chroma prediction value) and the chroma component of L1 prediction (chroma prediction value), and an index condition (e.g., GbiIdx) indicating the weighting to be applied to L0 prediction and L1 prediction, it is possible to restrictively apply DMVR only when weighting (i.e., weighting factor) is not explicitly applied to both the luma and chroma components.

[0343] For example, based on flag information (e.g., luma_weight_l0_flag) indicating whether to explicitly apply weighting to the luma component (luma prediction value) of L0 prediction and flag information (e.g., luma_weight_l1_flag) indicating whether to explicitly apply weighting to the luma component (luma prediction value) of L1 prediction, it is possible to determine whether a weighting factor for explicit weighted prediction is applied to the luma component of the current block.

[0344] In addition, based on flag information (e.g., chroma_weight_l0_flag) indicating whether to explicitly apply weighting to the chroma components (chroma prediction values) of L0 prediction and flag information (e.g., chroma_weight_l1_flag) indicating whether to explicitly apply weighting to the chroma components (chroma prediction values) of L1 prediction, it is possible to determine whether a weighting factor for explicit weighted prediction is applied to the chroma components of the current block.

[0345] That is, when the value of luma_weight_l0_flag is 0 and the value of luma_weight_l1_flag is 0, it can be determined that there is no explicit weighting factor for the luma component, and when the value of chroma_weight_l0_flag is 0 and the value of chroma_weight_l1_flag is 0, it can be determined that there is no explicit weighting factor for the chroma component. In this way, when there are no explicit weighting factors for both the luma component and the chroma component, DMVR can be applied in a limited manner.

[0346] Furthermore, it can be determined whether pairwise predictions (i.e., L0 prediction and L1 prediction) using different weights are applied to the current block based on the value of a pairwise prediction weight index (e.g., GbiIdx) representing the weights applied to L0 prediction and L1 prediction. That is, when the value of the pairwise prediction weight index (e.g., GbiIdx) is 0, this may be the default case in which different weights are not applied to L0 prediction and L1 prediction, as described in the embodiment of Table 4 above. Therefore, when the value of the pairwise prediction weight index (e.g., GbiIdx) is 0, DMVR can be applied in a limited manner.

[0347] Depending on the embodiment, when weighted prediction is not explicitly applied to L0 and L1 prediction of the luma component (when the value of luma_weight_l0_flag is 0 and the value of luma_weight_l1_flag is 0), and when weighted prediction is not explicitly applied to L0 and L1 prediction of the chroma component (when the value of chroma_weight_l0_flag is 0 and the value of chroma_weight_l1_flag is 0), pairwise prediction weight index (e.g., GbiIdx) information may be further obtained to determine whether the value of the pairwise prediction weight index (e.g., GbiIdx) is 0.

[0348] In this embodiment, whether or not DMVR is applied can be determined based on the conditions indicating whether or not weighting can be explicitly applied to L0 prediction and L1 prediction of the luma component (e.g., luma_weight_l0_flag, luma_weight_l1_flag), the conditions indicating whether or not weighting can be explicitly applied to L0 prediction and L1 prediction of the chroma component (e.g., chroma_weight_l0_flag, chroma_weight_l1_flag), the index conditions indicating the weighting to be applied to L0 prediction and L1 prediction (e.g., GbiIdx), as well as the remaining conditions listed in Table 36 above.

[0349] That is, the decoding device determines whether all the conditions listed in Table 36 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true pair prediction. If any one of the conditions listed in Table 36 is not met, the decoding device can not apply DMVR. The conditions in Table 36 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0350] In addition, as one embodiment of this document, whether to apply BDOF can be determined by taking into account the weighting factors of explicit weighted prediction for the luma and chroma components of the current block, and in this case, the conditions for applying BDOF can include the conditions presented in Table 37 below.

[0351] [Table 37]

[0352] Referring to Table 37 above, by adding conditions (e.g., luma_weight_l0_flag, luma_weight_l1_flag) indicating whether weighting can be explicitly applied to the luma component (luma prediction value) of L0 prediction and the luma component (luma prediction value) of L1 prediction, and conditions (e.g., chroma_weight_l0_flag, chroma_weight_l1_flag) indicating whether weighting can be explicitly applied to the chroma component (chroma prediction value) of L0 prediction and the chroma component (chroma prediction value) of L1 prediction, it is possible to apply BDOF in a limited manner only when weighting (i.e., weighting factor) is not explicitly applied to both the luma and chroma components.

[0353] For example, if the value of flag information (e.g., luma_weight_l0_flag) indicating whether to explicitly apply weighting to the luma component (luma prediction value) of L0 prediction is 0 and the value of flag information (e.g., luma_weight_l1_flag) indicating whether to explicitly apply weighting to the luma component (luma prediction value) of L1 prediction is 0, it can be determined that there are no explicit weighting factors for L0 prediction and L1 prediction for the luma component of the current block.

[0354] Furthermore, if the value of flag information (e.g., chroma_weight_l0_flag) indicating whether to explicitly apply weighting to the chroma components (chroma prediction values) of L0 prediction is 0 and the value of flag information (e.g., chroma_weight_l1_flag) indicating whether to explicitly apply weighting to the chroma components (chroma prediction values) of L1 prediction is 0, it can be determined that there are no explicit weighting factors for L0 prediction and L1 prediction for the chroma components of the current block.

[0355] In this way, when there are no weighting factors for both the luma component and the chroma component in L0 prediction and L1 prediction, BDOF can be applied in a limited manner.

[0356] In this embodiment, whether or not to apply BDOF can be determined based on the conditions indicating whether or not weighting can be explicitly applied to L0 prediction and L1 prediction of the luma component (e.g., luma_weight_l0_flag, luma_weight_l1_flag), the conditions indicating whether or not weighting can be explicitly applied to L0 prediction and L1 prediction of the chroma component (e.g., chroma_weight_l0_flag, chroma_weight_l1_flag), as well as the remaining conditions listed in Table 37 above.

[0357] That is, the decoding device determines whether all the conditions listed in Table 37 are met, and if all the conditions are met, it can apply BDOF to perform true pairwise prediction. If any one of the conditions listed in Table 37 is not met, the decoding device can not apply BDOF. The conditions in Table 37 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0358] Based on the conditions listed in Table 36 above or Table 37 above, the method of making a true pair prediction can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0359] Meanwhile, as described above, according to the embodiment of the present document, it is possible to determine whether to apply DMVR and BDOF by taking explicit weighted prediction into consideration. At this time, the type of slice can be taken into consideration to determine whether explicit weighted prediction can be applied. Therefore, this document proposes a method for determining whether to apply DMVR and BDOF by taking into consideration the type of slice and whether weighted prediction corresponding to the slice can be applied.

[0360] In one embodiment of this document, whether DMVR and BDOF are applied can be determined using flag information indicating whether weighted prediction is applicable according to the type of the current slice. Here, the flag information indicating whether weighted prediction is applicable according to the type of the current slice can be signaled from the encoding device to the decoding device via a PPS (Picture Parameter Set) or SPS (Sequence Parameter Set) syntax. As an example, Table 38 below shows the flag information signaled via the PPS syntax.

[0361] [Table 38]

[0362] Referring to Table 38, weighted_pred_flag and weighted_bipred_flag can be signaled from the encoding device to the decoding device, where weighted_pred_flag may be information indicating whether weighted prediction is applied to a P slice, and weighted_bipred_flag may be information indicating whether weighted prediction is applied to a B slice.

[0363] For example, a value of weighted_pred_flag of 0 may indicate that weighted prediction is not applied to a P slice, and a value of weighted_pred_flag of 1 may indicate that weighted prediction is applied to a P slice. Also, a value of weighted_bipred_flag of 0 may indicate that weighted prediction is not applied to a B slice, and a value of weighted_bipred_flag of 1 may indicate that weighted prediction is applied to a B slice.

[0364] Here, a P slice (Predictive slice) may refer to a slice decoded based on inter-prediction using one motion vector and a reference picture index, and a B slice (bi-predictive slice) may refer to a slice decoded based on inter-prediction using one or more, for example, two, motion vectors and reference picture indexes.

[0365] In one embodiment of this document, whether to apply DMVR can be determined based on flag information (e.g., weighted_pred_flag) indicating whether weighted prediction is applied to a P slice and flag information (e.g., weighted_bipred_flag) indicating whether weighted prediction is applied to a B slice, in which case the conditions for applying DMVR may include the conditions presented in Table 39 below.

[0366] [Table 39]

[0367] Referring to Table 39 above, by adding a condition indicating whether weighted prediction is applied to a P slice (e.g., weighted_pred_flag), a condition indicating whether weighted prediction is applied to a B slice (e.g., weighted_bipred_flag), and an index condition indicating the weighting applied to L0 prediction and L1 prediction (e.g., GbiIdx), it is possible to restrictively apply DMVR only when weighted prediction is not applied to P slices and B slices.

[0368] For example, DMVR can be applied when the value of weighted_pred_flag is 1 and the slice is not a P slice, or when the value of weighted_bipred_flag is 1 and the slice is not a B slice. In other words, when weighted prediction is not applied to a P slice and weighted prediction is not applied to a B slice, it can be determined that DMVR is applied.

[0369] Furthermore, it can be determined whether pairwise predictions (i.e., L0 prediction and L1 prediction) using different weights are applied to the current block based on the value of a pairwise prediction weight index (e.g., GbiIdx) representing the weights applied to L0 prediction and L1 prediction. That is, when the value of the pairwise prediction weight index (e.g., GbiIdx) is 0, this may be the default case in which different weights are not applied to L0 prediction and L1 prediction, as described in the embodiment of Table 4 above. Therefore, when the value of the pairwise prediction weight index (e.g., GbiIdx) is 0, DMVR can be applied in a limited manner.

[0370] In this embodiment, whether to apply DMVR can be determined based on the remaining conditions listed in Table 39 above, along with a condition indicating whether weighted prediction is applied to a P slice (e.g., weighted_pred_flag), a condition indicating whether weighted prediction is applied to a B slice (e.g., weighted_bipred_flag), and an index condition indicating the weighting applied to L0 prediction and L1 prediction (e.g., GbiIdx).

[0371] That is, the decoding device determines whether all the conditions listed in Table 39 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true pair prediction. If any one of the conditions listed in Table 39 is not met, the decoding device can not apply DMVR. The conditions in Table 39 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0372] Also, as one embodiment of this document, whether to apply BDOF can be determined based on flag information (e.g., weighted_pred_flag) indicating whether weighted prediction is applied to a P slice and flag information (e.g., weighted_bipred_flag) indicating whether weighted prediction is applied to a B slice, in which case the conditions for applying BDOF may include the conditions presented in Table 40 below.

[0373] [Table 40]

[0374] Referring to Table 40 above, by adding a condition indicating whether weighted prediction is applied to a P slice (e.g., weighted_pred_flag) and a condition indicating whether weighted prediction is applied to a B slice (e.g., weighted_bipred_flag), BDOF can be applied in a limited manner only when weighted prediction is not applied to P slices and B slices.

[0375] For example, if the value of weighted_pred_flag is 1 and the slice is not a P slice, or if the value of weighted_bipred_flag is 1 and the slice is not a B slice, BDOF can be applied. In other words, if weighted prediction is not applied to a P slice and weighted prediction is not applied to a B slice, it can be determined that BDOF is applied.

[0376] In this embodiment, whether to apply BDOF can be determined based on the remaining conditions listed in Table 40 above, along with the condition indicating whether weighted prediction is applied to a P slice (e.g., weighted_pred_flag) and the condition indicating whether weighted prediction is applied to a B slice (e.g., weighted_bipred_flag).

[0377] That is, the decoding device determines whether all the conditions listed in Table 40 are met, and if all the conditions are met, it can apply BDOF to perform true paired prediction. If any one of the conditions listed in Table 40 is not met, the decoding device can not apply BDOF. The conditions in Table 40 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0378] The method of making a true vs. prediction based on the conditions listed in Table 39 or Table 40 above can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0379] Meanwhile, whether to perform bi-prediction or uni-prediction can be determined depending on the block size. For example, if the size of the current block is small (e.g., a block of 4x8 or 8x4 size), only uni-prediction may be possible, while bi-prediction may be enabled if the size of the current block is large. In this case, if only uni-prediction is possible when the block size is small, the DMVR and BDOF performed in the case of bi-prediction must also take into account the block size limited in the case of uni-prediction. For example, if the size of the current block is 4x8 or 8x4 and only uni-prediction is possible, bi-prediction may be performed if at least one of the height or width of the current block is greater than 4. Therefore, this document proposes a method of applying DMVR and BDOF taking into account the block size applied in bi-prediction or uni-prediction.

[0380] In one embodiment of this document, we propose a method for determining whether to apply BDOF by considering the height (and / or width) of the current block. In this case, the conditions for applying BDOF may include the conditions presented in Table 41 below.

[0381] [Table 41]

[0382] Referring to Table 41 above, by changing the condition (e.g., cbHeight) related to the height (and / or width) of the current block, BDOF can be applied restrictively only to blocks larger than a certain size. For example, a condition for applying BDOF can be used when the height (e.g., CbHeight) of the current block is greater than 4. Alternatively, a condition for applying BDOF can be used when the width (e.g., CbWidth) of the current block is greater than 4. BDOF can be applied when such a condition related to the size (e.g., CbHeight or CbWidth) of the current block is met (i.e., when the height of the current block is greater than 4), and not applied when such a condition related to the size of the current block is not met (i.e., when the height of the current block is 4 or less).

[0383] In addition, in this embodiment, it can be determined whether to apply BDOF based on the remaining conditions listed in Table 41, along with conditions related to the size of the current block (e.g., CbHeight, CbWidth). At this time, bdofFlag can be derived depending on whether the conditions listed in Table 41 are met. At this time, if all the conditions listed in Table 41 are met, the value of bdofFlag can be derived as 1 (True), and if not, the value of bdofFlag can be derived as 0 (False). Here, bdofFlag can be flag information indicating whether BDOF is applied to the current block.

[0384] That is, the decoding device determines whether all the conditions listed in Table 41 are met, and if all the conditions are met, it can apply BDOF to perform true paired prediction. If any one of the conditions listed in Table 41 is not met, the decoding device can not apply BDOF. The conditions in Table 41 can also be applied to the encoding device, and the encoding device can perform the same method as the decoding device.

[0385] In addition, as an embodiment of this document, a method is proposed in which whether to apply DMVR is determined taking into account the height (and / or width) of the current block. In this case, the conditions for applying DMVR may include the conditions presented in Table 42 below.

[0386] [Table 42]

[0387] Referring to Table 42 above, by changing the condition (e.g., cbHeight) related to the height (and / or width) of the current block, DMVR can be applied restrictively only to blocks larger than a certain size. For example, a condition for applying DMVR can be used when the height (e.g., CbHeight) of the current block is greater than 4. Alternatively, a condition for applying DMVR can be used when the width (e.g., CbWidth) of the current block is greater than 4. If such a condition related to the size (e.g., CbHeight or CbWidth) of the current block is met (i.e., when the height of the current block is greater than 4), DMVR can be applied, and if such a condition related to the size of the current block is not met (i.e., when the height of the current block is 4 or less), DMVR can be not applied.

[0388] In addition, in this embodiment, it can be determined whether to apply DMVR based on the remaining conditions listed in Table 42, along with conditions related to the size of the current block (e.g., CbHeight, CbWidth). At this time, dmvrFlag can be derived depending on whether the conditions listed in Table 42 are satisfied. At this time, if all the conditions listed in Table 42 are satisfied, the value of dmvrFlag can be derived as 1 (True), and if not, the value of dmvrFlag can be derived as 0 (False). Here, dmvrFlag can be flag information indicating whether DMVR is applied to the current block.

[0389] That is, the decoding device determines whether all the conditions listed in Table 42 are met, and if all the conditions are met, the decoding device can apply DMVR to perform true pair prediction. If any one of the conditions listed in Table 42 is not met, the decoding device can not apply DMVR. The conditions in Table 42 can also be applied to the encoding device, and the encoding device can perform the same in a manner corresponding to that of the decoding device.

[0390] The method of making a true pair prediction based on the conditions listed in Table 41 or Table 42 above can be applied independently to DMVR and BDOF, or can be applied under the same conditions for DMVR and BDOF.

[0391] This document can derive DMVR flag information indicating whether DMVR is applicable or not and BDOF flag information indicating whether BDOF is applicable or not based on the application conditions of Tables 2 to 42 described above.

[0392] For example, DMVR flag information (e.g., dmvrFlag) can be derived based on the DMVR application conditions (e.g., at least one of the application conditions in Tables 2 to 42 or a combination of the application conditions). At this time, if the value of dmvrFlag is 1 (or true), it indicates that DMVR is applied, and if the value of dmvrFlag is 0 (or false), it indicates that DMVR is not applied. Also, BDOF flag information (e.g., bdofFlag) can be derived based on the BDOF application conditions (e.g., at least one of the application conditions in Tables 2 to 42 or a combination of the application conditions). At this time, if the value of bdofFlag is 1 (or true), it indicates that BDOF is applied, and if the value of bdofFlag is 0 (or false), it indicates that BDOF is not applied.

[0393] FIG. 14 is a flow chart that schematically illustrates an encoding method that can be performed by an encoding device according to an embodiment of the present document.

[0394] The method disclosed in FIG. 14 may be performed by the encoding device 200 disclosed in FIG. 2. Specifically, steps S1400 to S1430 of FIG. 14 may be performed by the prediction unit 220 and the inter prediction unit 221 disclosed in FIG. 2, step S1440 of FIG. 14 may be performed by the residual processing unit 230 disclosed in FIG. 2, and step S1450 of FIG. 14 may be performed by the entropy encoding unit 240 disclosed in FIG. 2. In addition, the method disclosed in FIG. 14 may include the embodiments described in detail in this document. Therefore, in FIG. 14, detailed descriptions of content that overlaps with the above-described embodiments will be omitted or simplified.

[0395] Referring to FIG. 14, the encoding apparatus may derive an L0 motion vector and an L1 motion vector for the current block (S1400).

[0396] In one embodiment, when inter-prediction is performed on a current block, the encoding apparatus may derive motion information (such as a motion vector and a reference picture index) of the current block. For example, the encoding apparatus may search for blocks similar to the current block within a certain region (search region) of a reference picture through motion estimation, and derive a reference block whose difference from the current block is minimum or equal to or less than a certain criterion. Based on this, the encoding apparatus may derive a reference picture index indicating the reference picture in which the reference block is located, and derive a motion vector based on the positional difference between the reference block and the current block.

[0397] In addition, the encoding apparatus can determine an inter prediction mode to be applied to the current block from among various prediction modes, and can compare RD costs for various prediction modes to determine the optimal prediction mode for the current block.

[0398] For example, the encoding device may determine whether to apply a merge mode as an optimal prediction mode for the current block. If the encoding device applies a merge mode to the current block, the encoding device may construct a merge candidate list based on neighboring blocks of the current block and generate merge index information. Specifically, the encoding device may derive a reference block whose difference from the current block is minimum or equal to or less than a certain criterion among reference blocks (i.e., neighboring blocks) pointed to by merge candidates included in the merge candidate list. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information pointing to the selected merge candidate may be generated and signaled to the decoding device. Motion information of the current block may be derived using motion information of the selected merge candidate.

[0399] Here, the motion information may include information such as a motion vector and a reference picture index, and may include L0 motion information and / or L1 motion information depending on the inter-prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). When pairwise prediction (Bi prediction) is applied to the current block, the motion information may include a motion vector in the L0 direction (L0 motion vector) and a motion vector in the L1 direction (L1 motion vector). The motion information may also include an L0 reference picture index and an L0 reference picture indicated by the L0 reference picture index in the L0 reference picture list, and an L1 reference picture indicated by the L1 reference picture index in the L1 reference picture list.

[0400] That is, when the merge mode is applied, the encoding device derives the L0 motion vector and L1 motion vector of the surrounding block of the current block indicated by the merge index information, and can use these as the L0 and L1 motion vectors of the current block.

[0401] The encoding device can apply DMVR to the current block based on DMVR flag information indicating whether DMVR is applied to the current block, and derive refined L0 motion vectors and refined L1 motion vectors based on the L0 motion vectors and L1 motion vectors (S1410).

[0402] In this case, the encoding apparatus may determine whether to apply DMVR to the current block in consideration of coding efficiency, complexity, prediction performance, etc. That is, the encoding apparatus may apply DMVR to the current block based on whether a predetermined DMVR application condition for the current block is satisfied. Here, the predetermined DMVR application condition may be some (or all) or a specific combination of the various application conditions described in Tables 2 to 42. In addition, the encoding apparatus may derive DMVR flag information based on whether the predetermined DMVR application condition is satisfied. The DMVR flag information may be information indicating whether DMVR is applied to the current block (e.g., the above-mentioned dmvrFlag).

[0403] In one embodiment, the pre-set DMVR application condition may include a condition that the values ​​of the L0 luma weighted prediction flag information and the L1 luma weighted prediction flag information are all zero.

[0404] In this case, when the case where the values ​​of L0 luma weighted prediction flag information (e.g., luma_weight_l0_flag) and L1 luma weighted prediction flag information (e.g., luma_weight_l1_flag) are all 0 is satisfied, the encoding device can derive the DMVR flag information to a value representing that DMVR is applied to the current block. For example, if luma_weight_l0_flag and luma_weight_l1_flag are all 0, the DMVR flag information can be derived as a value (e.g., 1 or true) representing that DMVR is applied to the current block. Otherwise, the DMVR flag information can be derived as a value (e.g., 0 or false) representing that DMVR is not applied to the current block.

[0405] Here, the L0 luma weighted prediction flag information may be information indicating whether a weighting factor for L0 prediction is present for the luma component of the current block, and the L1 luma weighted prediction flag information may be information indicating whether a weighting factor for L1 prediction is present for the luma component of the current block. For example, a value of 0 in the L0 luma weighted prediction flag information may indicate that a weighting factor is not present for the luma component of L0 prediction, and a value of 1 in the L0 luma weighted prediction flag information may indicate that a weighting factor is present for the luma component of L0 prediction. Furthermore, a value of 0 in the L1 luma weighted prediction flag information may indicate that a weighting factor is not present for the luma component of L1 prediction, and a value of 1 in the L1 luma weighted prediction flag information may indicate that a weighting factor is present for the luma component of L1 prediction.

[0406] In addition, as an example, the pre-set DMVR application condition may include a condition that the value of bi-prediction weight index information of the current block is 0.

[0407] In this case, when the value of the paired prediction weighting index information (e.g., GbiIdx) is 0, the encoding device can derive the DMVR flag information to a value indicating that DMVR is applied to the current block. For example, if GbiIdx is 0, the DMVR flag information can be derived as a value (e.g., 1 or true) indicating that DMVR is applied to the current block. Otherwise, the DMVR flag information can be derived as a value (e.g., 0 or false) indicating that DMVR is not applied to the current block.

[0408] Here, when the value of the pair prediction weighting index information (e.g., GbiIdx) is 0, it may be a default case in which different weights are not applied to the L0 prediction and the L1 prediction. For example, as shown in Table 4, when the value of the pair prediction weighting index information is 0, it may be a case in which 1 / 2 weighting is applied to each of the L0 prediction and the L1 prediction.

[0409] In addition, as one embodiment, the pre-set DMVR application condition may include a condition that a prediction mode combining inter prediction and intra prediction (CIIP mode) is not applied to the current block.

[0410] In this case, when a prediction mode in which inter prediction and intra prediction are combined is not applied to the current block (e.g., when ciip_flag is 0), the encoding apparatus may derive DMVR flag information as a value indicating that DMVR is applied to the current block. For example, when ciip_flag is 0, the DMVR flag information may be derived as a value (e.g., 1 or true) indicating that DMVR is applied to the current block. When ciip_flag is 1, the DMVR flag information may be derived as a value (e.g., 0 or false) indicating that DMVR is not applied to the current block.

[0411] In addition, as an example, the pre-set DMVR application condition may include a condition that the size of the current block is 16x16 or more, i.e., the height of the current block is 16 or more and the width of the current block is 16 or more.

[0412] In this case, when the size of the current block is 16x16 or more, the encoding apparatus may derive the DMVR flag information to a value indicating that the DMVR is applied to the current block. For example, if the size of the current block is 16x16 or more, the DMVR flag information may be derived as a value (e.g., 1 or true) indicating that DMVR is applied to the current block. Otherwise (i.e., if the size of the current block is smaller than 16x16), the DMVR flag information may be derived as a value (e.g., 0 or false) indicating that DMVR is not applied to the current block.

[0413] In addition, according to an embodiment, the pre-set DMVR application conditions may include a condition when DMVR-based inter bi-prediction is enabled, a condition when bi-prediction based on an L0 reference picture and an L1 reference picture is applied to the current block, a condition when a first POC (Picture Order Count) difference between the current picture and the L0 reference picture is the same as a second POC difference between the current picture and the L1 reference picture, a condition when a merge mode is applied to the current block, a condition when an MMVD (Merge Mode With Motion Vector Difference) mode is not applied to the current block, etc.

[0414] That is, the DMVR application conditions may be determined to include at least one of the above-described conditions, and the DMVR flag information may be derived based on the determined DMVR application conditions. In this case, if the DMVR application conditions are determined to include one or more conditions, the encoding apparatus may determine to apply the DMVR to the current block by deriving the value of the DMVR flag information as true or 1 if all of the preset DMVR application conditions are satisfied. Otherwise, if any of the preset DMVR application conditions is not satisfied, the encoding apparatus may derive the value of the DMVR flag information as false or 0.

[0415] Here, in deriving the DMVR flag information, the DMVR application conditions listed above are merely examples, and the conditions in Tables 2 to 42 can be used in various combinations.

[0416] If the DMVR flag information indicates that DMVR is to be applied to the current block (for example, if the DMVR flag information is derived as a value of true or 1), the encoding device can apply DMVR to the current block and derive refined L0 motion vectors and refined L1 motion vectors based on the L0 motion vectors and L1 motion vectors.

[0417] In one embodiment, the encoding device may calculate a minimum SAD (Sum of Absolute Differences) using reference samples in the L0 reference picture derived based on the L0 motion vector and reference samples in the L1 reference picture derived based on the L1 motion vector, and may derive a refined L0 motion vector for the L0 motion vector and a refined L1 motion vector for the L1 motion vector based on the sample positions corresponding to the minimum SAD.

[0418] Here, an L0 reference picture may be a reference picture indicated by an L0 reference picture index among the reference pictures included in the L0 reference picture list, and an L1 reference picture may be a reference picture indicated by an L1 reference picture index among the reference pictures included in the L1 reference picture list.

[0419] In addition, the refined L0 motion vector may be a motion vector that points to a sample position corresponding to the minimum SAD in the L0 reference picture, and the refined L1 motion vector may be a motion vector that points to a sample position corresponding to the minimum SAD in the L1 reference picture. The process of deriving a refined motion vector by applying DMVR has been described in detail in Figures 8 and 9, so a description thereof will be omitted here.

[0420] The encoding apparatus may derive an L0 predicted sample of the current block based on the refined L0 motion vector and an L1 predicted sample of the current block based on the refined L1 motion vector (S1420).

[0421] Here, the L0 prediction sample can be derived based on a reference sample indicated by a refined L0 motion vector in the L0 reference picture, and the L1 prediction sample can be derived based on a reference sample indicated by a refined L1 motion vector from the L1 reference picture.

[0422] The encoding apparatus may apply BDOF to the current block based on BDOF flag information indicating whether BDOF is applied to the current block, and derive a prediction sample for the current block based on the L0 prediction sample and the L1 prediction sample (S1430).

[0423] In deriving the predicted samples, the encoding apparatus may determine whether to apply BDOF to the current block in consideration of coding efficiency, complexity, prediction performance, etc. That is, the encoding apparatus may apply BDOF to the current block based on whether a predetermined BDOF application condition for the current block is satisfied. Here, the predetermined BDOF application condition may be some (or all) or a specific combination of the various application conditions described in Tables 2 to 42 above. Furthermore, the encoding apparatus may derive BDOF flag information based on whether the predetermined BDOF application condition is satisfied. The BDOF flag information may be information indicating whether BDOF is applied to the current block (for example, the above-mentioned bdofFlag).

[0424] In one embodiment, the pre-set BDOF application condition may include a condition that the values ​​of the L0 luma weighted prediction flag information and the L1 luma weighted prediction flag information are all zero.

[0425] In this case, when the values ​​of L0 luma weighted prediction flag information (e.g., luma_weight_l0_flag) and L1 luma weighted prediction flag information (e.g., luma_weight_l1_flag) are all 0, the encoding device can derive the BDOF flag information to a value indicating that BDOF is applied to the current block. For example, if luma_weight_l0_flag and luma_weight_l1_flag are all 0, the BDOF flag information can be derived as a value (e.g., 1 or true) indicating that BDOF is applied to the current block. Otherwise, the BDOF flag information can be derived as a value (e.g., 0 or false) indicating that BDOF is not applied to the current block.

[0426] In addition, as an embodiment, the preset BDOF application condition may include a condition that the size of the current block is 16x16 or more, in other words, the height of the current block is 16 or more and the width of the current block is 16 or more.

[0427] In this case, when the size of the current block is 16x16 or more, the encoding device may derive BDOF flag information to a value indicating that BDOF is applied to the current block. For example, if the size of the current block is 16x16 or more, the BDOF flag information may be derived as a value (e.g., 1 or true) indicating that BDOF is applied to the current block. Otherwise (i.e., if the size of the current block is smaller than 16x16), the BDOF flag information may be derived as a value (e.g., 0 or false) indicating that BDOF is not applied to the current block.

[0428] In addition, as an embodiment, the preset BDOF application condition may include a condition that the height of the current block is 8 or more.

[0429] In this case, when the height of the current block is equal to or greater than 8, the encoding device can derive the BDOF flag information to a value representing that BDOF is applied to the current block. For example, if the height of the current block is equal to or greater than 8, the BDOF flag information can be derived as a value (e.g., 1 or true) representing that BDOF is applied to the current block. Otherwise (i.e., if the height of the current block is less than 8), the BDOF flag information can be derived as a value (e.g., 0 or false) representing that BDOF is not applied to the current block.

[0430] In one embodiment, the pre-set BDOF application conditions may include a condition where a first POC (Picture Order Count) difference between the current picture and the L0 reference picture is the same as a second POC difference between the current picture and the L1 reference picture.

[0431] In this case, when the first POC difference (e.g., DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])) and the second POC difference (e.g., DiffPicOrderCnt(currPic, RefPicList1[refIdxL1])) are the same, the encoding device may derive BDOF flag information to a value indicating that BDOF is applied to the current block. For example, if DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1]) is 0, the BDOF flag information may be derived as a value (e.g., 1 or true) indicating that BDOF is applied to the current block. Otherwise, the BDOF flag information may be derived as a value (e.g., 0 or false) indicating that BDOF is not applied to the current block.

[0432] In addition, as an embodiment, the pre-set BDOF application conditions may include a condition for when a prediction mode combining inter prediction and intra prediction (CIIP mode) is not applied to the current block.

[0433] In this case, when a prediction mode in which inter prediction and intra prediction are combined is not applied to the current block (e.g., when ciip_flag is 0), the encoding device may derive BDOF flag information to a value indicating that BDOF is applied to the current block. For example, when ciip_flag is 0, the BDOF flag information may be derived as a value (e.g., 1 or true) indicating that BDOF is applied to the current block. When ciip_flag is 1, the BDOF flag information may be derived as a value (e.g., 0 or false) indicating that BDOF is not applied to the current block.

[0434] In addition, according to an embodiment, the pre-set BDOF application conditions may include a condition when BDOF-based inter-prediction is enabled, a condition when bi-prediction based on an L0 reference picture and an L1 reference picture is applied to the current block, a condition when the value of the bi-prediction weight index information of the current block is 0, a condition when affine mode is not applied to the current block, a condition when sub-block-based merge mode is not applied to the current block, and a condition when the current block is a luma component.

[0435] That is, the BDOF application condition may be determined to include at least one of the above-described conditions, and the BDOF flag information may be derived based on the determined BDOF application condition. In this case, if the BDOF application condition is determined to include one or more conditions, the encoding device may determine that the BDOF is applied to the current block by deriving the value of the BDOF flag information as true or 1 if all of the preset BDOF application conditions are met. Otherwise, if any of the preset BDOF application conditions is not met, the encoding device may derive the value of the BDOF flag information as false or 0.

[0436] Here, in deriving the BDOF flag information, the BDOF application conditions listed above are merely examples, and the conditions in Tables 2 to 42 can be used in various combinations.

[0437] If the BDOF flag information indicates that BDOF is to be applied to the current block (for example, if the BDOF flag information is derived as true or a value of 1), the encoding device can apply BDOF to the current block to derive a predicted sample.

[0438] In one embodiment, the encoding apparatus may calculate a first gradient for the L0 prediction sample derived based on the refined L0 motion vector and a second gradient for the L1 prediction sample derived based on the refined L1 motion vector. Then, the encoding apparatus may finally derive a prediction sample for the current block based on the L0 prediction sample, the L1 prediction sample, the first gradient, and the second gradient. For example, the process of deriving a prediction sample by applying BDOF may use the calculations shown in Equations 1 to 6 above.

[0439] The encoding apparatus may derive residual samples for the current block based on the predicted samples (S1440) and encode video information including information about the residual samples (S1450).

[0440] That is, the encoding apparatus may derive residual samples based on original samples for a current block and predicted samples of the current block, and may generate information about the residual samples, including information about the values, positions, transform techniques, transform kernels, quantization parameters, and the like of quantized transform coefficients derived by transforming and quantizing the residual samples.

[0441] The encoding device encodes information about the residual samples and outputs it in a bitstream, which can be transferred to the decoding device via a network or storage medium.

[0442] The encoding device may also encode the video information derived from steps S1400 to S1450 and output the encoded video information as a bitstream. For example, merge flag information, merge index information, L0 reference picture index, L1 reference picture index, L0 luma weighted prediction flag information, L1 luma weighted prediction flag information, pair prediction weight index information, etc. may be included in the video information and encoded, and the encoded video information may be signaled to a decoding device.

[0443] FIG. 15 is a flow chart that schematically illustrates a decoding method that can be performed by a decoding device according to an embodiment of the present document.

[0444] The method disclosed in FIG. 15 may be performed by the decoding device 300 disclosed in FIG. 3. Specifically, steps S1500 to S1530 of FIG. 15 may be performed by the prediction unit 330 and the inter prediction unit 332 disclosed in FIG. 3, and step S1540 of FIG. 15 may be performed by the addition unit 340 disclosed in FIG. 3. In addition, the method disclosed in FIG. 15 may include the embodiments described in detail in this document. Therefore, in FIG. 15, detailed descriptions of content that overlaps with the above-described embodiments will be omitted or simplified.

[0445] Referring to FIG. 15, the decoding apparatus can derive the L0 motion vector and the L1 motion vector of the current block (S1500).

[0446] The method disclosed in FIG. 15 may be performed by the decoding device 300 disclosed in FIG. 3. Specifically, steps S1500 to S1530 of FIG. 15 may be performed by the prediction unit 330 and the inter prediction unit 332 disclosed in FIG. 3, and step S1540 of FIG. 15 may be performed by the addition unit 340 disclosed in FIG. 3. In addition, the method disclosed in FIG. 15 may include the embodiments described in detail in this document. Therefore, in FIG. 15, detailed descriptions of content that overlaps with the above-described embodiments will be omitted or simplified.

[0447] Referring to FIG. 15, the decoding apparatus can derive the L0 motion vector and the L1 motion vector of the current block (S1500).

[0448] In one embodiment, the decoding device may determine a prediction mode for a current block based on prediction information signaled from the encoding device. The decoding device may then derive motion information (motion vectors, reference picture indexes, etc.) for the current block based on the prediction mode. Here, the prediction mode may include skip mode, merge mode, (A)MVP mode, etc.

[0449] For example, when a merge mode is applied to a current block, the decoding device may construct a merge candidate list based on neighboring blocks of the current block and select one merge candidate from among the merge candidates included in the merge candidate list. In this case, one merge candidate may be selected from the merge candidate list based on the merge index information. The decoding device may derive motion information of the current block using motion information of the selected merge candidate. That is, motion information of a merge candidate selected based on the merge index from among merge candidates included in the merge candidate list may be used as motion information of the current block.

[0450] Here, the motion information may include information such as a motion vector and a reference picture index, and may include L0 motion information and / or L1 motion information depending on the inter-prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). When pairwise prediction (Bi prediction) is applied to the current block, the motion information may include a motion vector in the L0 direction (L0 motion vector) and a motion vector in the L1 direction (L1 motion vector). The motion information may also include an L0 reference picture index and an L0 reference picture indicated by the L0 reference picture index in the L0 reference picture list, and an L1 reference picture indicated by the L1 reference picture index in the L1 reference picture list.

[0451] That is, the decoding device can determine whether the merge mode is applied to the current block based on prediction mode information of the current block (e.g., a merge flag (merge_flag) indicating whether the merge mode is applied). If the merge mode is applied to the current block based on the merge flag, the decoding device can acquire merge index information. Then, the decoding device can derive the L0 motion vector and L1 motion vector of the neighboring block indicated by the merge index information among the neighboring blocks included in the merge candidate list and use them as the L0 and L1 motion vectors of the current block.

[0452] The decoding device can apply DMVR to the current block based on DMVR flag information indicating whether DMVR is applied to the current block, and derive refined L0 motion vectors and refined L1 motion vectors based on the L0 motion vectors and L1 motion vectors (S1510).

[0453] In this case, the decoding device may determine whether to apply DMVR to the current block in consideration of coding efficiency, complexity, prediction performance, etc. That is, the decoding device may apply DMVR to the current block based on whether a predetermined DMVR application condition for the current block is satisfied. Here, the predetermined DMVR application condition may be some (or all) or a specific combination of the various application conditions described in Tables 2 to 42. In addition, the decoding device may derive DMVR flag information depending on whether the predetermined DMVR application condition is satisfied. The DMVR flag information may be information indicating whether DMVR is applied to the current block (e.g., the above-mentioned dmvrFlag).

[0454] In one embodiment, the pre-set DMVR application condition may include a condition that the values ​​of the L0 luma weighted prediction flag information and the L1 luma weighted prediction flag information are all zero.

[0455] In this case, when the case where the values ​​of L0 luma weighted prediction flag information (e.g., luma_weight_l0_flag) and L1 luma weighted prediction flag information (e.g., luma_weight_l1_flag) are all 0 is satisfied, the decoding device can derive the DMVR flag information to a value indicating that DMVR is applied to the current block. For example, if luma_weight_l0_flag and luma_weight_l1_flag are all 0, the DMVR flag information can be derived as a value (e.g., 1 or true) indicating that DMVR is applied to the current block. Otherwise, the DMVR flag information can be derived as a value (e.g., 0 or false) indicating that DMVR is not applied to the current block.

[0456] Here, the L0 luma weighted prediction flag information may be information indicating whether a weighting factor for L0 prediction is present for the luma component of the current block, and the L1 luma weighted prediction flag information may be information indicating whether a weighting factor for L1 prediction is present for the luma component of the current block. For example, a value of 0 in the L0 luma weighted prediction flag information may indicate that no weighting factor is present for the luma component of the L0 prediction, and a value of 1 in the L0 luma weighted prediction flag information may indicate that a weighting factor is present for the luma component of the L0 prediction. Furthermore, a value of 0 in the L1 luma weighted prediction flag information may indicate that no weighting factor is present for the luma component of the L1 prediction, and a value of 1 in the L1 luma weighted prediction flag information may indicate that a weighting factor is present for the luma component of the L1 prediction.

[0457] In addition, as an example, the pre-set DMVR application condition may include a condition that the value of bi-prediction weight index information of the current block is 0.

[0458] In this case, when the value of the paired prediction weighting index information (e.g., GbiIdx) is 0, the decoding device can derive the DMVR flag information to a value indicating that DMVR is applied to the current block. For example, if GbiIdx is 0, the DMVR flag information can be derived as a value (e.g., 1 or true) indicating that DMVR is applied to the current block. Otherwise, the DMVR flag information can be derived as a value (e.g., 0 or false) indicating that DMVR is not applied to the current block.

[0459] Here, when the value of the pair prediction weighting index information (e.g., GbiIdx) is 0, it may be a default case in which different weights are not applied to the L0 prediction and the L1 prediction. For example, as shown in Table 4, when the value of the pair prediction weighting index information is 0, it may be a case in which 1 / 2 weighting is applied to each of the L0 prediction and the L1 prediction.

[0460] In addition, as one embodiment, the pre-set DMVR application condition may include a condition that a prediction mode combining inter prediction and intra prediction (CIIP mode) is not applied to the current block.

[0461] In this case, when a prediction mode in which inter prediction and intra prediction are combined is not applied to the current block (e.g., when ciip_flag is 0), the decoding device may derive DMVR flag information to a value indicating that DMVR is applied to the current block. For example, when ciip_flag is 0, the DMVR flag information may be derived as a value (e.g., 1 or true) indicating that DMVR is applied to the current block. When ciip_flag is 1, the DMVR flag information may be derived as a value (e.g., 0 or false) indicating that DMVR is not applied to the current block.

[0462] In addition, as an example, the pre-set DMVR application condition may include a condition that the size of the current block is 16x16 or more, i.e., the height of the current block is 16 or more and the width of the current block is 16 or more.

[0463] In this case, when the size of the current block is equal to or larger than 16x16, the decoding device can derive DMVR flag information to a value indicating that DMVR is applied to the current block. For example, if the size of the current block is equal to or larger than 16x16, the DMVR flag information can be derived as a value (e.g., 1 or true) indicating that DMVR is applied to the current block. Otherwise (i.e., if the size of the current block is smaller than 16x16), the DMVR flag information can be derived as a value (e.g., 0 or false) indicating that DMVR is not applied to the current block.

[0464] In addition, according to an embodiment, the pre-set DMVR application conditions may include a condition when DMVR-based inter-bi-prediction is enabled, a condition when bi-prediction based on an L0 reference picture and an L1 reference picture is applied to the current block, a condition when a first POC (Picture Order Count) difference between the current picture and the L0 reference picture is the same as a second POC difference between the current picture and the L1 reference picture, a condition when a merge mode is applied to the current block, a condition when an MMVD (Merge Mode With Motion Vector Difference) mode is not applied to the current block, etc.

[0465] That is, the DMVR application conditions may be determined to include at least one of the above-described conditions, and the DMVR flag information may be derived based on the determined DMVR application conditions. In this case, if the DMVR application conditions are determined to include one or more conditions, the decoding device may determine that the DMVR is applied to the current block by deriving the value of the DMVR flag information as true or 1 if all of the preset DMVR application conditions are satisfied. Otherwise, if any of the preset DMVR application conditions is not satisfied, the decoding device may derive the value of the DMVR flag information as false or 0.

[0466] Here, in deriving the DMVR flag information, the DMVR application conditions listed above are merely examples, and the conditions in Tables 2 to 42 can be used in various combinations.

[0467] If the DMVR flag information indicates that DMVR is to be applied to the current block (for example, if the DMVR flag information is derived as true or a value of 1), the decoding device can apply DMVR to the current block and derive refined L0 motion vectors and refined L1 motion vectors based on the L0 motion vectors and L1 motion vectors.

[0468] In one embodiment, the decoding device can calculate a minimum SAD (Sum Of Absolute Differences) using reference samples in the L0 reference picture derived based on the L0 motion vector and reference samples in the L1 reference picture derived based on the L1 motion vector, and can derive a refined L0 motion vector for the L0 motion vector and a refined L1 motion vector for the L1 motion vector based on the sample positions corresponding to the minimum SAD.

[0469] Here, an L0 reference picture may be a reference picture indicated by an L0 reference picture index among the reference pictures included in the L0 reference picture list, and an L1 reference picture may be a reference picture indicated by an L1 reference picture index among the reference pictures included in the L1 reference picture list.

[0470] In addition, the refined L0 motion vector may be a motion vector that points to a sample position corresponding to the minimum SAD in the L0 reference picture, and the refined L1 motion vector may be a motion vector that points to a sample position corresponding to the minimum SAD in the L1 reference picture. The process of deriving a refined motion vector by applying DMVR has been described in detail in Figures 8 and 9, so a description thereof will be omitted here.

[0471] The decoding apparatus may derive an L0 predicted sample of the current block based on the refined L0 motion vector and an L1 predicted sample of the current block based on the refined L1 motion vector (S1520).

[0472] Here, the L0 prediction sample can be derived based on the reference sample indicated by the refined L0 motion vector in the L0 reference picture, and the L1 prediction sample can be derived based on the reference sample indicated by the refined L1 motion vector in the L1 reference picture.

[0473] The decoding device can apply BDOF to the current block based on BDOF flag information indicating whether BDOF is applied to the current block, and derive a prediction sample for the current block based on the L0 prediction sample and the L1 prediction sample (S1530).

[0474] In deriving the predicted samples, the decoding apparatus may determine whether to apply BDOF to the current block in consideration of coding efficiency, complexity, prediction performance, etc. That is, the decoding apparatus may apply BDOF to the current block based on whether a predetermined BDOF application condition for the current block is satisfied. Here, the predetermined BDOF application condition may be some (or all) or a specific combination of the various application conditions described in Tables 2 to 42 above. Furthermore, the decoding apparatus may derive BDOF flag information based on whether the predetermined BDOF application condition is satisfied. The BDOF flag information may be information indicating whether BDOF is applied to the current block (for example, the above-mentioned bdofFlag).

[0475] In one embodiment, the pre-set BDOF application condition may include a condition that the values ​​of the L0 luma weighted prediction flag information and the L1 luma weighted prediction flag information are all zero.

[0476] In this case, when the values ​​of L0 luma weighted prediction flag information (e.g., luma_weight_l0_flag) and L1 luma weighted prediction flag information (e.g., luma_weight_l1_flag) are all 0, the decoding device can derive the BDOF flag information as a value indicating that BDOF is applied to the current block. For example, when luma_weight_l0_flag and luma_weight_l1_flag are all 0, the BDOF flag information can be derived as a value (e.g., 1 or true) indicating that BDOF is applied to the current block. Otherwise, the BDOF flag information can be derived as a value (e.g., 0 or false) indicating that BDOF is not applied to the current block.

[0477] In addition, as an embodiment, the preset BDOF application condition may include a condition that the size of the current block is 16x16 or more, in other words, the height of the current block is 16 or more and the width of the current block is 16 or more.

[0478] In this case, when the size of the current block is 16x16 or more, the decoding device can derive BDOF flag information to a value indicating that BDOF is applied to the current block. For example, if the size of the current block is 16x16 or more, the BDOF flag information can be derived as a value (e.g., 1 or true) indicating that BDOF is applied to the current block. Otherwise (i.e., if the size of the current block is smaller than 16x16), the BDOF flag information can be derived as a value (e.g., 0 or false) indicating that BDOF is not applied to the current block.

[0479] In addition, as an embodiment, the preset BDOF application condition may include a condition that the height of the current block is 8 or more.

[0480] In this case, when the height of the current block is equal to or greater than 8, the decoding device can derive the BDOF flag information to a value representing that BDOF is applied to the current block. For example, if the height of the current block is equal to or greater than 8, the BDOF flag information can be derived as a value (e.g., 1 or true) representing that BDOF is applied to the current block. Otherwise (i.e., if the height of the current block is less than 8), the BDOF flag information can be derived as a value (e.g., 0 or false) representing that BDOF is not applied to the current block.

[0481] In one embodiment, the pre-set BDOF application conditions may include a condition where a first POC (Picture Order Count) difference between the current picture and the L0 reference picture is the same as a second POC difference between the current picture and the L1 reference picture.

[0482] In this case, when the first POC difference (e.g., DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])) and the second POC difference (e.g., DiffPicOrderCnt(currPic, RefPicList1[refIdxL1])) are the same, the decoding device can derive BDOF flag information to a value indicating that BDOF is applied to the current block. For example, if DiffPicOrderCnt(currPic, RefPicList0[refIdxL0])-DiffPicOrderCnt(currPic, RefPicList1[refIdxL1]) is 0, the BDOF flag information can be derived as a value (e.g., 1 or true) indicating that BDOF is applied to the current block. Otherwise, the BDOF flag information can be derived as a value (e.g., 0 or false) indicating that BDOF is not applied to the current block.

[0483] In addition, as an embodiment, the pre-set BDOF application conditions may include a condition for when a prediction mode combining inter prediction and intra prediction (CIIP mode) is not applied to the current block.

[0484] In this case, when a prediction mode in which inter prediction and intra prediction are combined is not applied to the current block (e.g., when ciip_flag is 0), the decoding device may derive BDOF flag information to a value indicating that BDOF is applied to the current block. For example, when ciip_flag is 0, the BDOF flag information may be derived as a value (e.g., 1 or true) indicating that BDOF is applied to the current block. When ciip_flag is 1, the BDOF flag information may be derived as a value (e.g., 0 or false) indicating that BDOF is not applied to the current block.

[0485] In addition, according to an embodiment, the pre-set BDOF application conditions may include a condition when BDOF-based inter-prediction is enabled, a condition when bi-prediction based on an L0 reference picture and an L1 reference picture is applied to the current block, a condition when the value of the bi-prediction weight index information of the current block is 0, a condition when affine mode is not applied to the current block, a condition when sub-block-based merge mode is not applied to the current block, and a condition when the current block is a luma component.

[0486] That is, the BDOF application conditions may be determined to include at least one of the above-described conditions, and the BDOF flag information may be derived based on the determined BDOF application conditions. In this case, if the BDOF application conditions are determined to include one or more conditions, the decoding device may determine that the BDOF is applied to the current block by deriving the value of the BDOF flag information as true or 1 if all of the preset BDOF application conditions are met. Otherwise, if any of the preset BDOF application conditions is not met, the decoding device may derive the value of the BDOF flag information as false or 0.

[0487] Here, in deriving the BDOF flag information, the BDOF application conditions listed above are merely examples, and the conditions in Tables 2 to 42 can be used in various combinations.

[0488] If the BDOF flag information indicates that BDOF is to be applied to the current block (for example, if the BDOF flag information is derived as true or a value of 1), the decoding device can apply BDOF to the current block to derive a predicted sample.

[0489] In one embodiment, the decoding device may calculate a first gradient for the L0 prediction sample derived based on the refined L0 motion vector and a second gradient for the L1 prediction sample derived based on the refined L1 motion vector. Then, the decoding device may finally derive a prediction sample for the current block based on the L0 prediction sample, the L1 prediction sample, the first gradient, and the second gradient. For example, the process of deriving a prediction sample by applying BDOF may use the calculations shown in Equations 1 to 6 above.

[0490] The decoding device may generate reconstructed samples for the current block based on the predicted samples (S1540).

[0491] In one embodiment, the decoding device can either use the predicted samples directly as reconstructed samples depending on the prediction mode, or generate reconstructed samples by adding residual samples to the predicted samples.

[0492] The decoding device may receive information about the residual for the current block if residual samples for the current block exist. The information about the residual may include transform coefficients for the residual samples. The decoding device may derive residual samples (or residual sample arrays) for the current block based on the residual information. The decoding device may generate reconstructed samples based on the prediction samples and the residual samples, and may derive a reconstructed block or a reconstructed picture based on the reconstructed samples.

[0493] In the above-described embodiments, the method is described based on a flowchart with a series of steps or blocks, but the embodiments of this document are not limited to the order of the steps, and some steps may occur in a different order or simultaneously than those described above. Furthermore, those skilled in the art will understand that the steps shown in the flowcharts are not exclusive, and other steps may be included, or one or more steps of the flowcharts may be deleted without affecting the scope of this document.

[0494] The above-described method according to the present document can be implemented in software form, and the encoding device and / or decoding device according to the present document can be included in a device that performs video processing, such as a TV, a computer, a smartphone, a set-top box, or a display device.

[0495] When the embodiments in this document are implemented in software, the methods described above may be implemented by modules (processes, functions, etc.) that perform the functions described above. The modules may be stored in memory and executed by a processor. The memory may be internal or external to the processor and may be connected to the processor by various known means. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and / or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described in this document may be implemented on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in each figure may be implemented on a computer, processor, microprocessor, controller, or chip. In this case, information (e.g., information on instructions) or algorithms for implementation may be stored on a digital storage medium.

[0496] In addition, decoding devices and encoding devices to which this document applies may be included in multimedia broadcast transmitting / receiving devices, mobile communication terminals, home cinema video devices, digital cinema video devices, surveillance cameras, video interaction devices, real-time communication devices such as video communications, mobile streaming devices, storage media, camcorders, video-on-demand (VoD) service providing devices, over-the-top (OTT) video devices, internet streaming service providing devices, three-dimensional (3D) video devices, virtual reality (VR) devices, augmented reality (AR) devices, image telephone video devices, transportation terminals (e.g., vehicle terminals (including autonomous vehicles), airplane terminals, ship terminals, etc.), medical video devices, etc., and may be used to process video signals or data signals. For example, over-the-top (OTT) video devices may include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, digital video recorders (DVRs), etc.

[0497] Furthermore, the processing method to which this document is applied can be produced in the form of a computer-executable program and stored in a computer-readable recording medium. Multimedia data having a data structure according to this document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium can include, for example, Blu-ray Disc (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. The computer-readable recording medium also includes media embodied in the form of a carrier wave (e.g., transmission via the Internet). The bitstream generated by the encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0498] Furthermore, the embodiments of the present document may be embodied in a computer program product by program code, which may be executed on a computer according to the embodiments of the present document. The program code may be stored on a computer-readable carrier.

[0499] FIG. 16 illustrates an example of a content streaming system in which the embodiments disclosed herein can be applied.

[0500] Referring to FIG. 16, a content streaming system applied to the embodiments of this document may generally include an encoding server, a streaming server, a web server, a media storage device (repository), a user device, and a multimedia input device.

[0501] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server may be omitted.

[0502] The bitstream can be generated by an encoding method or a bitstream generation method applied to an embodiment of this document, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0503] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, which then transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, which controls commands and responses between devices in the content streaming system.

[0504] The streaming server may receive content from a media storage device and / or an encoding server. For example, when receiving content from the encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a certain period of time to provide a smooth streaming service.

[0505] Examples of the user devices include mobile phones, smartphones, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smart watches, smart glasses, and HMDs (Head Mounted Displays)), digital TVs, desktop computers, and digital signage.

[0506] Each server in the content streaming system can be operated as a distributed server, and in this case, data received by each server can be processed in a distributed manner.

[0507] Meanwhile, a decoding device and an encoding device to which the embodiments of this document are applied may be included in a digital device. The term "digital device" includes, for example, all digital devices capable of performing at least one of transmitting, receiving, processing, and outputting data, content, services, etc. Here, processing data, content, services, etc. by a digital device includes encoding and / or decoding data, content, services, etc. Such digital devices pair or connect (hereinafter, referred to as "pairing") with other digital devices, external servers, etc. via a wired / wireless network to transmit and receive data and convert data as needed.

[0508] Digital devices include, for example, standing devices such as network TVs, hybrid broadcast broadband TVs (HBTVs), smart TVs, internet protocol televisions (IPTVs), and personal computers (PCs), as well as mobile or handheld devices such as personal digital assistants (PDAs), smartphones, tablet PCs, laptops, and VR / AR devices. In addition, the digital devices may include multimedia broadcasting transmitting / receiving devices, mobile communication terminals, home cinema video devices, digital cinema video devices, surveillance cameras, video interactive devices, real-time communication devices such as video communications, mobile streaming devices, storage media, camcorders, video-on-demand (VoD) service providing devices, OTT video (Over The Top video) devices, Internet streaming service providing devices, three-dimensional (3D) video devices, VR (Virtual Reality) devices, AR (Augmented Reality) devices, image telephone video devices, transportation terminals (e.g., vehicle (including autonomous vehicle) terminals, airplane terminals, ship terminals, etc.), and medical video devices.

[0509] Meanwhile, the term "wired / wireless network" as used in this document refers to a communication network that supports a variety of communication standards or protocols for interconnection and / or data transmission and reception between digital devices or between digital devices and external servers. Such wired / wireless networks may include all communication networks and communication protocols currently or in the future supported by standards, and therefore may include, for example, communication standards or protocols for wired connections such as USB (Universal Serial Bus), CVBS (Composite Video Banking Sync), component, S-Video (analog), DVI (Digital Visual Interface), HDMI (High Definition Multimedia Interface) (registered trademark), RGB, D-SUB, etc., as well as wireless networks such as Bluetooth (registered trademark), RFID (Radio Frequency Identification), infrared communication (IrDA, Infrared Data Association), UWB (Ultra WideBand), ZigBee, DLNA (Digital Living Network Alliance) (registered trademark), WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), LTE (Long Term Evolution), etc. It can be formed by communication standards for wireless connections such as Wi-Fi Evolution and Wi-Fi Direct.

[0510] Hereinafter, when the term "digital device" is used in this document, it may refer to a fixed device or a mobile device, or may include both, depending on the context.

[0511] Meanwhile, digital devices are intelligent devices that support, for example, broadcast reception functions, computer functions, and at least one external input, and can support e-mail, web browsing, banking, games, applications, and the like via the wired / wireless network. Digital devices can also be equipped with an interface to support at least one input or control means (hereinafter, "input means"), such as a manual input device, touch screen, or spatial remote control. Digital devices can use a standardized general-purpose operating system (OS). For example, digital devices can add, delete, amend, and update various applications on a general-purpose OS kernel, thereby providing a more user-friendly environment.

[0512] Meanwhile, the external input described in this document includes all input means or digital devices that are connected to the above-mentioned digital devices by wire / wireless and can transmit / receive and process related data therethrough, including digital devices such as HDMI (High Definition Multimedia Interface), game consoles such as PlayStation or X-Box, smartphones, tablet PCs, printers, and smart TVs.

[0513] Furthermore, the term "server" used in this document includes all digital devices or systems that provide data to clients, i.e., the digital devices described above, and may also be called processors. Such servers may include, for example, portal servers that provide web pages or web content, advertising servers that provide advertising data, content servers that provide content, SNS (Social Network Service) servers that provide SNS services, and service or manufacturing servers provided by manufacturers.

[0514] Additionally, the term "channel" used in this document refers to a path or means for transmitting and receiving data, such as a broadcasting channel. Here, broadcasting channels are expressed as physical channels, virtual channels, logical channels, etc., depending on the activation of digital broadcasting. A broadcasting channel can also be called a broadcasting network. A broadcasting channel refers to a channel for providing broadcast content provided by a broadcasting station or for access from a receiver. Because the broadcast content is primarily based on real-time broadcasting, it is also called a live channel. However, as broadcasting media have become more diverse in recent years, non-real-time broadcasting has become more prevalent in addition to real-time broadcasting. Therefore, a live channel can be understood as a term referring not only to real-time broadcasting but also to the entire broadcasting channel, including non-real-time broadcasting in some cases.

[0515] In addition to the above-mentioned broadcast channels, this document further defines an "arbitrary channel" in relation to the channels. The arbitrary channel may be provided together with a service guide such as an EPG (Electronic Program Guide) along with the broadcast channel, or a service guide, GUI (Graphical User Interface) or OSD (On-Screen Display) screen may be configured / provided for the arbitrary channel alone.

[0516] Meanwhile, unlike broadcast channels that have channel numbers pre-agreed between a transmitter and a receiver, arbitrary channels are channels arbitrarily assigned by a receiver, and are assigned channel numbers that do not generally overlap with channel numbers used to represent the broadcast channels. For example, when a receiver tunes to a specific broadcast channel, it receives a broadcast signal transmitting broadcast content and associated signaling information through the tuned channel. Here, the receiver parses channel information from the signaling information and creates a channel browser, EPG, etc. based on the parsed channel information and provides them to the user. When a user makes a channel change request through an input means, the receiver responds accordingly.

[0517] As described above, since broadcast channels are pre-agreed between the transmitting and receiving ends, assigning an arbitrary channel in overlap with a broadcast channel may cause confusion or may lead to confusion among users. Therefore, as described above, it is preferable not to assign an arbitrary channel in overlap with a broadcast channel. However, even if an arbitrary channel number is not assigned in overlap with a broadcast channel number, there is still a possibility of confusion during a user's channel surfing, so it is necessary to assign the arbitrary channel number taking this into consideration. This is because the arbitrary channel according to the embodiment of this document can also be implemented to be accessed like a broadcast channel by responding in the same manner to a user's channel change request via an input means, just like a conventional broadcast channel. Therefore, for the convenience of a user's access to an arbitrary channel and for the convenience of distinction or identification from a broadcast channel number, the arbitrary channel number may be defined and displayed in a form with letters added, such as arbitrary channel-1, arbitrary channel-2, etc., rather than in a numeric form like a broadcast channel, for the convenience of a user's access to an arbitrary channel and for the convenience of distinction or identification from a broadcast channel number. However, in this case, even if the arbitrary channel number is displayed in a form with letters added, such as arbitrary channel-1, it can be recognized and implemented in a numeric form like the above-mentioned broadcast channel number within the receiver. In addition, the arbitrary channel number can be provided in the form of a number like a broadcast channel, or the channel number can be defined and displayed in various ways that can be distinguished from the broadcast channel, such as Video Channel-1, Title-1, Video-1, etc.

[0518] A digital device executes a web browser for a web service and provides various types of web pages to a user. Here, the web pages include web pages containing video content. In the embodiment of this document, the video is separated and processed separately or independently from the web page. The separated video is assigned the arbitrary channel number described above and provided through a service guide, etc., and can be implemented so that it is output in response to a channel change request while the user is viewing the service guide or a broadcast channel. In addition to web services, for services such as broadcast content, games, and applications, predetermined content, images, audio, items, etc. can be separated and processed independently from the broadcast content, game, or application itself, and assigned an arbitrary channel number for playback, processing, etc., and implemented as described above.

[0519] FIG. 17 is a diagram schematically illustrating an example of a service system including digital devices.

[0520] A service system including digital devices includes a content provider (CP) 1710, a service provider (SP) 1720, a network provider (NP) 1730, and a home network end user (HNED) (customer) 1740. Here, the HNED 1740 is, for example, a client 1700, i.e., a digital device. The content provider 1710 produces and provides various content. As shown in FIG. 17, examples of such content providers 1710 include terrestrial broadcasters, cable system operators (SOs) or multiple SOs (MSOs), satellite broadcasters, various internet broadcasters, and private content providers (Private CPs). Meanwhile, the content provider 1710 provides various applications in addition to broadcast content.

[0521] The service provider 1720 packages the content provided by the content provider 1710 into a service package and provides it to the HNED 1740. For example, the service provider 20 in Fig. 17 packages first terrestrial broadcasting, second terrestrial broadcasting, cable MSO, satellite broadcasting, various internet broadcasting, applications, etc. and provides them to the HNED 1740.

[0522] The service provider 1720 provides services to the client 1700 in a unicast or multicast manner. Meanwhile, the service provider 1720 can transmit data to a large number of pre-registered clients 1700 at once, and for this purpose, the Internet Group Management Protocol (IGMP) can be used.

[0523] The content provider 1710 and the service provider 1720 may be the same or a single entity. For example, the content provider 1710 may package the content it produces into a service package and provide it to the HNED 1740, thereby performing the functions of the service provider 1720, or vice versa.

[0524] The network provider 1730 provides a network for data exchange between the content provider 1710 and / or the service provider 1720 and the client 1700 .

[0525] The client 1700 can create a home network and send and receive data.

[0526] Meanwhile, the content provider 1710 and / or the service provider 1720 in the service system can use conditional access or content protection means to protect the transferred content. In this case, the client 1700 can use processing means such as a cable card (POD: Point of Deployment) or a downloadable access server (DCAS) to support the conditional access or content protection.

[0527] In addition, the client 1700 can also use the interactive service via a network (or a communication network). In such a case, the client 1700 can function as a content provider, and the conventional service provider 1720 can receive the content and forward it to other clients.

[0528] 18 is a block diagram illustrating an embodiment of a digital device, which may correspond to, for example, the client 1700 in FIG. 17 and refer to the digital device described above.

[0529] The digital device 1800 is configured to include a network interface unit (Network Interface) 1801, a TCP / IP manager (TCP / IP Manager) 1802, a service delivery manager (Service Delivery Manager) 1803, an SI decoder 1804, a demultiplexing unit (Demux) 1805, an audio decoder (Audio Decoder) 1806, a video decoder (Video Decoder) 1807, a display unit (Display A / V and OSD Module) 1808, a service control manager (Service Control Manager) 1809, a service discovery manager (Service Discovery Manager) 1810, an SI & metadata database (SI & Meta data DB) 1811, a metadata manager (Metadata Manager) 1812, a service manager 1813, a UI manager 1814, and the like.

[0530] The network interface unit 1801 receives or transfers Internet Protocol (IP) packets via the network. That is, the network interface unit 1801 receives services, content, etc. from the service provider 1720 via the network.

[0531] The TCP / IP manager 1802 is responsible for transmitting IP packets received by the digital device 1800 and IP packets forwarded by the digital device 1800, i.e., between a source and a destination. The TCP / IP manager 1802 classifies the received packet(s) to correspond to the appropriate protocol and outputs the classified packet(s) to the service delivery manager 1805, the service discovery manager 1810, the service control manager 1809, the metadata manager 1812, etc. The service delivery manager 1803 is responsible for controlling received service data. For example, the service delivery manager 1803 can use RTP / RTCP when controlling real-time streaming data. When the real-time streaming data is transmitted using RTP, the service delivery manager 1803 parses the received data packets according to RTP and transmits them to the demultiplexer 1805, or stores them in the SI & metadata database 1811 under the control of the service manager 1813. The service delivery manager 203 then feeds back the network reception information to the server providing the service using RTCP. The demultiplexer 1805 demultiplexes the received packets into audio, video, SI (System Information) data, etc., and transmits them to the audio / video decoders 1806 / 1807 and the SI decoder 1804, respectively.

[0532] The SI decoder 1804 decodes service information such as Program Specific Information (PSI), Program and System Information Protocol (PSIP), and Digital Video Broadcasting-Service Information (DVB-SI).

[0533] Furthermore, the SI decoder 1804 stores the decoded service information, for example, in the SI & metadata database 1811. The service information stored in this manner can be read out and used by a corresponding component, for example, in response to a user request.

[0534] The audio / video decoders 1806 / 1807 decode the audio data and video data demultiplexed in the demultiplexer 1805. The decoded audio data and video data are provided to the user via a display unit 1808.

[0535] The application manager can be configured to include, for example, a UI manager 1814 and a service manager 1813. The application manager manages the overall state of the digital device 1800, provides a user interface, and can manage other managers.

[0536] The UI manager 1814 provides a GUI (Graphical User Interface) for the user using an OSD (On Screen Display) or the like, receives key input from the user, and performs device operations in response to the input. For example, when the UI manager 1814 receives key input related to channel selection from the user, it transfers the key input signal to the service manager 1813.

[0537] The service manager 1813 controls the managers associated with the service, such as the service delivery manager 1803, the service discovery manager 1810, the service control manager 1809, and the metadata manager 1812.

[0538] The service manager 1813 also creates a channel map and selects a channel using the channel map in response to a key input received from the user interface manager 1814. The service manager 1813 receives channel service information from the SI decoder 1804 and sets the audio / video PID (Packet IDentifier) ​​of the selected channel in the demultiplexer 1805. The PID thus set is used in the demultiplexing process described above. Therefore, the demultiplexer 1805 filters audio data, video data, and SI data using the PID.

[0539] The service discovery manager 1810 provides the information necessary to select a service provider to provide a service. Upon receiving a signal regarding channel selection from the service manager 1813, the service discovery manager 1810 uses the information to search for services.

[0540] The service control manager 1809 is responsible for selecting and controlling services. For example, if a user selects a live broadcasting service such as a conventional broadcasting method, the service control manager 1809 uses IGMP or RTSP to select and control the service, and if a user selects a service such as video on demand (VOD), the service control manager 1809 uses RTSP to select and control the service. The RTSP protocol can provide a trick mode for real-time streaming. The service control manager 1809 can also initialize and manage sessions via the IMS gateway 1850 using the IP Multimedia Subsystem (IMS) and Session Initiation Protocol (SIP). The above protocols are merely exemplary, and other protocols may be used depending on the implementation.

[0541] The Metadata Manager 1812 manages the metadata associated with the services and stores the metadata in the SI & Metadata Database 1811 .

[0542] The SI & metadata database 1811 stores service information decoded by the SI decoder 1804, metadata managed by the metadata manager 1812, and information required to select a service provider provided by the service discovery manager 1810. The SI & metadata database 1811 can also store setup data for the system.

[0543] The SI & metadata database 1811 may be implemented using a non-volatile memory (NVRAM) or a flash memory.

[0544] On the other hand, the IMS gateway 1850 is a gateway that collects functions required to access IMS-based IPTV services.

[0545] Fig. 19 is a configuration block diagram shown to explain another embodiment of a digital device, in particular, Fig. 19 illustrates a configuration block diagram of a mobile device as another embodiment of a digital device.

[0546] 19, a mobile device 1900 may include a wireless communication unit 1910, an A / V (Audio / Video) input unit 1920, a user input unit 1930, a sensing unit 1940, an output unit 1950, a memory 1960, an interface unit 1970, a control unit 1980, and a power supply unit 1990. The components shown in FIG. 19 are not essential, and a mobile device having more or fewer components may be implemented.

[0547] The wireless communication unit 1910 may include one or more modules that enable wireless communication between the mobile device 1900 and a wireless communication system or between the mobile device and a network in which the mobile device is located. For example, the wireless communication unit 1910 may include a broadcast reception module 1911, a mobile communication module 1912, a wireless Internet module 1913, a short-range communication module 1914, and a location information module 1915.

[0548] The broadcast receiving module 1911 receives broadcast signals and / or broadcast-related information from an external broadcast management server via a broadcast channel. Here, the broadcast channel may include a satellite channel or a terrestrial channel. The broadcast management server may refer to a server that generates and transmits broadcast signals and / or broadcast-related information, or a server that receives pre-generated broadcast signals and / or broadcast-related information and transmits them to a terminal. The broadcast signals may include TV broadcast signals, radio broadcast signals, and data broadcast signals, as well as broadcast signals in the form of TV broadcast signals or radio broadcast signals combined with data broadcast signals.

[0549] The broadcast-related information may refer to information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast-related information may also be provided via a mobile communication network. In such a case, the broadcast-related information may be received by the mobile communication module 1912.

[0550] Broadcast-related information can exist in various forms, for example, an Electronic Program Guide (EPG) or an Electronic Service Guide (ESG).

[0551] The broadcast receiving module 1911 can receive digital broadcast signals using digital broadcast systems such as ATSC, DVB-T (Digital Video Broadcasting-Terrestrial), DVB-S (Satellite), MediaFLO (Media Forward Link Only), DVB-H (Handheld), ISDB-T (Integrated Services Digital Broadcast-Terrestrial), etc. Of course, the broadcast receiving module 1911 can be configured to be compatible with not only the above-mentioned digital broadcast systems but also other broadcast systems.

[0552] Broadcast signals and / or broadcast-related information received via broadcast receiving module 1911 may be stored in memory 1960 .

[0553] The mobile communication module 1912 transmits and receives wireless signals to and from at least one of a base station, an external terminal, and a server in a mobile communication network. The wireless signals may include various types of data corresponding to transmission and reception of voice signals, video call signals, or text / multimedia messages.

[0554] The wireless internet module 1913 includes a module for wireless internet connection and may be built into or built into the mobile device 1900. Wireless internet technologies that may be used include WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World interoperability for Microwave Access), and HSDPA (High Speed ​​Downlink Packet Access).

[0555] The short-range communication module 1914 is a module for short-range communication. Short-range communication technologies that can be used include Bluetooth (registered trademark), RFID (Radio Frequency Identification), IrDA (Infrared Data Association), UWB (Ultra Wideband), ZigBee, RS-232, and RS-485.

[0556] The location information module 315 is a module for acquiring location information of the mobile device 1900, and may be, for example, a GPS (Global Position System) module.

[0557] The A / V input unit 1920 is for inputting audio and / or video signals, which may include a camera 1921 and a microphone 1922. The camera 1921 processes image frames, such as still images or videos, acquired by an image sensor in a video call mode or a shooting mode. The processed image frames can be displayed on the display unit 1951.

[0558] The image frames processed by the camera 1921 can be stored in the memory 1960 or transferred to the outside via the wireless communication unit 1910. Two or more cameras 1921 can be provided depending on the usage environment.

[0559] The microphone 1922 receives an external acoustic signal through a microphone in a call mode, a recording mode, a voice recognition mode, etc., and processes it as electrical voice data. In the call mode, the processed voice data can be converted into a form that can be transmitted to a mobile communication base station via the mobile communication module 1912 and output. The microphone 1922 can implement various noise reduction algorithms to remove noise generated in the process of receiving an external acoustic signal.

[0560] The user input unit 1930 generates input data for the user to control the operation of the terminal, and may be configured with a keypad, a dome switch, a touchpad (static / electrostatic), a jog wheel, a jog switch, etc.

[0561] The sensing unit 1940 senses the current state of the mobile device 300, such as whether the mobile device 1900 is opened or closed, the position of the mobile device 1900, whether or not a user touches the mobile device, the orientation of the mobile device, and acceleration / deceleration of the mobile device, and generates a sensing signal for controlling the operation of the mobile device 1900. For example, if the mobile device 1900 is moved or tilted, the sensing unit 1940 can sense the position or gradient of the mobile device. The sensing unit 1940 can also sense whether the power supply unit 1990 can supply power and whether the interface unit 1970 can be connected to an external device. Meanwhile, the sensing unit 1940 may include a proximity sensor 1941 including near field communication (NFC).

[0562] The output unit 1950 is for generating output related to vision, hearing, touch, or the like, and may include a display unit 1951, an audio output module 1952, an alarm unit 1953, and a haptic module 1954, etc.

[0563] The display unit 1951 displays (outputs) information processed in the mobile device 1900. For example, when the mobile device is in a call mode, it displays a UI (User Interface) or GUI (Graphical User Interface) related to the call. When the mobile device 1900 is in a video call mode or a photographing mode, it displays a photographed and / or received image or UI or GUI.

[0564] The display unit 1951 may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, and a 3D display.

[0565] Some of these displays may be transparent or light-transmitting so that the outside can be seen through them. This may be called a transparent display, and a representative example of the transparent display is a TOLED (Transparent OLED). The rear structure of the display unit 1951 may also be light-transmitting. This structure allows the user to see objects located behind the terminal body through the area of ​​the terminal body occupied by the display unit 1951.

[0566] Depending on the implementation of the mobile device 1900, there may be two or more display units 1951. For example, the mobile device 1900 may have multiple display units spaced apart or integrally arranged on one surface, or may be arranged on different surfaces.

[0567] When the display unit 1951 and a sensor for detecting a touch (hereinafter referred to as a touch sensor) are hierarchically structured (hereinafter referred to as a touch screen), the display unit 1951 can be used as an input device as well as an output device. The touch sensor can have the form of, for example, a touch film, a touch sheet, a touch pad, etc.

[0568] The touch sensor may be configured to convert into an electrical input signal a pressure applied to a specific portion of the display unit 1951 or a change in capacitance or the like generated at a specific portion of the display unit 1951. The touch sensor may be configured to detect not only the position and area of ​​a touch, but also the pressure at the time of touch.

[0569] When there is a touch input to the touch sensor, a corresponding signal(s) is sent to the touch controller, which processes the signal(s) and then transfers corresponding data to the control unit 1980. This allows the control unit 1980 to determine which area of ​​the display unit 1951 has been touched.

[0570] A proximity sensor 1941 may be disposed in an internal area of ​​a mobile device surrounded by a touchscreen or near the touchscreen. The proximity sensor is a sensor that detects the presence or absence of an object approaching or near a predetermined detection surface using electronic or infrared light without mechanical contact. Proximity sensors have a longer lifespan and are more widely used than contact-type sensors.

[0571] Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitance type proximity sensor, a magnetic type proximity sensor, an infrared proximity sensor, etc. If the touch screen is a capacitive type, it is configured to detect the proximity of the pointer by a change in the electric field caused by the proximity of the pointer. In this case, the touch screen (touch sensor) can be classified as a proximity sensor.

[0572] Hereinafter, for convenience of explanation, an action of bringing a pointer close to the touch screen without touching it so that the pointer is recognized as being located on the touch screen will be referred to as a “proximity touch,” and an action of actually contacting the touch screen will be referred to as a “contact touch.” The position of the pointer close to the touch screen means the position of the pointer that corresponds vertically to the touch screen when the pointer is close to the touch screen.

[0573] The proximity sensor detects proximity touches and proximity touch patterns (e.g., proximity touch distance, proximity touch direction, proximity touch speed, proximity touch duration, proximity touch position, proximity touch movement state, etc.) Information corresponding to the detected proximity touch actions and proximity touch patterns can be output on a touch screen.

[0574] The audio output module 1952 can output audio data received from the wireless communication unit 1910 or stored in the memory 1960 in a call signal reception mode, a call mode, a recording mode, a voice recognition mode, a broadcast reception mode, etc. The audio output module 1952 can also output audio signals associated with functions performed in the mobile device 1900 (e.g., a call signal reception sound, a message reception sound, etc.). Such an audio output module 1952 can include a receiver, a speaker, a buzzer, etc.

[0575] The alarm unit 1953 outputs a signal to notify the occurrence of an event in the mobile device 1900. Examples of events that occur in the mobile device include a call signal reception, a message reception, a key signal input, a touch input, etc. The alarm unit 1953 can output a signal to notify the occurrence of an event in other forms, such as vibration, in addition to a video signal or an audio signal.

[0576] The video or audio signal can also be output via a display unit 1951 or a sound output module 1952 , which can be classified as part of an alarm unit 1953 .

[0577] The haptic module 1954 generates various haptic effects that can be felt by the user. A representative example of the haptic effect generated by the haptic module 1954 is vibration. The intensity and pattern of the vibration generated by the haptic module 1954 can be controlled. For example, different vibrations can be synthesized and output, or output sequentially.

[0578] In addition to vibration, the haptic module 1954 can generate various tactile effects, such as a pin arrangement that moves vertically relative to the contacting skin surface, the force of air injection or suction through an injection or intake port, friction against the skin surface, contact of an electrode, magnetic effects such as static electricity, and effects that reproduce the sensation of cold or warmth using elements that can absorb or generate heat.

[0579] The haptic module 1954 can not only transmit haptic effects through direct contact, but can also be implemented so that the user can feel the haptic effects through the muscle sensation of the fingers, arms, etc. Two or more haptic modules 1954 can be provided depending on the configuration of the mobile device 1900.

[0580] The memory 1960 can store programs for the operation of the control unit 1980 and temporarily store input / output data (e.g., a phone book, messages, still images, videos, etc.) The memory 1960 can store data related to various patterns of vibration and sound output when a touch input is made on the touch screen.

[0581] The memory 1960 may include at least one type of storage medium selected from the group consisting of flash memory, hard disk, micro multimedia card, card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The mobile device 1900 may also operate in association with web storage that performs the storage function of the memory 1960 over the Internet.

[0582] The interface unit 1970 serves as a communication path with all external devices connected to the mobile device 1900. The interface unit 1970 receives data from an external device, receives power and transmits it to each component inside the mobile device 1900, or transmits data inside the mobile device 1900 to an external device. For example, the interface unit 1970 may include a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O port, an earphone port, etc.

[0583] The identification module is a chip that stores various information for authenticating the usage authority of the mobile device 1900, and may include a User Identify Module (UIM), a Subscriber Identify Module (SIM), a Universal Subscriber Identity Module (USIM), etc. A device equipped with an identification module (hereinafter, referred to as an identification device) may be manufactured in the form of a smart card. Therefore, the identification device may be connected to the terminal 1900 through a port.

[0584] When the mobile terminal 1900 is connected to an external cradle, the interface unit 1970 can be a communication path through which power from the cradle is supplied to the mobile terminal 1900 or a communication path through which various command signals input from the cradle by a user are transmitted to the mobile terminal. The various command signals or the power input from the cradle can be operated as a signal for recognizing that the mobile terminal is correctly attached to the cradle.

[0585] The control unit 1980 typically controls the overall operation of the mobile device. For example, it performs related control and processing for voice calls, data communications, video calls, etc. The control unit 380 may also include a multimedia module 1981 for multimedia playback. The multimedia module 1981 may be implemented within the control unit 1980 or may be implemented separately from the control unit 1980. The control unit 1980, particularly the multimedia module 1981, may include the encoding device and / or decoding device described above.

[0586] The control unit 1980 can perform pattern recognition processing that can recognize written or drawn inputs made on the touch screen as characters and images, respectively.

[0587] A power supply unit 1990 receives external and internal power sources under the control of the control unit 1980, and supplies the power required for the operation of each component.

[0588] The various embodiments described herein may be implemented, for example, using software, hardware, or a combination thereof in a recording medium readable by a computer or similar device.

[0589] Depending on the hardware implementation, the embodiments described herein may be implemented using at least one of Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or other electrical units for performing functions. In some cases, the embodiments described herein may be implemented in the control unit 1980 itself.

[0590] In a software implementation, the procedures, functions, and other embodiments described herein may be implemented by separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. The software code may be implemented by a software application written in an appropriate programming language. Here, the software code may be stored in memory 1960 and executed by controller 1980.

[0591] 20 is a schematic diagram illustrating an architecture for providing 3D image / video services to which one or more embodiments of the present document can be applied. This may represent a 360-degree or omnidirectional video / image processing system. This may also be implemented in, for example, an extended reality (XR)-supporting device. That is, the system may provide a way to provide virtual reality to users.

[0592] Augmented reality is a general term for virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR technology provides real-world objects or backgrounds as computer graphics images only, AR technology provides virtual computer graphics images on top of images of real objects, and MR technology is a computer graphics technology that combines virtual objects with the real world.

[0593] MR technology is similar to AR technology in that it displays both real and virtual objects. However, the difference is that in AR technology, virtual objects are used to complement real objects, while in MR technology, virtual and real objects are used with equal characteristics.

[0594] XR technology can be applied to head-mounted displays (HMDs), head-up displays (HUDs), mobile phones, tablet PCs, laptops, desktops, TVs, digital signage, etc., and a device to which XR technology is applied can be called an XR device. The XR device can include a first digital device and / or a second digital device, which will be described later.

[0595] 360-degree content generally refers to content for implementing and providing VR and can include 360-degree video and / or 360-degree audio. 360-degree video can refer to video or image content that is simultaneously captured or played in all directions (360 degrees or less) necessary to provide VR. Hereinafter, 360-degree video can refer to 360-degree video. 360-degree audio can also refer to spatial audio content, as audio content for providing VR, in which the sound source can be perceived as being located in a specific three-dimensional space. 360-degree content can be generated, processed, and transmitted to a user, who can then consume a VR experience using the 360-degree content. 360-degree video can be referred to as omnidirectional video, and 360-degree images can be referred to as omnidirectional images. Furthermore, while the following description will be based on 360-degree video, one or more embodiments of this document are not limited to VR and can include processing of video / image content such as AR and MR. 360-degree video can refer to a video or image that is displayed in various forms in 3D space according to a 3D model. For example, 360-degree video can be displayed on a spherical surface.

[0596] This method particularly proposes a method for effectively providing 360-degree video. To provide 360-degree video, the 360-degree video can first be captured using one or more cameras. The captured 360-degree video is then transmitted through a series of processes, and the receiving side can process and render the received data as the original 360-degree video. This allows the 360-degree video to be provided to the user.

[0597] Specifically, the entire process for providing 360-degree video may include a capture process, a preparation process, a transfer process, a processing process, a rendering process, and / or a feedback process.

[0598] The capture process may refer to the process of capturing images or videos for each of multiple viewpoints via one or more cameras. Image / video data such as (2010) in FIG. 20 may be generated through the capture process. Each plane in (2010) in FIG. 20 may represent an image / video for each viewpoint. These captured images / videos may be referred to as raw data. Metadata related to the capture may be generated during the capture process.

[0599] For this capture, a special camera for VR may be used. Depending on the embodiment, when providing a 360-degree video of a computer-generated virtual space, capture via an actual camera may not be performed. In this case, the capture process may be replaced by a process in which related data is simply generated.

[0600] The preparation process may be a process of processing the captured image / video and metadata generated during the capture process. During this preparation process, the captured image / video may undergo a stitching process, a projection process, a region-wise packing process, and / or an encoding process.

[0601] First, each image / video can undergo a stitching process, which can be a process of connecting each captured image / video to create a single panoramic or spherical image / video.

[0602] After this, the stitched image / video may undergo a projection process. In the projection process, the stitched image / video may be projected onto a 2D image. This 2D image may be called a 2D image frame depending on the context. Projecting onto a 2D image may also be expressed as mapping onto a 2D image. The projected image / video data may be in the form of a 2D image such as (2020) in FIG. 20.

[0603] The video data projected onto the 2D image may undergo a region-wise packing process to improve video coding efficiency. Region-wise packing may refer to a process of dividing the video data projected onto the 2D image into regions and processing them. Here, a region may refer to an area into which the 2D image onto which the 360-degree video data is projected is divided. Depending on the embodiment, the regions may be divided by equally dividing the 2D image or by arbitrarily dividing the 2D image. Depending on the embodiment, the regions may also be divided according to a projection scheme. The region-wise packing process is an optional process and may be omitted in the preparation process.

[0604] Depending on the embodiment, this process may include rotating or rearranging each region on the 2D image to improve video coding efficiency, for example, by rotating the regions so that certain edges of the regions are closer to each other, thereby improving coding efficiency.

[0605] Depending on the embodiment, this processing process may include a process of increasing or decreasing the resolution of a specific region to differentiate the resolution of each region on the 360-degree video. For example, a region corresponding to a relatively more important region on the 360-degree video may have a higher resolution than other regions. The video data projected onto the 2D image or the video data packed by region may undergo an encoding process using a video codec.

[0606] Depending on the embodiment, the preparation process may additionally include an editing process, etc. In this editing process, editing of image / video data before and after projection may be further performed. Similarly, in the preparation process, metadata related to stitching / projection / encoding / editing may be generated. In addition, metadata related to the initial viewpoint (time point) or ROI (Region of Interest) of the video data projected onto the 2D image may be generated.

[0607] The transmission process may be a process of processing and transmitting image / video data and metadata that have undergone a preparation process. Processing according to any transmission protocol may be performed for transmission. Data that has undergone processing for transmission may be transmitted via a broadcast network and / or broadband. This data may also be transmitted to a receiving side on an on-demand basis. The receiving side may receive the data via various routes.

[0608] The processing process may refer to the process of decoding the received data and re-projecting the projected image / video data onto a 3D model. In this process, the image / video data projected onto a 2D image can be re-projected onto 3D space. This process may also be called mapping or projection depending on the context. In this case, the mapped 3D space may have different shapes depending on the 3D model. For example, the 3D model may be a sphere, cube, cylinder, or pyramid.

[0609] Depending on the embodiment, the processing process may further include an editing process, an upscaling process, etc. In this editing process, editing of image / video data before and after reprojection may be further performed. If the image / video data is reduced in size, the size may be increased through sample upscaling in the upscaling process. If necessary, size reduction may also be performed through downscaling.

[0610] The rendering process may refer to a process of rendering and displaying image / video data reprojected onto a 3D space. Depending on the expression, reprojection and rendering may be combined to express rendering onto a 3D model. The image / video reprojected onto (or rendered onto) a 3D model may have a form such as (2030) in FIG. 20. (2030) in FIG. 20 illustrates a case where the image / video is reprojected onto a spherical 3D model. A user can view a portion of the rendered image / video through a VR display, etc. In this case, the area viewed by the user may have a form such as (2040) in FIG. 20.

[0611] The feedback process may refer to a process of transmitting various feedback information acquired during the display process to a transmitting side. Interactivity can be provided in the consumption of 360-degree videos through the feedback process. Depending on the embodiment, head orientation information, viewport information indicating the area the user is currently viewing, etc. may be transmitted to the transmitting side during the feedback process. Depending on the embodiment, the user may interact with something embodied in the VR environment, and in this case, information related to the interaction may be transmitted to the transmitting side or the service provider during the feedback process. Depending on the embodiment, the feedback process may not be performed.

[0612] Head orientation information can refer to information about the position, angle, movement, etc. of the user's head. Based on this information, information about the area the user is currently looking at in the 360-degree video, i.e., viewport information, can be calculated.

[0613] Viewport information can be information about the area the user is currently looking at in a 360-degree video. This can be used for gaze analysis to determine how the user is consuming the 360-degree video, and how long they are staring at which area of ​​the 360-degree video. Gaze analysis can be performed on the receiving side and transmitted to the sending side via a feedback channel. Devices such as VR displays can extract the viewport area based on the user's head position / direction, the vertical or horizontal field of view (FOV) information supported by the device, etc.

[0614] Meanwhile, 360-degree video / images can be processed based on subpictures. Projected pictures or packed pictures containing 2D images can be divided into subpictures, and processing can be performed on a subpicture-by-subpicture basis. For example, a specific subpicture(s) can be given a high resolution based on a user viewport, or only a specific subpicture(s) can be encoded and signaled to a receiving device (decoding device side). In this case, the decoding device can receive the subpicture bitstream, restore / decode the specific subpicture(s), and render them according to the user viewport.

[0615] According to an embodiment, the feedback information may be transmitted to the transmitting side and consumed by the receiving side. That is, the receiving side may perform decoding, reprojection, rendering, etc. using the feedback information. For example, the head orientation information and / or viewport information may be used to preferentially decode and render only the 360-degree video for the area currently viewed by the user.

[0616] Here, the term "viewport" or "viewport area" refers to the area a user views in a 360-degree video. The term "viewpoint" refers to the point at which a user views a 360-degree video, and may refer to the center point of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape of the area can be determined by the field of view (FOV).

[0617] Within the overall architecture for providing 360-degree video described above, image / video data that undergoes a series of processes of capture / projection / encoding / transfer / decoding / reprojection / rendering can be referred to as 360-degree video data. The term 360-degree video data can also be used as a concept that includes metadata or signaling information associated with such image / video data.

[0618] In order to store and transfer media data such as the above-mentioned audio or video, a standardized media file format may be defined. In some embodiments, the media file may have a file format based on the ISO Base Media File Format (ISO BMFF).

[0619] 21 is a diagram illustrating a configuration of a first digital device to which the present method can be applied. The first digital device may correspond to a transmitter for providing 3D image / video services such as VR / AR content. The first digital device may include an encoding device as described in detail herein. The encoding device may be included in or correspond to the data encoder.

[0620] The first digital device may perform operations related to the preparation process or the transmission process described above, and may include a data input unit, a stitcher, a projection processing unit, a region-specific packing processing unit (not shown), a metadata processing unit, a (transmitting side) feedback processing unit, a data encoder, an encapsulation processing unit, a transmission processing unit, and / or a transmission unit as internal / external elements.

[0621] The data input unit can receive each captured viewpoint image / video. The viewpoint image / video can be an image / video captured by one or more cameras. The data input unit can also receive metadata generated during the capture process. The data input unit can transfer the input viewpoint image / video to the stitcher and transfer the metadata generated during the capture process to the signaling processing unit.

[0622] The stitcher can perform stitching operations on captured viewpoint images / videos. The stitcher can transmit stitched 360-degree video data to the projection processing unit. If necessary, the stitcher can receive necessary metadata from the metadata processing unit and use it for the stitching operation. The stitcher can transmit metadata generated during the stitching process to the metadata processing unit. The metadata for the stitching process can include information such as whether stitching has been performed and the stitching type.

[0623] The projection processor can project the stitched 360-degree video data onto a 2D image. The projection processor can perform projection according to various schemes, which will be described later. The projection processor can perform mapping taking into account the corresponding depth of the 360-degree video data for each viewpoint. If necessary, the projection processor can receive metadata required for projection from the metadata processor and use it for the projection work. The projection processor can transmit metadata generated during the projection process to the metadata processor. The metadata of the projection processor can include types of projection schemes.

[0624] A region-specific packing processor (not shown) may perform the region-specific packing process described above. That is, the region-specific packing processor may divide the projected 360-degree video data into regions and perform processes such as rotating and rearranging each region or changing the resolution of each region. As described above, the region-specific packing process is an optional process, and the region-specific packing processor may be omitted if region-specific packing is not performed. If necessary, the region-specific packing processor may receive metadata required for region-specific packing from the metadata processor and use it for the region-specific packing operation. The region-specific packing processor may transmit metadata generated in the region-specific packing process to the metadata processor. The metadata of the region-specific packing processor may include the degree of rotation and size of each region.

[0625] The stitcher, projection processor, and / or region-specific packing processor may be implemented as a single hardware component, depending on the embodiment.

[0626] The metadata processing unit may process metadata that may be generated during the capture process, stitching process, projection process, region-specific packing process, encoding process, encapsulation process, and / or transmission process. The metadata processing unit may generate 360-degree video-related metadata using such metadata. According to an embodiment, the metadata processing unit may generate the 360-degree video-related metadata in the form of a signaling table. Depending on the signaling context, the 360-degree video-related metadata may also be referred to as metadata or 360-degree video-related signaling information. The metadata processing unit may also transmit the acquired or generated metadata to internal elements of the first digital device as needed. The metadata processing unit may transmit the 360-degree video-related metadata to a data encoder, an encapsulation processing unit, and / or a transmission processing unit so that the 360-degree video-related metadata can be transmitted to a receiving side.

[0627] The data encoder can encode 360-degree video data projected onto a 2D image and / or 360-degree video data packed by region. The 360-degree video data can be encoded in various formats. The 360-degree video data projected onto a 2D image and / or 360-degree video data packed by region can correspond to a (2D-based) input picture that is input to an encoding device. In this case, one or a series of input pictures can be encoded and output in the form of a bitstream based on the method proposed in the embodiments of this document. The bitstream can be included in the 360-degree video data.

[0628] The encapsulation processor may encapsulate the 360-degree video data and / or 360-degree video-related metadata in the form of a file, etc. Here, the 360-degree video-related metadata may be received from the metadata processor described above. The encapsulation processor may encapsulate the corresponding data in a file format such as ISOBMFF or CFF, or process it in other formats such as DASH segments. Depending on the embodiment, the encapsulation processor may include the 360-degree video-related metadata in a file format. For example, the 360-degree video-related metadata may be included in boxes at various levels in the ISOBMFF file format, or may be included as data in a separate track within the file. Depending on the embodiment, the encapsulation processor may encapsulate the 360-degree video-related metadata itself in a file. The transfer processor may apply transfer processing to the 360-degree video data encapsulated in the file format. The transfer processor may process the 360-degree video data according to any transfer protocol. The transfer processing may include processing for transmission via a broadcast network and processing for transmission via broadband. Depending on the embodiment, the transfer processing unit may receive not only 360-degree video data but also 360-degree video-related metadata from the metadata processing unit and process it for transfer.

[0629] The transfer unit may transfer the transferred 360-degree video data and / or 360-degree video-related metadata via a broadcast network and / or broadband. The transfer unit may include an element for transfer via a broadcast network and / or an element for transfer via broadband.

[0630] According to one embodiment of the first digital device according to this method, the first digital device may further include a data storage unit (not shown) as an internal / external element. The data storage unit may store encoded 360-degree video data and / or 360-degree video-related metadata before transmitting it to the transfer processing unit. This data may be stored in a file format such as ISOBMFF. If the 360-degree video is transferred in real time, a data storage unit may not be required. However, if the transfer is on-demand, non-real-time (NRT), or via broadband, the encapsulated 360-degree data may be stored in the data storage unit for a certain period of time before being transferred.

[0631] According to another embodiment of the first digital device according to this method, the first digital device may further include a (sending-side) feedback processor and / or a network interface (not shown) as internal / external elements. The network interface may receive feedback information from the second digital device according to this method and transmit it to the sending-side feedback processor. The sending-side feedback processor may transmit the feedback information to a stitcher, a projection processor, a region-specific packing processor, a data encoder, an encapsulation processor, a metadata processor, and / or a transmission processor. Depending on the embodiment, the feedback information may be transmitted to the metadata processor first and then transmitted to each internal element again. The internal elements that receive the feedback information may incorporate the feedback information into subsequent processing of the 360-degree video data.

[0632] According to another embodiment of the first digital device according to the present invention, the region-specific packing processor may rotate each region and map it onto a 2D image. In this case, each region may be rotated in a different direction or angle and mapped onto the 2D image. The region rotation may be performed taking into account adjacent portions, stitched portions, etc. of the 360-degree video data on a spherical surface before projection. Information regarding the region rotation, i.e., the rotation direction and angle, may be signaled by 360-degree video-related metadata. According to another embodiment of the first digital device according to the present invention, the data encoder may perform different encoding for each region. The data encoder may perform encoding at a high quality in a specific region and at a low quality in another region. The transmitting-side feedback processor may transmit feedback information received from the 360-degree video receiving device to the data encoder so that the data encoder can use different (differentiated) encoding methods for each region. For example, the transmitting-side feedback processor may transmit viewport information received from the receiving side to the data encoder. The data encoder can encode the region containing the area indicated by the viewport information at a higher quality (such as UHD) than other regions.

[0633] According to another embodiment of the first digital device according to the present method, the transmission processor may perform different transmission processing for each region, and may apply different transmission parameters (e.g., modulation order, coding rate, etc.) to each region to vary the robustness of the data transmitted for each region.

[0634] In this case, the transmitting side feedback processor may transmit feedback information received from the second digital device to the transfer processor, so that the transfer processor may perform different (differentiated) transfer processing for each region. For example, the transmitting side feedback processor may transmit viewport information received from the receiving side to the transfer processor. The transfer processor may perform transfer processing for a region including an area indicated by the viewport information so that the region has higher robustness than other regions.

[0635] The internal / external elements of the first digital device according to the method described above may be hardware elements implemented in hardware. Depending on the embodiment, the internal / external elements may be modified, omitted, replaced with, or integrated with other elements. Depending on the embodiment, additional elements may be added to the first digital device.

[0636] 22 is a diagram illustrating a configuration of a second digital device to which the present method can be applied. The second digital device may correspond to a receiving side for providing 3D image / video services such as VR / AR content. The second digital device may include a decoding device as described in detail herein. The decoding device may be included in or correspond to the data decoder.

[0637] The second digital device according to this method can perform operations related to the processing and / or rendering steps described above. The second digital device can include a receiver, a receiver processing unit, a decapsulation processing unit, a data decoder, a metadata parser, a (receiving side) feedback processing unit, a reprojection processing unit, and / or a renderer as internal / external elements. Meanwhile, the signaling parser can be called a metadata parser.

[0638] The receiving unit can receive the 360-degree video data transmitted by the first digital device according to this method. Depending on the channel, the receiving unit can receive the 360-degree video data via a broadcast network or a communication network such as broadband.

[0639] The receiving processor may perform processing on the received 360-degree video data according to a transfer protocol. In response to processing for transfer performed by the transfer side, the receiving processor may perform the reverse process of the transfer processor. The receiving processor may transfer the acquired 360-degree video data to a decapsulation processor and transfer the acquired 360-degree video-related metadata to a metadata parser. The 360-degree video-related metadata acquired by the receiving processor may be in the form of a signaling table.

[0640] The decapsulation processor may decapsulate 360-degree video data in file format received from the receiving processor. The decapsulation processor may decapsulate a file conforming to ISOBMFF or the like to acquire 360-degree video data or 360-degree video-related metadata. The acquired 360-degree video data may be transmitted to a data decoder, and the acquired 360-degree video-related metadata may be transmitted to a metadata parser. The 360-degree video-related metadata acquired by the decapsulation processor may be in the form of a box or track in a file format. The decapsulation processor may also receive metadata required for decapsulation from the metadata parser, if necessary.

[0641] The data decoder is capable of decoding 360-degree video data, obtaining an image / video-related bitstream from the 360-degree video data, and decoding / reconstructing the bitstream to decode / restore one or a series of (2D) pictures.

[0642] The data decoder may receive metadata necessary for decoding from the metadata parser. The 360-degree video-related metadata acquired during the data decoding process may be transmitted to the metadata parser.

[0643] The metadata parser can parse / decode the 360-degree video-related metadata and transmit the acquired metadata to a data decapsulator, a data decoder, a reprojector, and / or a renderer.

[0644] The reprojection processing unit may perform reprojection on one or more decoded / reconstructed 2D pictures related to the 360-degree video data. The reprojection processing unit may reproject the 2D pictures into a 3D space. The 3D space may have different forms depending on the 3D model used. The reprojection processing unit may also receive metadata required for reprojection from a metadata parser. For example, the reprojection processing unit may receive information regarding the type of 3D model used and detailed information about the model from the metadata parser. According to an embodiment, the reprojection processing unit may use the metadata required for reprojection to reproject only the 360-degree video data (or a region of the decoded / reconstructed picture corresponding to) a specific region in the 3D space into the 3D space.

[0645] The renderer can render the reprojected 360-degree video (or image). As described above, since the 360-degree video can also be expressed as being rendered in 3D space, when these two processes occur simultaneously, the reprojection processing unit and the renderer are integrated, and the entire process can be performed by the renderer. Depending on the embodiment, the renderer can also render only the portion that the user is viewing according to user viewpoint information.

[0646] A user can view a portion of the rendered 360-degree video through a VR display, which can be included in a second digital device (tethered) as a device playing the 360-degree video, or can be connected to the second digital device (un-tethered) as a separate device.

[0647] According to one embodiment of the second digital device according to this method, the second digital device may further include a (receiving) feedback processor and / or a network interface (not shown) as internal / external elements. The receiving feedback processor may acquire and process feedback information from a renderer, a reprojection processor, a data decoder, a decapsulation processor, and / or a VR display. The feedback information may include viewport information, head orientation information, gaze information, etc. The network interface may receive the feedback information from the receiving feedback processor and transfer it to the first digital device.

[0648] As described above, feedback information can be not only transmitted to the transmitting side but also consumed by the receiving side. The receiving-side feedback processor can transmit the acquired feedback information to internal elements of the second digital device so that it can be reflected in processes such as rendering. The receiving-side feedback processor can transmit the feedback information to a renderer, a reprojection processor, a data decoder, and / or a decapsulation processor. For example, the renderer can use the feedback information to prioritize rendering the area viewed by the user. Furthermore, the decapsulation processor, data decoder, etc. can prioritize decapsulate and decode the area viewed by the user or the area to be viewed by the user.

[0649] The internal / external elements of the second digital device according to the method described above may be hardware elements implemented in hardware. Depending on the embodiment, the internal / external elements may be modified, omitted, replaced with, or integrated with other elements. Depending on the embodiment, additional elements may be added to the second digital device.

[0650] The XR device can acquire information about the surrounding space or real objects by analyzing 3D point cloud data or image data acqui...

Claims

1. A method for video decoding, The steps involve deriving the L0 motion vector and L1 motion vector of the current block, A step of deriving a refined L0 motion vector and a refined L1 motion vector based on the L0 motion vector and the L1 motion vector, The steps include: deriving the L0 predicted sample and the L1 predicted sample of the current block based on the refined L0 motion vector and the refined L1 motion vector, respectively; The steps include: determining whether BDOF should be applied to the current block using Bi-Directional Optical Flow (BDOF) flag information, applying BDOF to the current block, thereby deriving a predicted sample for the current block based on the L0 predicted sample and the L1 predicted sample; The step of generating a restoration sample for the current block based on the prediction sample, A method for deriving the value of the BDOF flag information by applying the BDOF to the current block based on at least one of the following: (i) the prediction weighting index information is equal to 0; (ii) both the L0 luma weighted prediction flag information and the L1 luma weighted prediction flag information are equal to 0; or (iii) both the L0 chroma weighted prediction flag information and the L1 chroma weighted prediction flag information are equal to 0.

2. The value of the L0 luma weighted prediction flag information being equal to 0 indicates that there is no weighted factor for the luma component of the L0 prediction. The value of the L1 Luma weighted prediction flag information being equal to 0 indicates that there is no weighted factor for the Luma component of the L1 prediction. The value of the L0 chroma weighted prediction flag information being equal to 0 indicates that there is no weighted factor for the chroma component of the L0 prediction. A value equal to 0 for the L1 chroma weighted prediction flag information indicates that there is no weighted factor for the chroma component of the L1 prediction. The method according to claim 1.

3. The method according to claim 2, wherein the BDOF flag information is derived as a value related to indicating that the BDOF is not applied to the current block, based on the fact that at least one of the values ​​of the L0 luma-weighted prediction flag information, the L1 luma-weighted prediction flag information, the L0 chroma-weighted prediction flag information, or the L1 chroma-weighted prediction flag information is equal to 1.

4. The refined L0 motion vector and the refined L1 motion vector are derived by applying the DMVR to the current block based on the determination of whether the DMVR should be applied to the current block using the Decoder-side Motion Vector Refinement (DMVR) flag information. The method according to claim 1, wherein the DMVR flag information is derived based on the fact that at least one of the values ​​of the L0 luma-weighted prediction flag information, the L1 luma-weighted prediction flag information, the L0 chroma-weighted prediction flag information, or the L1 chroma-weighted prediction flag information is equal to 1.

5. The method according to claim 4, wherein the value of the aforementioned prediction weighting index information being equal to 0 indicates the default case in which no different weights are applied to the L0 prediction and the L1 prediction.

6. If the value of the aforementioned predicted weighting index information is equal to 1, The DMVR flag information is derived as indicating that the DMVR is not applied to the current block. The method according to claim 5, wherein the BDOF flag information is derived as indicating that the BDOF is not applied to the current block.

7. The DMVR flag information is derived based on whether the combined interprediction and intraprediction modes are applied to the current block. The method according to claim 4, wherein the value of the DMVR flag information indicates that the DMVR is applied to the current block based on the fact that the combined interprediction and intraprediction modes are not applied to the current block.

8. The DMVR flag information is derived based on whether the difference in the first Picture Order Count (POC) between the current picture and the L0 reference picture is equal to the difference in the second POC between the current picture and the L1 reference picture. The method according to claim 4, wherein the value of the DMVR flag information indicates that the DMVR is applied to the current block based on the first POC difference being equal to the second POC difference.

9. The BDOF flag information is derived based on whether the combined inter-prediction and intra-prediction modes are applied to the current block. The method according to claim 1, wherein the value of the BDOF flag information indicates that the BDOF is applied to the current block when the combined inter-prediction and intra-prediction modes are not applied to the current block.

10. The BDOF flag information is derived based on whether the difference in the first Picture Order Count (POC) between the current picture and the L0 reference picture is equal to the difference in the second POC between the current picture and the L1 reference picture. The method according to claim 1, wherein the BDOF flag information is derived as a value related to indicating that the BDOF is applied to the current block based on the first POC difference being equal to the second POC difference.

11. The steps for deriving the refined L0 motion vector and the refined L1 motion vector are as follows: The steps include: deriving the Sum of Absolute Differences (SAD) using the reference sample in the L0 reference picture derived based on the L0 motion vector and the reference sample in the L1 reference picture derived based on the L1 motion vector; A step of deriving the refined L0 motion vector for the L0 motion vector and the refined L1 motion vector for the L1 motion vector based on the sample position corresponding to the minimum SAD, The method according to claim 1, including the method described in claim 1.

12. The step of deriving the aforementioned predicted sample is: The steps include calculating the first gradient for the L0 predicted sample and the second gradient for the L1 predicted sample, A step of deriving the prediction sample based on the L0 prediction sample, the L1 prediction sample, the first grayscale, and the second grayscale, The method according to claim 1, including the method described in claim 1.

13. The steps for deriving the L0 motion vector and the L1 motion vector are as follows: The steps include determining whether merge mode is applied to the current block, The steps include: configuring a merge candidate list based on the surrounding blocks of the current block, based on the application of the merge mode to the current block; The steps include: deriving the L0 motion vector and the L1 motion vector based on the motion vector of the surrounding block indicated by the merge index information among the surrounding blocks included in the merge candidate list; The method according to claim 1, including the method described in claim 1.

Images