Image encoding / decoding method and apparatus for performing bdoF and method for transmitting bitstream

Abstract

An image encoding / decoding method and apparatus are provided. An image decoding method according to the disclosure is an image decoding method performed by an image decoding apparatus. The image decoding method can include the steps of deriving prediction samples of a current block based on motion information of the current block, determining whether to apply bi-directional optical flow (BDOF) to the current block, if the BDOF is to be applied to the current block, deriving a gradient for a current sub-block in the current block, deriving a refined motion vector (v x ,v y ) for the current sub-block based on the gradient, deriving a BDOF offset based on the gradient and the refined motion vector, and deriving refined prediction samples of the current block based on the prediction samples of the current block and the BDOF offset.

Description

TECHNICAL FIELD

[0001] The disclosure relates to an image encoding / decoding method and apparatus and a method of transmitting a bitstream, and more particularly, to an image encoding / decoding method and apparatus for performing bi-directional optical flow (BDOF) and a method of transmitting a bitstream generated by the image encoding method / apparatus of the disclosure. BACKGROUND

[0002] Recently, demands for high-resolution and high-quality images, such as high definition (HD) images and ultra-high definition (UHD) images, are increasing in various fields. As the resolution and quality of image data are improved, the amount of information or bits to be transmitted is relatively increased compared to existing image data. The increase in the amount of information or bits to be transmitted results in an increase in transmission cost and storage cost.

[0003] Therefore, an efficient image compression technique is needed to effectively transmit, store, and reproduce information about high-resolution and high-quality images. SUMMARY

[0004] TECHNICAL PROBLEM

[0005] An object of the disclosure is to provide an image encoding / decoding method and apparatus having improved encoding / decoding efficiency.

[0006] Another object of the disclosure is to provide an image encoding / decoding method and apparatus for deriving a BDOF offset.

[0007] Another object of the disclosure is to provide an image encoding / decoding method and apparatus for performing BDOF.

[0008] Another object of the disclosure is to provide a method of transmitting a bitstream generated by an image encoding method or apparatus according to the disclosure.

[0009] Another object of the disclosure is to provide a recording medium storing a bitstream generated by an image encoding method or apparatus according to the disclosure.

[0010] Another object of the disclosure is to provide a recording medium storing a bitstream generated by an image encoding method or apparatus according to the disclosure.

[0011] The technical problems solved by the disclosure are not limited to the above-mentioned technical problems, and other technical problems not described herein will be clearly understood by those skilled in the art from the following description.

[0012] TECHNICAL SOLUTION

[0013] The image decoding method according to an aspect of the disclosure can include deriving prediction samples of a current block based on motion information of the current block, determining whether to apply bi-directional optical flow (BDOF) to the current block, deriving a gradient of a current sub-block in the current block based on the application of the BDOF to the current block, deriving motion refinement (v x ,v y ) of the current sub-block based on the gradient, deriving a BDOF offset based on the gradient and the motion refinement, and deriving refined prediction samples of the current block based on the prediction samples of the current block and the BDOF offset.

[0014] In the image decoding method according to the disclosure, the deriving of the gradient can include right-shifting the prediction samples of the current block by a first shift, and the first shift can be set to a fixed value regardless of a bit depth of the current block.

[0015] In the image decoding method according to the disclosure, the first shift can be 6.

[0016] In the image decoding method according to the disclosure, the deriving of the motion refinement (v x ,v y ) can include deriving a first intermediate parameter diff based on the prediction samples of the current block, and deriving second intermediate parameters tempH and tempV based on the gradient.

[0017] In the image decoding method according to the disclosure, the deriving of the first intermediate parameter diff can include right-shifting the prediction samples of the current block by a second shift, and the second shift can be set to a fixed value regardless of a bit depth of the current block.

[0018] In the image decoding method according to the disclosure, the second shift can be 4.

[0019] In the image decoding method according to the disclosure, the deriving of the second intermediate parameters tempH and tempV can include right-shifting values derived based on the gradient by a third shift, and the third shift can be set to a fixed value regardless of a bit depth of the current block.

[0020] In the image decoding method according to the disclosure, the third shift can be 1.

[0021] In the image decoding method according to the disclosure, the motion refinement (v x ,v y ) can be clipped within a predetermined range.

[0022] In the image decoding method according to the disclosure, the predetermined range in which the motion refinement (v x ,v y ) is clipped can be set to a fixed range regardless of a bit depth of the current block.

[0023] In the image decoding method according to the disclosure, the deriving of the BDOF offset can include right-shifting a value derived based on the gradient and the motion refinement by a predetermined shift, and the predetermined shift can be set to a fixed range regardless of a bit depth of the current block.

[0024] In the image decoding method according to the disclosure, the deriving of the refined prediction sample of the current block can include clipping the BDOF offset within a predetermined range, and the predetermined range can be set based on a bit depth of the current block.

[0025] The image decoding apparatus according to another aspect of the disclosure can include a memory and at least one processor. The at least one processor can derive a prediction sample of a current block based on motion information of the current block; determine whether to apply bi-directional optical flow (BDOF) to the current block; derive a gradient of a current sub-block in the current block based on the application of the BDOF to the current block; derive motion refinement (v x ,v y ) of the current sub-block based on the gradient; derive a BDOF offset based on the gradient and the motion refinement; and derive a refined prediction sample of the current block based on the prediction sample of the current block and the BDOF offset.

[0026] The image encoding method according to another aspect of the disclosure can include deriving a prediction sample of a current block based on motion information of the current block; determining whether to apply bi-directional optical flow (BDOF) to the current block; deriving a gradient of a current sub-block in the current block based on the application of the BDOF to the current block; deriving motion refinement (v x ,v y ) of the current sub-block based on the gradient; deriving a BDOF offset based on the gradient and the motion refinement; and deriving a refined prediction sample of the current block based on the prediction sample of the current block and the BDOF offset.

[0027] In addition, the transmission method according to another aspect of the disclosure can transmit a bitstream generated by the image encoding apparatus or the image encoding method of the disclosure.

[0028] Furthermore, the computer-readable recording medium according to another aspect of the disclosure can store a bitstream generated by the image encoding apparatus or the image encoding method of the disclosure.

[0029] The features described above with respect to the brief overview of the disclosure are merely exemplary aspects of the following detailed description of the disclosure and do not limit the scope of the disclosure.

[0030] Advantageous Effects

[0031] According to the disclosure, it is possible to provide an image encoding / decoding method and apparatus having improved encoding / decoding efficiency.

[0032] Also, according to the present disclosure, it is possible to provide an image encoding / decoding method and apparatus for deriving BDOF offset.

[0033] Also, according to the present disclosure, it is possible to provide an image encoding / decoding method and apparatus for performing BDOF.

[0034] Also, according to the present disclosure, it is possible to provide a method of transmitting a bitstream generated by the image encoding method or apparatus according to the present disclosure.

[0035] Also, according to the present disclosure, it is possible to provide a recording medium storing a bitstream generated by the image encoding method or apparatus according to the present disclosure.

[0036] Also, according to the present disclosure, it is possible to provide a recording medium storing a bitstream generated by the image encoding method or apparatus according to the present disclosure.

[0037] Those skilled in the art will appreciate that the effects realized by the present disclosure are not limited to what has been particularly described hereinabove and other advantages of the present disclosure will be more clearly understood from the detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0038] FIG. 1 is a view schematically illustrating a video encoding system to which embodiments of the present disclosure are applicable.

[0039] FIG. 2 is a view schematically illustrating an image encoding apparatus to which embodiments of the present disclosure are applicable.

[0040] FIG. 3 is a view schematically illustrating an image decoding apparatus to which embodiments of the present disclosure are applicable.

[0041] FIG. 4 is a flowchart illustrating a video / image encoding method based on inter prediction.

[0042] FIG. 5 is a view illustrating a configuration of an inter prediction unit 180 according to the present disclosure.

[0043] FIG. 6 is a flowchart illustrating a video / image decoding method based on inter prediction.

[0044] FIG. 7 is a view illustrating a configuration of an inter prediction unit 260 according to the present disclosure.

[0045] FIG. 8 is a view illustrating neighboring blocks available as spatial merge candidates.

[0046] FIG. 9is a view that schematically illustrates a merge candidate list construction method according to an example of the present disclosure.

[0047] FIG. 10 is a view that illustrates a pair of candidates for a redundancy check performed on a spatial candidate.

[0048] FIG. 11 is a view that illustrates a method of scaling a motion vector of a temporal candidate.

[0049] FIG. 12 is a view that illustrates derivation of a position of a temporal candidate.

[0050] FIG. 13 is a view that schematically illustrates a motion vector predictor candidate list configuration method according to an example of the present disclosure.

[0051] FIG. 14 is a view that illustrates a CU extended to perform BDOF.

[0052] FIG. 15 is a view that illustrates a process of deriving prediction samples of a current block by applying BDOF.

[0053] FIG. 16 is a view that illustrates input and output of a BDOF process according to an embodiment of the present disclosure.

[0054] FIG. 17 is a view that illustrates variables used for a BDOF process according to an embodiment of the present disclosure.

[0055] FIG. 18 is a view that illustrates a method of generating prediction samples of each sub-block in a current CU based on whether BDOF is applied according to an embodiment of the present disclosure.

[0056] FIG. 19 is a view that illustrates a method of deriving gradients, auto-correlations, and cross-correlations of a current sub-block according to an embodiment of the present disclosure.

[0057] FIG. 20 is a view that illustrates a method of deriving motion refinements (vx, vy), deriving BDOF offsets, and generating prediction samples of a current sub-block according to an embodiment of the present disclosure.

[0058] FIG. 21 is a view that illustrates variables used for a BDOF process according to another embodiment of the present disclosure.

[0059] FIG. 22 is a view that illustrates a method of deriving gradients, auto-correlations, and cross-correlations of a current sub-block according to another embodiment of the present disclosure.

[0060] FIG. 23is a view illustrating a method of deriving motion refinement (vx, vy), deriving BDOF offset, and generating a prediction sample of a current sub-block according to another embodiment of the disclosure.

[0061] FIG. 24 is a view illustrating variables for BDOF processing according to another embodiment of the disclosure.

[0062] FIG. 25 is a view illustrating a method of deriving a gradient, an autocorrelation, and a cross-correlation of a current sub-block according to another embodiment of the disclosure.

[0063] FIG. 26 is a view illustrating variables for BDOF processing according to another embodiment of the disclosure.

[0064] FIG. 27 is a view illustrating a method of deriving a gradient, an autocorrelation, and a cross-correlation of a current sub-block according to another embodiment of the disclosure.

[0065] FIG. 28 is a view illustrating a content streaming system to which an embodiment of the disclosure is applicable. DETAILED DESCRIPTION

[0066] Hereinafter, embodiments of the disclosure will be described in detail with reference to the accompanying drawings so as to be easily implemented by those skilled in the art. The disclosure can, however, be implemented in various different forms and is not limited to the embodiments described herein.

[0067] In describing the disclosure, if it is determined that a detailed description of relevant known functions or configurations makes the scope of the disclosure unnecessarily obscure, a detailed description thereof will be omitted. In the drawings, parts irrelevant to the description of the disclosure are omitted, and like reference numerals are assigned to like parts.

[0068] In the disclosure, when one component is "connected", "coupled", or "linked" to another component, it can include not only a direct connection relationship but also an indirect connection relationship in which a middle component exists. In addition, when one component "includes" or "has" another component, it means that it can further include the other component unless otherwise specified, rather than excluding the other component.

[0069] In the disclosure, the terms first, second, and the like are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of the components unless otherwise specified. Accordingly, within the scope of the disclosure, a first component in one embodiment can be referred to as a second component in another embodiment, and similarly, a second component in one embodiment can be referred to as a first component in another embodiment.

[0070] In the disclosure, components distinguished from each other are intended to clearly describe each feature, and do not mean that the components must be separated. That is, a plurality of components can be integrated in one hardware or software unit, or one component can be distributed and implemented in a plurality of hardware or software units. Therefore, even if not specifically mentioned, the implementation of the components integrated or distributed is included in the scope of the disclosure.

[0071] In the disclosure, the components described in the various embodiments are not necessarily essential components, and some components can be optional components. Therefore, embodiments consisting of a subset of the components described in the embodiments are also included in the scope of the disclosure. Furthermore, embodiments including other components in addition to the components described in the various embodiments are included in the scope of the disclosure.

[0072] The disclosure relates to encoding and decoding of an image, and unless redefined in the disclosure, the terms used in the disclosure can have a general meaning commonly used in the technical field to which the disclosure belongs.

[0073] In the disclosure, a "picture" generally refers to a unit representing one image for a specific time period, and a slice / tile is a coding unit constituting a part of a picture, and one picture can consist of one or more slices / tiles. Furthermore, a slice / tile can include one or more coding tree units (CTU).

[0074] In the disclosure, "pixel" or "pel" can mean a minimum unit constituting one picture (or image). Furthermore, "sample" can be used as a term corresponding to a pixel. One sample can generally represent a pixel or a value of a pixel, and can represent only a pixel / pixel value of a luminance component or only a pixel / pixel value of a chrominance component.

[0075] In the disclosure, "unit" can mean a basic unit of image processing. The unit can include at least one of a specific region of a picture and information related to the region. In some cases, the unit can be used interchangeably with terms such as "sample array", "block", or "region". In general, an MxN block can include a set (or array) of M columns and N rows of samples (or sample array) or transform coefficients.

[0076] In the disclosure, "current block" can mean one of "current coding block", "current coding unit", "coding target block", "decoding target block", or "processing target block". When performing prediction, "current block" can mean "current prediction block" or "prediction target block". When performing transform (inverse transform) / quantization (dequantization), "current block" can mean "current transform block" or "transform target block". When performing filtering, "current block" can mean "filtering target block".

[0077] In the disclosure, the term " / " or "," can be interpreted to indicate "and / or". For example, "A / B" and "A,B" can mean "A and / or B". Also, "A / B / C" and "A / B / C" can mean "at least one of A, B, and / or C".

[0078] In the disclosure, the term "or" should be interpreted to indicate "and / or". For example, the expression "A or B" can include 1) only "A", 2) only "B", or 3) both "A and B". In other words, in the disclosure, "or" should be interpreted to indicate "additionally or alternatively".

[0079] Overview of a video encoding system

[0080] FIG. 1 is a view schematically illustrating a video coding system according to the disclosure.

[0081] The video coding system according to the embodiment can include an encoding apparatus 10 and a decoding apparatus 20. The encoding apparatus 10 can deliver encoded video and / or image information or data in the form of a file or a stream to the decoding apparatus 20 via a digital storage medium or a network.

[0082] The encoding apparatus 10 according to the embodiment can include a video source generator 11, an encoding unit 12, and a transmitter 13. The decoding apparatus 20 according to the embodiment can include a receiver 21, a decoding unit 22, and a renderer 23. The encoding unit 12 can be referred to as a video / image encoding unit, and the decoding unit 22 can be referred to as a video / image decoding unit. The transmitter 13 can be included in the encoding unit 12. The receiver 21 can be included in the decoding unit 22. The renderer 23 can include a display and the display can be configured as a separate device or an external component.

[0083] The video source generator 11 can acquire a video / image through a process of capturing, synthesizing, or generating a video / image. The video source generator 11 can include a video / image capturing device and / or a video / image generating device. The video / image capturing device can include, for example, one or more cameras, a video / image archive including previously captured videos / images, etc. The video / image generating device can include, for example, a computer, a tablet, and a smart phone, and can generate (electronically) a video / image. For example, a virtual video / image can be generated through a computer, etc. In this case, the video / image capturing process can be replaced by a process of generating related data.

[0084] The encoding unit 12 can encode an input video / image. For compression and coding efficiency, the encoding unit 12 can perform a series of processes such as prediction, transform, and quantization. The encoding unit 12 can output encoded data (encoded video / image information) in the form of a bitstream.

[0085] The transmitter 13 can transmit the encoded video / image information or data output in the form of a bitstream to the receiver 21 of the decoding device 20 in the form of a file or a stream through a digital storage medium or a network. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter 13 can include an element for generating a media file through a predetermined file format and can include an element for transmission through a broadcasting / communication network. The receiver 21 can extract / receive a bitstream from a storage medium or a network and transmit the bitstream to the decoding unit 22.

[0086] The decoding unit 22 can decode a video / image by performing a series of processes corresponding to the operations of the encoding unit 12, such as dequantization, inverse transform, and prediction.

[0087] The renderer 23 can render the decoded video / image. The rendered video / image can be displayed through a display.

[0088] Overview of an image encoding device

[0089] FIG. 2 is a view schematically showing an image encoding apparatus to which embodiments of the present disclosure are applicable.

[0090] As shown in FIG. 2 , the image encoding apparatus 100 can include an image partitioner 110, a subtractor 115, a transformer 120, a quantizer 130, a dequantizer 140, an inverse transformer 150, an adder 155, a filter 160, a memory 170, an inter prediction unit 180, an intra prediction unit 185, and an entropy encoder 190. The inter prediction unit 180 and the intra prediction unit 185 can be collectively referred to as a "prediction unit". The transformer 120, the quantizer 130, the dequantizer 140, and the inverse transformer 150 can be included in a residual processor. The residual processor can further include the subtractor 115.

[0091] In some embodiments, all or at least some of the plurality of components configuring the image encoding apparatus 100 can be configured by one hardware component (e.g., an encoder or a processor). Also, the memory 170 can include a decoded picture buffer (DPB) and can be configured by a digital storage medium.

[0092] Image segmenter 110 can segment an input image (or picture or frame) input to image encoding device 100 into one or more processing units. For example, a processing unit may be called an encoding unit (CU). Encoding units can be obtained by recursively segmenting encoding tree units (CTUs) or maximum encoding units (LCUs) according to a quadtree / binary tree / tritree (QT / BT / TT) structure. For example, an encoding unit can be segmented into multiple encoding units of greater depth based on a quadtree structure, a binary tree structure, and / or a ternary tree structure. For the segmentation of encoding units, a quadtree structure can be applied first, followed by a binary tree structure and / or a ternary tree structure. The encoding process according to this disclosure can be performed based on the final encoding unit that is no longer segmented. The maximum encoding unit can be used as the final encoding unit, or a deeper encoding unit obtained by segmenting the maximum encoding unit can be used as the final encoding unit. Here, the encoding process may include prediction, transformation, and reconstruction processes, which will be described later. As another example, the processing unit of the encoding process may be a prediction unit (PU) or a transformation unit (TU). Prediction units and transform units can be partitioned or segmented from the final coding unit. Prediction units can be sample prediction units, and transform units can be units used to derive transform coefficients and / or units used to derive residual signals from transform coefficients.

[0093] The prediction unit (inter-frame prediction unit 180 or intra-frame prediction unit 185) can perform prediction on the block to be processed (the current block) and generate a prediction block that includes prediction samples of the current block. The prediction unit can determine whether to apply intra-frame prediction or inter-frame prediction based on the current block or CU. The prediction unit can generate various information related to the prediction of the current block and transmit the generated information to the entropy encoder 190. The information about the prediction can be encoded in the entropy encoder 190 and output as a bitstream.

[0094] Intra-prediction unit 185 can predict the current block by referencing samples in the current frame. Depending on the intra-prediction mode and / or intra-prediction technique, the reference samples may be located among the neighbors of the current block or may be placed separately. Intra-prediction modes may include multiple non-directional modes and multiple directional modes. Non-directional modes may include, for example, DC mode and planar mode. Depending on the level of detail in the prediction direction, directional modes may include, for example, 33 or 65 directional prediction modes. However, this is merely an example, and more or fewer directional prediction modes may be used depending on the settings. Intra-prediction unit 185 can determine the prediction mode to be applied to the current block by using prediction modes applied to neighboring blocks.

[0095] The inter prediction unit 180 can derive a prediction block of the current block based on a reference block (a reference sample array) specified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of a block, a sub-block, or a sample based on the correlation of the motion information between the neighboring block and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, bi-prediction, etc.) information. In the case of inter prediction, the neighboring block can include a spatial neighboring block present in the current picture and a temporal neighboring block present in a reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block can be the same or different. The temporal neighboring block can be referred to as a collocated reference block, a collocated CU (colCU), etc. The reference picture including the temporal neighboring block can be referred to as a collocated picture (colPic). For example, the inter prediction unit 180 can configure a motion information candidate list based on the neighboring block and generate information indicating which candidate to use to derive a motion vector and / or a reference picture index of the current block. The inter prediction can be performed based on various prediction modes. For example, in the case of a skip mode and a merge mode, the inter prediction unit 180 can use motion information of the neighboring block as motion information of the current block. In the case of the skip mode, unlike the merge mode, a residual signal can not be transmitted. In the case of a motion vector prediction (MVP) mode, a motion vector of the neighboring block can be used as a motion vector predictor, and a motion vector of the current block can be signaled by encoding a motion vector difference and an indicator of the motion vector predictor. The motion vector difference can mean a difference between the motion vector of the current block and the motion vector predictor.

[0096] The prediction unit can generate a prediction signal based on various prediction methods and prediction techniques described below. For example, the prediction unit can not only apply intra prediction or inter prediction, but also simultaneously apply intra prediction and inter prediction to predict the current block. The prediction method of simultaneously applying both intra prediction and inter prediction to predict the current block can be referred to as combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict the current block. Intra block copy can be used for content image / video coding of games, etc., for example, screen content coding (SCC). IBC is a method of predicting a current picture using a reference block previously reconstructed in the current picture at a position separated by a predetermined distance. When IBC is applied, the position of the reference block in the current picture can be coded as a vector (a block vector) corresponding to the predetermined distance.

[0097] The prediction signal generated by the prediction unit can be used to generate a reconstructed signal or to generate a residual signal. The subtractor 115 can generate a residual signal (a residual block or a residual sample array) by subtracting the prediction signal (a prediction block or a prediction sample array) output from the prediction unit from the input image signal (an original block or an original sample array). The generated residual signal can be transmitted to the transformer 120.

[0098] The transformer 120 can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique can include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a karhunen-loève transform (KLT), a graph-based transform (GBT), or a conditional non-linear transform (CNT). Here, the GBT refers to a transform obtained from a graph when relationship information between pixels is represented by a graph. The CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process can be applied to a square pixel block having the same size or can be applied to a block having a variable size other than a square.

[0099] The quantizer 130 can quantize the transform coefficients and transmit them to the entropy encoder 190. The entropy encoder 190 can encode the quantized signal (information about the quantized transform coefficients) and output a bitstream. The information about the quantized transform coefficients can be referred to as residual information. The quantizer 130 can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0100] The entropy encoder 190 can perform various encoding methods, such as exponential golomb, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), and the like. The entropy encoder 190 can encode information (e.g., values of syntax elements, etc.) required for video / image reconstruction other than the quantized transform coefficients together or individually. The encoded information (e.g., encoded video / image information) can be transmitted or stored in the form of a bitstream in units of a network abstraction layer (NAL). The video / image information can further include information about various parameter sets, such as an adaptive parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information can further include general constraint information. The information signaled, transmitted, and / or syntax elements described in the disclosure can be encoded through the above-described encoding process and included in the bitstream.

[0101] The bitstream can be transmitted through a network or can be stored in a digital storage medium. The network can include a broadcasting network and / or a communication network, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that transmits a signal output from the entropy encoder 190 and / or a storage unit (not shown) that stores the signal can be included as an internal / external element of the image encoding apparatus 100. Alternatively, the transmitter can be provided as a component of the entropy encoder 190.

[0102] The quantized transform coefficients output from the quantizer 130 can be used to generate a residual signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients through the dequantizer 140 and the inverse transformer 150.

[0103] The adder 155 adds the reconstructed residual signal to a prediction signal output from the inter prediction unit 180 or the intra prediction unit 185 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array). If there is no residual for a block to be processed, for example, in the case of applying a skip mode, a prediction block can be used as a reconstructed block. The adder 155 can be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal can be used for intra prediction of a next block to be processed in the current picture, and can be used for inter prediction of a next picture by filtering as described below.

[0104] In addition, luma mapping with chroma scaling (LMCS) is applied to the picture encoding process as described below.

[0105] The filter 160 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 160 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 170, specifically, in the DPB of the memory 170. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. The filter 160 can generate various information related to filtering and transmit the generated information to the entropy encoder 190, as described later in the description of each filtering method. The information related to filtering can be encoded by the entropy encoder 190 and output in the form of a bitstream.

[0106] The modified reconstructed picture transmitted to the memory 170 can be used as a reference picture in the inter prediction unit 180. When inter prediction is applied by the image encoding apparatus 100, prediction mismatch between the image encoding apparatus 100 and an image decoding apparatus can be avoided and encoding efficiency can be improved.

[0107] The DPB of memory 170 can store modified reconstructed frames for use as reference frames in inter-frame prediction unit 180. Memory 170 can store motion information of blocks from which motion information in the current frame is derived (or encoded) and / or motion information of already reconstructed blocks in the frame. The stored motion information can be transmitted to inter-frame prediction unit 180 and used as motion information for spatially or temporally neighboring blocks. Memory 170 can store reconstructed samples of reconstructed blocks in the current frame and can transmit the reconstructed samples to intra-frame prediction unit 185.

[0108] Overview of an image decoding device

[0109] FIG. 3 This is a schematic view illustrating an image decoding device to which embodiments of the present disclosure may be applied.

[0110] like FIG. 3 As shown, the image decoding device 200 may include an entropy decoder 210, a dequantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-frame prediction unit 260, and an intra-frame prediction unit 265. The inter-frame prediction unit 260 and the intra-frame prediction unit 265 may be collectively referred to as "prediction units". The dequantizer 220 and the inverse transformer 230 may be included in a residual processor.

[0111] According to an implementation, all or at least some of the components of the image decoding device 200 can be configured by hardware components (e.g., a decoder or a processor). Furthermore, the memory 250 may include a decoded screen buffer (DPB) or may be configured by a digital storage medium.

[0112] The image decoding device 200, having received a bitstream including video / image information, can perform operations related to... FIG. 2 The image is reconstructed by processing corresponding to the processing performed by the image encoding device 100. For example, the image decoding device 200 can perform decoding using a processing unit applied in the image encoding device. Therefore, the decoding processing unit can be, for example, an encoding unit. The encoding unit can be obtained by segmenting a coding tree unit or a maximum coding unit. The reconstructed image signal decoded and output by the image decoding device 200 can be reproduced by a reproduction device (not shown).

[0113] Image decoding device 200 can receive data in bitstream form from... FIG. 2The received signal can be decoded by the entropy decoder 210. For example, the entropy decoder 210 can parse a bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information can further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information can further include general constraint information. The image decoding apparatus can also decode a picture based on the information on the parameter sets and / or the general constraint information. The information and / or the syntax elements described in the disclosure to be signaled / received can be decoded through a decoding process and obtained from the bitstream. For example, the entropy decoder 210 decodes information in the bitstream based on an encoding method such as exponential Golomb coding, CAVLC, or CABAC, and outputs values of syntax elements required for image reconstruction and quantized values of transform coefficients of a residual. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream, determine a context model using information of a decoded target syntax element, decoded information of a neighboring block and a decoded target block, or a symbol / bin decoded in a previous stage, perform arithmetic decoding on the bins by predicting a probability of occurrence of the bins according to the determined context model, and generate a symbol corresponding to a value of each syntax element. In this case, the CABAC entropy decoding method can update the context model by using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information related to prediction among the information decoded by the entropy decoder 210 can be provided to the prediction unit (inter prediction unit 260 and intra prediction unit 265), and residual values, i.e., quantized transform coefficients and related parameter information, on which entropy decoding is performed in the entropy decoder 210 can be input to the dequantizer 220. In addition, information on filtering among the information decoded by the entropy decoder 210 can be provided to the filter 240. Further, a receiver (not shown) for receiving a signal output from the image encoding apparatus can be further configured as an internal / external element of the image decoding apparatus 200, or the receiver can be a component of the entropy decoder 210.

[0114] Further, the image decoding apparatus according to the disclosure can be referred to as a video / image / picture decoding apparatus. The image decoding apparatus can be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder can include the entropy decoder 210. The sample decoder can include at least one of the dequantizer 220, the inverse transformer 230, the adder 235, the filter 240, the memory 250, the inter prediction unit 260, or the intra prediction unit 265.

[0115] The dequantizer 220 can dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 220 can rearrange the quantized transform coefficients in the form of a two-dimensional block. In this case, the rearrangement can be performed based on a coefficient scanning order performed in the image encoding apparatus. The dequantizer 220 can perform dequantization on the quantized transform coefficients by using a quantization parameter (e.g., quantization step length information) and obtain the transform coefficients.

[0116] The inverse transformer 230 can inverse-transform the transform coefficients to obtain a residual signal (a residual block, a residual sample array).

[0117] The prediction unit can perform prediction on the current block and generate a prediction block including prediction samples of the current block. The prediction unit can determine whether to apply intra prediction or inter prediction to the current block based on information about prediction output from the entropy decoder 210, and can determine a specific intra / inter prediction mode (prediction technique).

[0118] As the same as described in the prediction unit of the image encoding apparatus 100, the prediction unit can generate a prediction signal based on various prediction methods (techniques) described later.

[0119] The intra prediction unit 265 can predict the current block by referring to samples in the current picture. The description of the intra prediction unit 185 is equally applicable to the intra prediction unit 265.

[0120] The inter prediction unit 260 can derive a prediction block of the current block based on a reference block (a reference sample array) on a reference picture specified by a motion vector. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of a block, a sub-block, or a sample based on the correlation of the motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, bi-prediction, etc.) information. In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter prediction unit 260 can configure a motion information candidate list based on the neighboring blocks, and derive a motion vector and / or a reference picture index of the current block based on received candidate selection information. The inter prediction can be performed based on various prediction modes, and the information about prediction can include information indicating the inter prediction mode of the current block.

[0121] The adder 235 can generate a reconstructed signal (a reconstructed picture, a reconstructed block, a reconstructed sample array) by adding the obtained residual signal and a prediction signal (a prediction block, a prediction sample array) output from the prediction unit (including the inter prediction unit 260 and / or the intra prediction unit 265). The description of the adder 155 is equally applicable to the adder 235.

[0122] In addition, as described below, luminance mapping with chroma scaling (LMCS) is applied to a picture decoding process.

[0123] The filter 240 can improve subjective / objective picture quality by applying filtering to the reconstructed signal. For example, the filter 240 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 250, specifically, in the DPB of the memory 250. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc.

[0124] The (modified) reconstructed picture stored in the DPB of the memory 250 can be used as a reference picture in the inter prediction unit 260. The memory 250 can store motion information of a block from which motion information in a current picture is derived (or decoded) and / or motion information of a block in a picture that has been reconstructed. The stored motion information can be transferred to the inter prediction unit 260 to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 250 can store reconstructed samples of a reconstructed block in a current picture and transfer the reconstructed samples to the intra prediction unit 265.

[0125] In the disclosure, the embodiments described in the filter 160, the inter prediction unit 180, and the intra prediction unit 185 of the image encoding apparatus 100 can be equally or correspondingly applied to the filter 240, the inter prediction unit 260, and the intra prediction unit 265 of the image decoding apparatus 200.

[0126] Overview of inter prediction

[0127] The image encoding apparatus / image decoding apparatus can perform inter prediction in a unit of a block to derive a prediction sample. Inter prediction can mean prediction that is derived in a manner of depending on a data element of a picture other than a current picture. When inter prediction is applied to a current block, a prediction block of the current block can be derived based on a reference block on a reference picture specified by a motion vector.

[0128] In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information of the current block can be derived based on the correlation of the motion information between neighboring blocks and the current block, and the motion information can be derived in a unit of a block, a sub-block, or a sample. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction type information. Here, the inter prediction type information can mean direction information of inter prediction. The inter prediction type information can indicate that the current block is predicted using one of L0 prediction, L1 prediction, or bi-prediction.

[0129] When inter prediction is applied to the current block, neighboring blocks of the current block can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference blocks of the current block and the reference picture including the temporal neighboring blocks can be the same or different. The temporal neighboring block can be referred to as a collocated reference block or a collocated CU (colCU), and the reference picture including the temporal neighboring block can be referred to as a collocated picture (colPic).

[0130] Further, a motion information candidate list can be constructed based on the neighboring blocks of the current block, and in this case, flag or index information indicating which candidate to use can be signaled in order to derive the motion vector and / or the reference picture index of the current block.

[0131] According to the inter prediction type, the motion information can include L0 motion information and / or L1 motion information. A motion vector in the L0 direction can be defined as an L0 motion vector or MVL0, and a motion vector in the L1 direction can be defined as an L1 motion vector or MVL1. Prediction based on the L0 motion vector can be defined as L0 prediction, prediction based on the L1 motion vector can be defined as L1 prediction, and prediction based on both the L0 motion vector and the L1 motion vector can be defined as bi-prediction. Here, the L0 motion vector can mean a motion vector associated with a reference picture list L0, and the L1 motion vector can mean a motion vector associated with a reference picture list L1.

[0132] The reference picture list L0 can include pictures preceding the current picture in the output order as reference pictures, and the reference picture list L1 can include pictures succeeding the current picture in the output order. A preceding picture can be defined as a forward (reference) picture, and a succeeding picture can be defined as a backward (reference) picture. Further, the reference picture list L0 can also include pictures succeeding the current picture in the output order as reference pictures. In this case, within the reference picture list L0, the preceding pictures can be indexed first, and then the succeeding pictures can be indexed. The reference picture list L1 can also include pictures preceding the current picture in the output order as reference pictures. In this case, within the reference picture list L1, the succeeding pictures can be indexed first, and then the preceding pictures can be indexed. Here, the output order can correspond to a picture order count (POC) order.

[0133] FIG. 4 is a flowchart illustrating a video / image encoding method based on inter prediction.

[0134] FIG. 5 is a view illustrating a configuration of an inter prediction unit 180 according to the disclosure.

[0135] FIG. 6 The encoding method of can be performed byFIG. 2 The image encoding device performs the following steps: Specifically, step S410 can be performed by the inter-frame prediction unit 180, and step S420 can be performed by the residual processor. Specifically, step S420 can be performed by the subtractor 115. Step S430 can be performed by the entropy encoder 190. The prediction information of step S630 can be derived by the inter-frame prediction unit 180, and the residual information of step S630 can be derived by the residual processor. The residual information is information about the residual samples. The residual information may include information about the quantization transform coefficients used for the residual samples. As described above, the residual samples can be derived into transform coefficients by the transformer 120 of the image encoding device, and the transform coefficients can be derived into quantized transform coefficients by the quantizer 130. The information about the quantized transform coefficients can be encoded by the entropy encoder 190 through the residual encoding process.

[0136] The image coding device can perform inter-frame prediction for the current block (S410). The image coding device can deduce the inter-frame prediction mode and motion information for the current block and generate prediction samples for the current block. Here, the inter-frame prediction mode determination, motion information deduction, and prediction sample generation processes can be performed simultaneously, or any one of them can be performed before other processes. For example, as... FIG. 5 As shown, the inter-frame prediction unit 180 of the image coding apparatus may include a prediction mode determination unit 181, a motion information derivation unit 182, and a prediction sample derivation unit 183. The prediction mode determination unit 181 determines the prediction mode for the current block, the motion information derivation unit 182 derives the motion information for the current block, and the prediction sample derivation unit 183 derives the prediction samples for the current block. For example, the inter-frame prediction unit 180 of the image coding apparatus can search for blocks similar to the current block within a predetermined region (search region) of a reference frame using motion estimation, and derive a reference block whose difference from the current block is equal to or less than a predetermined criterion or minimum value. Based on this, a reference frame index indicating the reference frame in which the reference block is located can be derived, and a motion vector can be derived based on the positional difference between the reference block and the current block. The image coding apparatus can determine the mode applicable to the current block among various inter-frame prediction modes. The image coding apparatus can compare rate distortion (RD) costs for various prediction modes and determine the optimal inter-frame prediction mode for the current block. However, the methods by which an image coding device determines the inter-frame prediction mode of the current block are not limited to the examples above, and various methods can be used.

[0137] For example, the inter prediction mode of the current block can be determined as at least one of a merge mode, a merge skip mode, a motion vector prediction (MVP) mode, a symmetric motion vector difference (SMVD) mode, an affine mode, a subblock-based merge mode, an adaptive motion vector resolution (AMVR) mode, a history-based motion vector predictor (HMVP) mode, a pair-wise average merge mode, a merge mode with motion vector difference (MMVD) mode, a decoder-side motion vector refinement (DMVR) mode, a combined inter and intra prediction (CIIP) mode, or a geometric partition mode (GPM).

[0138] For example, when the skip mode or the merge mode is applied to the current block, the image encoding apparatus can derive merge candidates from neighboring blocks of the current block, and construct a merge candidate list using the derived merge candidates. Also, the image encoding apparatus can derive a reference block whose difference from the current block is equal to or less than a predetermined criterion or a minimum value among reference blocks indicated by the merge candidates included in the merge candidate list. In this case, a merge candidate associated with the derived reference block can be selected, and merge index information indicating the selected merge candidate can be generated and signaled to the image decoding apparatus. Motion information of the selected merge candidate can be used to derive motion information of the current block.

[0139] As another example, when the MVP mode is applied to the current block, the image encoding apparatus can derive motion vector predictor (MVP) candidates from neighboring blocks of the current block, and construct an MVP candidate list using the derived MVP candidates. Also, the image encoding apparatus can use a motion vector of an MVP candidate selected from among the MVP candidates included in the MVP candidate list as an MVP of the current block. In this case, for example, a motion vector indicating a reference block derived through the above-described motion estimation can be used as a motion vector of the current block, an MVP candidate having a motion vector with a minimum difference from a motion vector of the current block among the MVP candidates can be the selected MVP candidate. A motion vector difference (MVD) that is a difference obtained by subtracting the MVP from the motion vector of the current block can be derived. In this case, index information indicating the selected MVP candidate and information about the MVD can be signaled to the image decoding apparatus. Also, when the MVP mode is applied, a value of a reference picture index can be constructed as reference picture index information and separately signaled to the image decoding apparatus.

[0140] The image encoding apparatus can derive residual samples based on the prediction samples (S420). The image encoding apparatus can derive the residual samples through a comparison between original samples of the current block and the prediction samples. For example, the residual samples can be derived by subtracting corresponding prediction samples from the original samples.

[0141] The image encoding apparatus can encode image information including prediction information and residual information (S430). The image encoding apparatus can output the encoded image information in the form of a bitstream. The prediction information can include prediction mode information (e.g., a skip flag, a merge flag, or a mode index, etc.) and information on motion information as information related to a prediction process. Among the prediction mode information, the skip flag indicates whether a skip mode is applied to the current block, and the merge flag indicates whether a merge mode is applied to the current block. Alternatively, the prediction mode information can indicate one of a plurality of prediction modes, e.g., a mode index. When the skip flag and the merge flag are 0, it can be determined that the MVP mode is applied to the current block. The information on motion information can include candidate selection information (e.g., a merge index, an mvp flag, or an mvp index) as information for deriving a motion vector. Among the candidate selection information, the merge index can be signaled when the merge mode is applied to the current block, and can be information for selecting one of merge candidates included in a merge candidate list. Among the candidate selection information, the MVP flag or the MVP index can be signaled when the MVP mode is applied to the current block, and can be information for selecting one of MVP candidates in an MVP candidate list. Specifically, the MVP flag can be signaled using a syntax element mvp_10_flag or mvp_11_flag. In addition, the information on motion information can include information on the above-described MVD and / or reference picture index information. In addition, the information on motion information can include information indicating whether to apply L0 prediction, L1 prediction, or bi-prediction. The residual information is information on residual samples. The residual information can include information on quantized transform coefficients for the residual samples.

[0142] The output bitstream can be stored in a (digital) storage medium and transmitted to an image decoding apparatus or can be transmitted to an image decoding apparatus via a network.

[0143] As described above, the image encoding apparatus can generate a reconstructed picture (a picture including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is to enable the image encoding apparatus to derive the same prediction result as that performed by the image decoding apparatus, thereby improving encoding efficiency. Accordingly, the image encoding apparatus can store the reconstructed picture (or the reconstructed samples and the reconstructed blocks) in a memory and use it as a reference picture for inter prediction. As described above, the in-loop filtering process is also applied to the reconstructed picture.

[0144] FIG. 6 is a flowchart illustrating a video / image decoding method based on inter prediction.

[0145] FIG. 7 is a view illustrating a configuration of the inter prediction unit 260 according to the disclosure.

[0146] The image decoding device can perform operations corresponding to those performed by the image encoding device. The image decoding device can perform prediction for the current block based on the received prediction information and derive prediction samples.

[0147] FIG. 6 The decoding method of the image decoding device can be performed by FIG. 3 Steps S610 to S630 can be performed by the inter prediction unit 260, and the prediction information of step S610 and the residual information of step S640 can be obtained from the bitstream by the entropy decoder 210. The residual processor of the image decoding device can derive the residual samples of the current block based on the residual information (S640). Specifically, the dequantizer 220 of the residual processor can perform dequantization based on the quantized transform coefficients derived from the residual information to derive the transform coefficients, and the inverse transformer 230 of the residual processor can perform inverse transform on the transform coefficients to derive the residual samples of the current block. Step S650 can be performed by the adder 235 or the reconstructor.

[0148] Specifically, the image decoding device can determine the prediction mode of the current block based on the received prediction information (S610). The image decoding device can determine which inter prediction mode is applied to the current block based on the prediction mode information in the prediction information.

[0149] For example, it can be determined whether the skip mode is applied to the current block based on the skip flag. In addition, it can be determined whether the merge mode or the MVP mode is applied to the current block based on the merge flag. Alternatively, one of various inter prediction mode candidates can be selected based on the mode index. The inter prediction mode candidates can include the skip mode, the merge mode, and / or the MVP mode or can include various inter prediction modes to be described below.

[0150] The image decoding device can derive the motion information of the current block based on the determined inter prediction mode (S620). For example, when the skip mode or the merge mode is applied to the current block, the image decoding device can construct a merge candidate list to be described below, and select one of the merge candidates included in the merge candidate list. The selection can be performed based on the above-described candidate selection information (merge index). The motion information of the selected merge candidate can be used to derive the motion information of the current block. For example, the motion information of the selected merge candidate can be used as the motion information of the current block.

[0151] As another example, when the MVP mode is applied to the current block, the image decoding device can construct an MVP candidate list and use the motion vector of the MVP candidate selected from the MVP candidates included in the MVP candidate list as the MVP of the current block. Selection can be performed based on the aforementioned candidate selection information (MVP flag or MVP index). In this case, the MVD of the current block can be derived based on information about the MVD, and the motion vector of the current block can be derived based on the MVP and MVD of the current block. Additionally, the reference frame index of the current block can be derived based on reference frame index information. The frame indicated by the reference frame index in the reference frame list of the current block can be derived as the reference frame to be referenced for inter-frame prediction of the current block.

[0152] The image decoding device can generate a prediction sample for the current block based on the motion information of the current block (S630). In this case, a reference frame can be derived based on the reference frame index of the current block, and the prediction sample for the current block can be derived using samples of the reference block indicated by the motion vector of the current block on the reference frame. In some cases, a prediction sample filtering process can also be performed on all or some of the prediction samples of the current block.

[0153] For example, such as FIG. 7 As shown, the inter-frame prediction unit 260 of the image decoding device may include a prediction mode determination unit 261, a motion information derivation unit 262, and a prediction sample derivation unit 263. In the inter-frame prediction unit 260 of the image decoding device, the prediction mode determination unit 261 can determine the prediction mode of the current block based on the received prediction mode information, the motion information derivation unit 262 can derive the motion information (motion vector and / or reference frame index, etc.) of the current block based on the received motion information, and the prediction sample derivation unit 263 can derive the prediction samples of the current block.

[0154] The image decoding device can generate residual samples for the current block based on the received residual information (S640). The image decoding device can generate reconstructed samples for the current block based on the predicted samples and residual samples, and generate a reconstructed image based on this (S650). Thereafter, the in-loop filtering process is applied to the reconstructed image as described above.

[0155] As described above, the inter-frame prediction process may include the steps of determining an inter-frame prediction mode, deriving motion information based on the determined prediction mode, and performing prediction (generating prediction samples) based on the derived motion information. As described above, the inter-frame prediction process may be performed by an image encoding device and an image decoding device.

[0156] The steps for deriving motion information based on the prediction pattern will be described in more detail below.

[0157] As described above, inter prediction can be performed using motion information of a current block. The image encoding apparatus can derive the best motion information of the current block through a motion estimation process. For example, the image encoding apparatus can use an original block in an original picture of the current block in a fractional pixel unit to search for a similar reference block having high correlation in a predetermined search range in a reference picture, and use the same to derive motion information. The similarity of the blocks can be calculated based on a sum of absolute differences (SAD) between the current block and the reference block. In this case, the motion information can be derived based on the reference block having the smallest SAD in the search area. The derived motion information can be signaled to the image decoding apparatus according to various methods based on an inter prediction mode.

[0158] When the merge mode is applied to the current block, the motion information of the current block is not directly transmitted, and the motion information of the current block is derived using the motion information of the neighboring block. Accordingly, the motion information of the current prediction block can be indicated by transmitting flag information indicating that the merge mode is used and candidate selection information (e.g., a merge index) indicating which neighboring block is used as a merge candidate. In the present disclosure, since the current block is used as the same meaning as the current prediction block, and the neighboring block is used as the same meaning as the neighboring prediction block, the current block can be used as the same meaning as the current prediction block, and the neighboring block can be used as the same meaning as the neighboring prediction block.

[0159] The image encoding apparatus can search for a merge candidate block for deriving the motion information of the current block to perform the merge mode. For example, a maximum of five merge candidate blocks can be used, but is not limited thereto. The maximum number of the merge candidate blocks can be transmitted in a slice header or a tile group header, but is not limited thereto. After finding the merge candidate blocks, the image encoding apparatus can generate a merge candidate list and select a merge candidate block having the smallest RD cost as a final merge candidate block.

[0160] The present disclosure provides various embodiments for configuring merge candidate blocks of a merge candidate list. The merge candidate list can use, for example, five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate can be used.

[0161] FIG. 8 is a view exemplifying neighboring blocks available as spatial merge candidates.

[0162] FIG. 9 is a view schematically exemplifying a merge candidate list construction method according to an example of the present disclosure.

[0163] The image encoding / decoding apparatus can insert a spatial merge candidate derived by searching for a spatial neighboring block of the current block into the merge candidate list (S910). For example, as described above, the spatial merge candidate can be inserted into the merge candidate list in the order of the spatial neighboring block. FIG. 8As shown, spatial neighbor blocks can include the current block's lower-left neighbor block A0, left neighbor block A1, upper-right neighbor block B0, upper neighbor block B1, and upper-left neighbor block B2. However, this is just an example, and in addition to the spatial neighbor blocks described above, additional neighbor blocks such as right neighbor blocks, lower neighbor blocks, and lower-right neighbor blocks can be further used as spatial neighbor blocks. Image encoding / decoding devices can detect available blocks by searching spatial neighbor blocks based on priority and derive motion information of the detected blocks as spatial merging candidates. For example, the image encoding / decoding device can search in the order of A1, B1, B0, A0, and B2. FIG. 8 The five blocks shown are used to construct a list of merged candidates by indexing the available candidates in turn.

[0164] The image encoding / decoding device can insert time merging candidates derived by searching for time neighboring blocks of the current block into the merging candidate list (S920). Time neighboring blocks can be located on a reference screen different from the current screen in which the current block is located. The reference screen in which the time neighboring block is located can be called the juxtaposed screen or col screen. Time neighboring blocks can be searched in the order of the lower right corner neighboring block and the lower right center block of the juxtaposed block of the current block on the col screen. Furthermore, when applying motion data compression to reduce memory load, specific motion information can be stored as representative motion information of each predetermined storage unit of the col screen. In this case, it is not necessary to store the motion information of all blocks in the predetermined storage unit, thereby achieving the motion data compression effect. In this case, the predetermined storage unit can be predetermined as, for example, a 16×16 sample unit or an 8×8 sample unit, or the size information of the predetermined storage unit can be signaled from the image encoding device to the image decoding device. When applying motion data compression, the motion information of the time neighboring block can be replaced with representative motion information of the predetermined storage unit in which the time neighboring block is located. In other words, from an implementation perspective, in this case, temporal merging candidates can be derived based on the motion information of the prediction block that overwrites the prediction block at the arithmetically shifted position (rather than the prediction block located at the coordinates of the temporally neighboring block) after an arithmetic right shift of a predetermined value based on the coordinates (top-left sample position) of the temporally neighboring block. For example, when the predetermined storage unit is 2... n ×2 nWhen the predetermined storage unit is 16x16 sample units and the coordinates of the temporal neighboring block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb»n)<<n), (yTnb»n)<<n)) can be used for the temporal merge candidate. Specifically, for example, when the predetermined storage unit is 16x16 sample units and the coordinates of the temporal neighboring block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb»4)<<4), (yTnb»4)<<4)) can be used for the temporal merge candidate. Alternatively, for example, when the predetermined storage unit is 8x8 sample units and the coordinates of the temporal neighboring block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb»3)<<3), (yTnb»3)<<3)) can be used for the temporal merge candidate.

[0165] Referring again to FIG. 9 , the image encoding / decoding apparatus can check whether the current number of merge candidates is less than the maximum number of merge candidates (S930). The maximum number of merge candidates can be predefined or signaled from the image encoding apparatus to the image decoding apparatus. For example, the image encoding apparatus can generate information on the maximum number of merge candidates and encode the same, and transmit the encoded information to the image decoding apparatus in the form of a bitstream. When the maximum number of merge candidates is satisfied, a subsequent candidate addition process S940 can not be performed.

[0166] When the current number of merge candidates is less than the maximum number of merge candidates as a result of the check of step S930, the image encoding / decoding apparatus can derive additional merge candidates according to a predetermined method, and then insert the additional merge candidates into the merge candidate list (S940). For example, the additional merge candidates can include at least one of a history-based merge candidate, a pair-wise average merge candidate, an ATMVP, a combined bi-predictive merge candidate (when the slice / tile group type of the current slice / tile group is a B type), and / or a zero vector merge candidate.

[0167] When the current number of merge candidates is not less than the maximum number of merge candidates as a result of the check of step S930, the image encoding / decoding apparatus can end the construction of the merge candidate list. In this case, the image encoding apparatus can select a best merge candidate from among the merge candidates configuring the merge candidate list, and signal candidate selection information (e.g., a merge candidate index or a merge index) indicating the selected merge candidate to the image decoding apparatus. The image decoding apparatus can select the best merge candidate based on the merge candidate list and the candidate selection information.

[0168] As described above, the motion information of the selected merge candidate can be used as the motion information of the current block, and the prediction samples of the current block can be derived based on the motion information of the current block. The image encoding apparatus can derive the residual samples of the current block based on the prediction samples, and signal the residual information of the residual samples to the image decoding apparatus. As described above, the image decoding apparatus can generate the reconstructed samples based on the residual samples derived from the residual information and the prediction samples, and generate the reconstructed picture based thereon.

[0169] When the skip mode is applied to the current block, the motion information of the current block can be derived using the same method as in the case where the merge mode is applied. However, when the skip mode is applied, the residual signal of the corresponding block is omitted, and thus the prediction samples can be directly used as the reconstructed samples. For example, when the value of cu_skip_flag is 1, the skip mode described above can be applied.

[0170] Hereinafter, a method of deriving a spatial candidate in the merge mode and / or the skip mode will be described. The spatial candidate can denote the spatial merge candidate described above.

[0171] The derivation of the spatial candidate can be performed based on spatial neighboring blocks. For example, the spatial candidate can be derived from the candidate blocks existing at positions A0, B0, A1, and B1 illustrated in FIG. 7. The derivation of the spatial candidate can be performed based on the spatial neighboring blocks existing at positions A0, B0, A1, and B2 illustrated in FIG. 8. The derivation of the spatial candidate can be performed based on the spatial neighboring blocks existing at positions A0, B0, A1, and B2 illustrated in FIG. 9. FIG. 8 A maximum of four spatial candidates can be derived from the candidate blocks existing at the positions illustrated in FIG. 7. The order of deriving the spatial candidates can be A1->B1->B0->A0->B2. However, the order of deriving the spatial candidates is not limited to the above-described order, and can be, for example, B1->A1->B0->A0->B2. When at least one of the current four positions (A1, B1, B0, and A0 in the above-described example) is unavailable, the last position (position B2 in the above-described example) in the order can be considered. In this case, the block at the predetermined position which is unavailable can include a corresponding block belonging to a different slice or tile than the current block or a corresponding block which is an intra prediction block. When the spatial candidate is derived from the first position (A1 or B1 in the above-described example) in the order, a redundancy check can be performed on the spatial candidates of the subsequent positions. For example, when the motion information of the subsequent spatial candidate is the same as that of the spatial candidate already included in the merge candidate list, the subsequent spatial candidate can not be included in the merge candidate list, thereby improving coding efficiency. The redundancy check performed on the subsequent spatial candidate can be performed on some candidate pairs rather than all possible candidate pairs, thereby reducing computational complexity.

[0172] FIG. 10 is a view illustrating candidate pairs for which the redundancy check is performed on the spatial candidate.

[0173] In FIG. 10In the illustrated example, the redundancy check for the spatial candidate at position B0 can be performed only for the spatial candidate at position A0. In addition, the redundancy check for the spatial candidate at position B1 can be performed only for the spatial candidate at position B0. In addition, the redundancy check for the spatial candidate at position A1 can be performed only for the spatial candidate at position A0. Finally, the redundancy check for the spatial candidate at position B2 can be performed only for the spatial candidates at positions A0 and B0.

[0174] In FIG. 10 In the illustrated example, the order of deriving the spatial candidates is A0 -> B0 -> B1 -> A1 -> B2. However, the present disclosure is not limited thereto, and even if the order of deriving the spatial candidates is changed, as in FIG. 10 In the illustrated example, the redundancy check can be performed only for some candidate pairs.

[0175] Hereinafter, a method of deriving a temporal candidate in the case of a merge mode and / or a skip mode will be described. The temporal candidate can denote the above-described temporal merge candidate. In addition, the motion vector of the temporal candidate can correspond to the temporal candidate of the MVP mode.

[0176] In the case of the temporal candidate, only one candidate can be included in the merge candidate list. In the process of deriving the temporal candidate, the motion vector of the temporal candidate can be scaled. For example, the scaling can be performed based on a collocated block (CU) (hereinafter, referred to as a "col block") belonging to a collocated reference picture (colPic) (hereinafter, referred to as a "col picture"). The reference picture list for deriving the col block can be explicitly signaled in a slice header.

[0177] FIG. 11 is a view illustrating a method of scaling the motion vector of the temporal candidate.

[0178] In FIG. 11 In the above-described Equation 1, curr_CU and curr_pic denote a current block and a current picture, respectively, and col_CU and col_pic denote a col block and a col picture, respectively. In addition, curr_ref denotes a reference picture of the current block, and col_ref denotes a reference picture of the col block. In addition, tb denotes a distance between the reference picture of the current block and the current picture, and td denotes a distance between the reference picture of the col block and the col picture. The tb and td can denote values corresponding to a difference in POC (Picture Order Count) between pictures. The scaling of the motion vector of the temporal candidate can be performed based on the tb and td. In addition, the reference picture index of the temporal candidate can be set to 0.

[0179] FIG. 12 is a view illustrating a position of deriving the temporal candidate.

[0180] In FIG. 12In the text, the thick solid line indicates the current block. This can be seen from the `col` view. FIG. 12 Timing candidates are derived for the block corresponding to position C0 (bottom right) or C1 (center). First, it can be determined whether position C0 is available, and if position C0 is available, timing candidates can be derived based on position C0. If position C0 is unavailable, timing candidates can be derived based on position C1. For example, if the block at position C0 in the col frame is an intra-predicted block or is located outside the current CTU line, it can be determined that position C0 is unavailable.

[0181] As described above, when applying motion data compression, motion vectors for each predetermined cell block can be stored. In this case, to derive the motion vector of the block covering position C0 or position C1, position C0 or position C1 can be modified. For example, when the predetermined cell block is an 8×8 block and position C0 or position C1 is (xColCi, yColCi), the position used to derive the time candidate can be modified to ((xColCi>>3)<<3, (yColCi>>3)<<3).

[0182] The following describes a method for deriving history-based candidates in merge and / or skip modes. History-based candidates can be represented by history-based merge candidates.

[0183] After spatial and temporal candidates are added to the merge candidate list, history-based candidates can be added. For example, motion information from previously encoded / decoded blocks can be stored in a table and used as a history-based candidate for the current block. This table can store multiple history-based candidates during encoding / decoding. The table can be initialized at the start of a new CTU row. Initializing the table can mean clearing the corresponding table by deleting all history-based candidates stored in it. Whenever an inter-frame prediction block exists, the relevant motion information can be added to the table as the last entry. In this case, the inter-frame prediction block may not be a block predicted based on sub-blocks. The motion information added to the table can be used as a new history-based candidate.

[0184] The table for history-based candidates can have a predetermined size. For example, this size could be 5. In this case, the table can store up to five history-based candidates. When a new candidate is added to the table, a finite first-in-first-out (FIFO) rule is applied to check for redundancy by checking if the same candidate already exists in the table. If the same candidate already exists in the table, it can be removed from the table, and the positions of all subsequent history-based candidates can be shifted forward.

[0185] A history-based candidate can be used in a process of configuring a merge candidate list. In this case, a history-based candidate recently included in a table can be checked in order and positioned after a temporal candidate of the merge candidate list. When a history-based candidate is included in the merge candidate list, a redundancy check with a spatial candidate or a temporal candidate already included in the merge candidate list can be performed. If a spatial candidate or a temporal candidate already included in the merge candidate list overlaps with the history-based candidate, the history-based candidate can not be included in the merge candidate list. The amount of calculation can be reduced by simplifying the redundancy check as follows.

[0186] The number of history-based candidates used to generate a merge candidate list can be set to (N<=4)?M:(8-N). In this case, N can denote the number of candidates already included in the merge candidate list, and M can denote the number of available history-based candidates included in a table. That is, when 4 or less candidates are included in the merge candidate list, the number of history-based candidates used to generate the merge candidate list can be M, and when N candidates greater than 4 are included in the merge candidate list, the number of history-based candidates used to generate the merge candidate list can be set to (8-N).

[0187] When the total number of available merge candidates reaches (the maximum allowed number of merge candidates - 1), the configuration of the merge candidate list using the history-based candidate can end.

[0188] Hereinafter, a method of deriving a pair-wise average candidate in the case of a merge mode and / or a skip mode will be described. The pair-wise average candidate can be represented by a pair-wise average merge candidate or a pair candidate.

[0189] The pair-wise average candidate can be generated by obtaining a predefined pair of candidates from the candidates included in the merge candidate list and averaging them. The predefined pair of candidates can be {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)} and the number of configuring each pair of candidates can be the index of the merge candidate list. That is, the predefined pair of candidates (0, 1) can mean a pair of index 0 candidate and index 1 candidate of the merge candidate list, and the pair-wise average candidate can be generated by the average of the index 0 candidate and the index 1 candidate. The derivation of the pair-wise average candidate can be performed in the order of the predefined pair of candidates. That is, after deriving the pair-wise average candidate of the pair of candidates (0, 1), the process of deriving the pair-wise average candidate can be performed in the order of the pair of candidates (0, 2) and the pair of candidates (1, 2). The pair-wise average candidate derivation process can be performed until the configuration of the merge candidate list is completed. For example, the pair-wise average candidate derivation process can be performed until the number of merge candidates included in the merge candidate list reaches the maximum number of merge candidates.

[0190] Pairwise averaging candidates can be calculated individually for each list of reference frames. When two motion vectors are available for a list of reference frames (L0 list or L1 list), the average of the two motion vectors can be calculated. In this case, the averaging of the two motion vectors can be performed even if the two motion vectors point to different reference frames. If only one motion vector is available for a list of reference frames, the available motion vector can be used as the motion vector for the pairwise averaging candidate. If neither motion vector is available for a list of reference frames, the list of reference frames can be determined to be invalid.

[0191] Even if the configuration of the merge candidate list is not completed after the pairwise average candidate has been included in the merge candidate list, the zero vector can be added to the merge candidate list until the maximum number of merge candidates is reached.

[0192] When applying the MVP pattern to the current block, spatially adjacent blocks of the reconstructed block can be used (e.g., FIG. 8 The motion vectors of the spatially neighboring blocks (as shown) and / or the motion vectors corresponding to the temporally neighboring blocks (or Col blocks) are used to generate a motion vector predictor (MVP) candidate list. That is, the motion vectors of the reconstructed spatially neighboring blocks and the motion vectors corresponding to the temporally neighboring blocks can be used as MVP candidates for the current block. When bidirectional prediction is applied, MVP candidate lists for L0 motion information derivation and MVP candidate lists for L1 motion information derivation are generated and used separately. The prediction information (or information about the prediction) for the current block can include candidate selection information (e.g., MVP flag or MVP index) indicating the best MVP candidate to be selected from the MVP candidate candidates included in the MVP candidate list. In this case, the prediction unit can use the candidate selection information to select the MVP for the current block from the MVP candidate candidates. The prediction unit of the image coding device can obtain and encode the motion vector difference (MVD) between the motion vectors of the current block and the MVP, and output the encoded MVD as a bitstream. That is, the MVD can be obtained by subtracting the MVP from the motion vectors of the current block. The prediction unit of the image decoding device can obtain the motion vector difference included in the information about the prediction and deduce the motion vector of the current block by adding the motion vector difference to the motion vector predictor. The prediction unit of the image decoding device can obtain or deduce the reference frame index indicating the reference frame from the information about the prediction.

[0193] FIG. 13 This is a schematic illustration of a method for constructing a list of motion vector prediction sub-candidates according to an example of this disclosure.

[0194] First, spatial candidate blocks of the current block can be searched and available candidate blocks can be inserted into the MVP candidate list (S1010). Thereafter, it is determined whether the number of MVP candidates included in the MVP candidate list is less than 2 (S1020), and when the number of MVP candidates is 2, the construction of the MVP candidate list can be completed.

[0195] In step S1020, when the number of available spatial candidate blocks is less than 2, temporal candidate blocks of the current block can be searched and available candidate blocks can be inserted into the MVP candidate list (S1030). When temporal candidate blocks are not available, a zero motion vector can be inserted into the MVP candidate list (S1040), thereby completing the construction of the MVP candidate list.

[0196] Further, when the mvp mode is applied, a reference picture index can be explicitly signaled. In this case, a reference picture index refidxL0 for L0 prediction and a reference picture index refidxL1 for L1 prediction can be differentially signaled. For example, when the MVP mode is applied and bi-prediction is applied, information about refidxL0 and information about refidxL1 can be signaled.

[0197] As described above, when the MVP mode is applied, information about the MVP derived by the image encoding apparatus can be signaled to the image decoding apparatus. For example, information about the MVD can include an MVD absolute value and information indicating x and y components for a sign. In this case, when the MVD absolute value is greater than 0, whether the MVD absolute value is greater than 1 and information indicating an MVD remainder can be signaled stepwise. For example, the information indicating whether the MVD absolute value is greater than 1 can be signaled only when the value of flag information indicating whether the MVD absolute value is greater than 0 is 1.

[0198] Hereinafter, an embodiment of a prediction execution method of the present disclosure will be described. The following prediction execution method can be performed in step S410 of the method of FIG. 4 or step S630 of the method of FIG. 6. FIG. 4 FIG. 6 The following prediction execution method can be performed in step S410 of the method of FIG. 4 or step S630 of the method of FIG. 6.

[0199] ​A prediction block for the current block can be generated based on motion information derived from a prediction pattern. The prediction block (the predicted block) can include prediction samples (an array of prediction samples) for the current block. When the motion vector of the current block specifies partial sample units, an interpolation process can be performed, and thus, prediction samples for the current block can be derived based on reference samples, using partial samples within the reference frame as units. When affine inter-frame prediction is applied to the current block, prediction samples can be generated based on sample / sub-block units MV. When dual prediction is applied, prediction samples derived by a weighted sum or weighted average (based on phase) of prediction samples derived from L0 prediction (i.e., using MVL0 and predictions of reference frames within the reference frame list L0) and prediction samples derived from L1 prediction (i.e., using MLV1 and predictions of reference frames within the reference frame list L1) can be used as prediction samples for the current block. When dual prediction is applied and the reference frames used for L0 prediction and L1 prediction are located in different time directions relative to the current frame (i.e., if it corresponds to dual prediction and bidirectional prediction), this can be called true dual prediction.

[0200] In an image decoding device, reconstructed samples and reconstructed images can be generated based on derived prediction samples, and then an in-loop filtering process can be performed. Conversely, in an image encoding device, residual samples can be derived based on the derived prediction samples, and image information including both prediction and residual information can be encoded.

[0201] Bi-directional optical flow (BDOF)

[0202] According to this disclosure, BDOF can be used to refine dual prediction signals. When dual prediction is applied to the current block (e.g., CU), BDOF generates prediction samples by calculating refined motion information. Therefore, the process of calculating refined motion information by applying BDOF can be included in the motion information derivation steps described above.

[0203] For example, BDOF can be applied at the 4×4 sub-block level. That is, BDOF can be performed within the current block in units of 4×4 sub-blocks.

[0204] For example, BODF can be applied to CUs that satisfy at least one or all of the following conditions.

[0205] -CU uses true dual-predictive mode encoding, meaning that one of the two reference frames is displayed before the current frame in the order of display, and the other is displayed after the current frame in the order of display.

[0206] -CU is not in affine mode or ATMVP merge mode

[0207] -CU has 64 or more luminance samples

[0208] - height and width of the CU is 8 or more luma samples

[0209] - the BCW weight index specifies equal weights, i.e. equal weights are applied to the L0 prediction samples and the L1 prediction samples

[0210] - weighted prediction (WP) is not applied to the current CU

[0211] - CIIP mode is used for the current CU

[0212] In addition, the BDOF can be applied only to the luma component. However, the present disclosure is not limited thereto, and the BDOF can be applied to the chroma component or both the luma component and the chroma component.

[0213] The BDOF mode is based on the concept of optical flow. That is, it is assumed that the motion of an object is smooth. When the BDOF is applied, for each 4x4 sub-block, motion refinements (v x ,v y ) can be calculated. The motion refinements can be calculated by minimizing the difference between the L0 prediction samples and the L1 prediction samples. The motion refinements can be used to adjust the bi-prediction sample values within the 4x4 sub-block.

[0214] Hereinafter, the process of performing the BDOF will be described in more detail.

[0215] First, horizontal gradients and vertical gradients of the two prediction signals can be calculated. In this case, k can be 0 or 1. The gradients can be calculated by directly calculating the difference between two adjacent samples. For example, the gradients can be calculated as follows.

[0216] [Equation 1]

[0217]

[0218] In the above Equation 1, I (k) (i,j) denotes a sample value of a coordinate (i,j) of a prediction signal in a list k (k=0,1). For example, I (0) (i,j) can denote a sample value at a position (i,j) in the L0 prediction block, I (1) (i,j) can denote a sample value at a position (i,j) in the L1 prediction block. In the above Equation 1, a first shift shift1 can be determined based on a bit depth of the luma component. For example, when the bit depth of the luma component is bitDepth, shift1 can be determined as max(6,bitDepth-6).

[0219] As described above, after the gradients are calculated, the autocorrelations and cross-correlations S1, S2, S3, S5, and S6 between the gradients can be calculated as follows.

[0220] [Formula 2]

[0221]

[0222] wherein,

[0223]

[0224] θ(i,j) = (I (1) (i,j) » n b ) - (I (0) (i,j) » n b )

[0225] wherein, Ω is a 6x6 window around the 4x4 sub-block.

[0226] In the above Formula 2, n a and n b may be set to min(1, bitDepth-11) and min(4, bitDepth-8), respectively.

[0227] The motion refinement (v x , v y ) can be derived using the above auto-correlation and cross-correlation between gradients as follows.

[0228] [Formula 3]

[0229]

[0230] wherein, S_(2,s) = S_2 & (2^(n_(S_2))-1), th ′ BIO = 2 13-BD . And is a floor function.

[0231] In the above Formula 3, n S2 may be 12. Based on the derived motion refinement and gradients, the following adjustment can be performed for each sample in the 4x4 sub-block.

[0232] [Formula 4]

[0233]

[0234] Finally, the prediction sample pred BDOF of the CU to which BDOF is applied can be calculated by adjusting the bi-prediction samples of the CU as follows.

[0235] [Formula 5]

[0236] pred BDOF (x,y) = (I (0)(x, y) + I (1) (x, y) + b(x, y) + o offset ) » shift

[0237] In the above equation, n a , n b , and n S2 may be 3, 6, and 12, respectively. These values can be selected such that the multiplier does not exceed 15 bits and the bit width of the intermediate parameter is maintained within 32 bits in the BDOF process.

[0238] To derive gradient values, prediction samples I (k) (i, j) existing outside the current CU in the list k (k = 0, 1) can be generated. FIG. 14 is a view illustrating a CU to which the extension is performed to perform BDOF.

[0239] As shown in FIG. 14 , to perform BDOF, rows / columns extended around the boundary of the CU can be used. To control the computational complexity of generating prediction samples outside the boundary, prediction samples in the white region in the extension region ( FIG. 14 ) can be generated using a bilinear filter, and prediction samples in the gray region in the CU ( FIG. 14 ) can be generated using a normal 8-tap motion compensation interpolation filter. Sample values at the extension positions can be used only for gradient calculation. When sample values and / or gradient values located outside the CU boundary are needed to perform the remaining steps of the BDOF process, the nearest neighboring sample values and / or gradient values can be padded (repeated) and used.

[0240] When the width and / or height of the CU is greater than 16 luma samples, the corresponding CU can be divided into sub-blocks having a width and / or height of 16 luma samples. The boundaries of the sub-blocks can be treated in the same way as the above-described CU boundaries in the BDOF process. The maximum unit size to perform the BDOF process can be limited to 16x16.

[0241] For each sub-block, it can be determined whether to perform BDOF. That is, the BDOF process for each sub-block can be skipped. For example, when the SAD value between the initial L0 prediction sample and the initial L1 prediction sample is less than a predetermined threshold, the BDOF process can not be applied to the corresponding sub-block. In this case, when the width and height of the corresponding sub-block are W and H, the predetermined threshold can be set to (8*W*(H>>1). Considering the complexity of additional SAD calculation, the SAD between the initial L0 prediction sample and the initial L1 prediction sample calculated in the DMVR process can be reused.

[0242] BDOF can not be applied when WP is available for the current block, e.g., when luma_weight_lx_flag of at least one of two reference pictures is 1. In this case, luma_weight_lx_flag can be information that specifies whether a weighting factor for a luma component of lx prediction (x is 0 or 1) exists in a bitstream or information that specifies whether WP is applied to the luma component of lx prediction. BDOF can not be applied when a CU is coded in a symmetric MVD (SMVD) mode or a CIIP mode.

[0243] As described above, by applying BDOF in inter prediction processing to refine reference samples in motion compensation processing, compression performance of an image can be increased. BDOF can be performed in a normal mode. That is, BDOF is not performed in the case of an affine mode, a GPM mode, or a CIIP mode.

[0244] The present disclosure proposes various methods capable of preventing potential errors of BDOF and improving performance by applying normalization and clipping when deriving a BDOF offset bdofOffset(b(x, y)) for reference sample refinement in BDOF processing. In the present disclosure, normalization can mean that a value expressed in various units (e.g., 1 / 64 pel, 1 / 32 pel, 2 pel, etc.) is unified to a value of a predetermined unit (e.g., 1 pel). In addition, in the present disclosure, [a, b] can mean a range of values from a to b, and clipping a specific value x in the range [a, b] can mean that the range of x is limited to have a value a when x is smaller than a, to have a value b when x is greater than b, and to have a value x in other cases. In addition, in the present disclosure, bit depth is not limited to bit depth of a luma component, and can include, for example, bit depth when bit depths of a luma component and a chroma component are the same.

[0245] FIG. 15 is a view illustrating a process of deriving prediction samples of a current block by applying BDOF.

[0246] FIG. 15 The BDOF-based inter prediction process of can be performed by an image encoding apparatus and an image decoding apparatus.

[0247] First, in step S1510, motion information of a current block can be derived. The motion information of the current block can be derived by various methods described in the disclosure. For example, the motion information of the current block can be derived by a regular merge mode, an MMVD mode, or an AMVP mode. The motion information can include bi-prediction motion information (L0 motion information and L1 motion information). For example, the L0 motion information can include MVL0 (L0 motion vector) and refldxL0 (L0 reference picture index), and the L1 motion information can include MVL1 (L1 motion vector) and refldxL1 (L1 reference picture index).

[0248] Thereafter, predicted samples of the current block can be derived based on the motion derivation information of the current block (S1520). Specifically, L0 predicted samples of the current block can be derived based on the L0 motion information. In addition, L1 predicted samples of the current block can be derived based on the L1 motion information.

[0249] Thereafter, a BDOF offset can be derived based on the derived predicted samples (S1530). The BDOF of step S1530 can be performed according to the methods described in the disclosure. For example, the BDOF offset can be derived based on a gradient (according to a phase) of the L0 predicted samples and a gradient (according to a phase) of the L1 predicted samples.

[0250] Thereafter, based on the LX (X=0 or 1) predicted samples and the BDOF offset, refined predicted samples of the current block can be derived (S1540). The refined predicted samples can be used to generate a final prediction block of the current block.

[0251] The image encoding apparatus can generate a predicted sample of a current block based on a method according to FIG. 15 The residual sample is derived by comparison with the original sample. As described above, information about the residual sample (residual information) can be included in the image / video information and encoded and output in the form of a bitstream. In addition, as described above, the image decoding apparatus can generate a reconstructed current block based on the predicted sample of the current block generated according to the method according to FIG. 15

[0252] FIG. 16 is a view illustrating input and output of a BDOF process according to an embodiment of the disclosure.

[0253] As described above, the image encoding apparatus can generate a predicted sample of a current block based on a method according to FIG. 16 ​As shown, the input to BDOF processing can include the width nCbW and height CbH of the current block, the predicted sub-blocks predSamplesL0 and predSamplesL1 with a predetermined length (e.g., 2) of the boundary region extension, the prediction direction indices predFlagL0 and predFlagL1, and the reference screen indices refIdxL0 and refIdxL1. Additionally, the input to BDOF processing can also include the BDOF utilization flag bdofUtilizationFlag. In this case, the BDOF utilization flag can be an input specifying whether BDOF is applied to the corresponding sub-block on a per-sub-block basis within the current block.

[0254] In addition, BDOF processing can generate refined prediction blocks pbSamples by applying BDOF based on the input information.

[0255] FIG. 17 This is a view illustrating variables used for BDOF processing according to embodiments of this disclosure. FIG. 17 It can be FIG. 16 Subsequent processing.

[0256] like FIG. 17 As shown, in order to perform BDOF processing, the input bit depth bitDepth of the current block can be set to BitDepth. Y In this case, BitDepth Y It can be derived based on information about the bit depth signaled via the bit stream. Furthermore, various right shifts can be set based on the bit depth. For example, the first shift (shift1), second shift (shift2), third shift (shift3), and fourth shift (shift4) can be set as follows: FIG. 17 The derivation shown is based on bit depth. Additionally, the offset 4 can be set based on shift4. Furthermore, the variable mvRefineThres, used to specify the limiting range for motion refinement, can be set based on bit depth. This will be described below. FIG. 17 The uses of the various variables described in the text.

[0257] FIG. 18 This is a view illustrating a method for generating prediction samples of each sub-block in the current CU based on whether or not BDOF is applied, according to an embodiment of the present disclosure. FIG. 18 It can be FIG. 17 Subsequent processing.

[0258] It can execute on each sub-block in the current CU. FIG. 18The processing shown, and in this case, the size of the sub-block can be 4x4. When the BDOF utilization flag bdofUtilizationFlag for the current sub-block is a first value (false, "0"), BDOF can not be applied to the current sub-block. In this case, the prediction samples of the current sub-block are derived from a weighted sum of the L0 prediction samples and the L1 prediction samples, and in this case, the weight applied to the L0 prediction samples and the weight applied to the L1 prediction samples can be the same. FIG. 18 The shift4 and offset4 used in the formula (1) can be FIG. 17 The values set in the formula (1) and the formula (2). When the BDOF utilization flag bdofUtilizationFlag for the current sub-block is a second value (true, "1"), BDOF can be applied to the current sub-block. In this case, the prediction samples of the current sub-block can be generated by the BDOF processing according to the present disclosure.

[0259] FIG. 19 is a view illustrating a method of deriving a gradient, an autocorrelation, and a cross-correlation of a current sub-block according to an embodiment of the present disclosure. FIG. 19 The processing shown, and in this case, the size of the sub-block can be 4x4. FIG. 18 The processing shown, and in this case, the size of the sub-block can be 4x4.

[0260] The processing shown, and in this case, the size of the sub-block can be 4x4. FIG. 19 The processing shown, and in this case, the size of the sub-block can be 4x4.

[0261] According to FIG. 19 , according to the formula (1) and the formula (2), the position (h x ,h y ) of each sample position (x, y) in the current sub-block can be derived. Thereafter, the horizontal gradient and the vertical gradient of each sample position can be derived according to the formula (3) to the formula (6). Thereafter, the variables (the first intermediate parameter diff and the second intermediate parameters tempH and tempV) used to derive the autocorrelation and the cross-correlation can be derived according to the formula (7) to the formula (9). For example, the first intermediate parameter diff can be derived using a value obtained by applying a right shift of a second shift shift2 to the prediction samples predSamplesL0 and predSamplesL1 of the current block. For example, the second intermediate parameters tempH and tempV can be derived by applying a right shift of a third shift shift3 to the sum of the gradient in the L0 direction and the gradient in the L1 direction as in the formula (8) and the formula (9). Thereafter, the autocorrelation and the cross-correlation can be derived based on the derived first intermediate parameter and the second intermediate parameters according to the formula (10) to the formula (16).

[0262] FIG. 20 is a view illustrating a method of deriving a motion refinement (v x ,v yview of the method of deriving the BDOF offset and generating the prediction samples of the current sub-block. FIG. 20 may be FIG. 19 the subsequent processing.

[0263] performing the method described with reference to FIG. 20 FIG. 6, and in this case, the size of the sub-block can be 4x4.

[0264] According to Equation 20, the motion refinement (v x ,v y ) can be derived according to Equations (1) and (2). The motion refinement can be clipped within a range specified by mvRefineThres. In addition, based on the motion refinement and the gradient, the BDOF offset bdofOffset can be derived according to Equation (3). The derived BDOF offset can be used to generate the prediction samples pbSamples of the current sub-block according to Equation (4).

[0265] By continuously performing the method described with reference to FIG. 16 to FIG. 20 , the BDOF processing according to the first embodiment of the present disclosure can be implemented. In the embodiment according to FIG. 16 to FIG. 20 , the first shift shiftl is set to Max(6, bitDepth-6), and mvRefineThres is set to 1 « Max(, bitDepth-7). Therefore, the bit width of predSample and each parameter of the BDOF according to BitDepth can be derived as shown in the following table.

[0266] [Table 1]

[0267]

[0268]

[0269] In Table 1 above, for example, when BitDepth is 8, predSample has a value with a range of 16 bits, the gradient uses 11 bits, v x and v y use 6 bits, and as a result, the range of the bdofOffset value is [-49856, 48298]. As shown in Table 1 above, the bit width of predSample changes as BitDepth changes. However, even when BitDepth changes, the gradient highly associated with BitDepth has a fixed bit width (11 bits). In addition, the bit width of v x and v y that are not associated with BitDepth changes as BitDepth changes.

[0270] According to other embodiments of the present disclosure, by refining normalization and clipping of parameters used in the BDOF process, the correlation with the parameters and BitDept can be more accurately reflected. Thus, the parameters can have more accurate values and the memory overflow problem in the BDOF process can be solved.

[0271] Other embodiments of the present disclosure can be implemented by changing parts of the first embodiment of the present disclosure described above with reference to FIG. 16 to FIG. 20 .

[0272] Hereinafter, the BDOF process according to the second embodiment of the present disclosure will be described.

[0273] The second embodiment of the present disclosure applies a different normalization method from the BDOF process of the first embodiment of the present disclosure. According to the present disclosure, the gradients gradientHLXand gradientVLX(in this case, X is 0 or 1) represent the slopes at a 2-pixel distance in the horizontal direction and the vertical direction of the current sample position. In addition, when v x and v y are 1 / 32 pel precision and have a range of values of [-32, 31] or [-32, 32], v x and v y of the value 1 represent an actual 1 / 32 pel distance. Thus, v x and v y can be regarded as applying a "1<<5" operation to a 1-pixel unit value.

[0274] According to the second embodiment of the present disclosure, the parameters (horizontal gradient, vertical gradient, v x and v y ) used to calculate the BDOF offset can be normalized to a 1-pixel unit value. For example, for the gradient which is the slope at a 2-pixel distance, normalization to a 1-pixel unit value can be performed by applying a ">>1" operation. In addition, for v x and v y of 1 / 32 pel precision, normalization to a 1-pixel unit value can be performed by applying a ">>5" operation. In consideration of this, as shown in Equation (3) of v FIG. 23 , in order to normalize, the value obtained by multiplying the gradient by v x and v y can be right-shifted by a first shift shift1. In this case, shift1 can be set to a fixed value (for example, 7) regardless of the bit depth. In addition, v x and v y can be clipped within a range of values set regardless of the bit depth. For example, v x and v yThe variable mvRefineThres, which is a limit range of clipping, can be set to a value of "1<<5". As described above, the normalization according to the second embodiment of the present disclosure can be performed by considering the gradient and v x and v y .

[0275] The second embodiment of the present disclosure can be implemented by improving the first embodiment of the present disclosure FIG. 17 , FIG. 19 and FIG. 20 .

[0276] FIG. 21 is a view illustrating variables for BDOF processing according to another embodiment of the present disclosure. FIG. 21 may be an example obtained by modifying the example of FIG. 17 . Accordingly, the description of common parts in FIG. 17 and FIG. 21 will be omitted.

[0277] As shown in FIG. 21 , in order to perform the BDOF processing, the input bit depth bitDepth of the current block can be set to BitDepth Y . In this case, BitDepth Y may be derived based on information about the bit depth signaled through a bitstream. In addition, regardless of the bit depth, the first shift shift1, the second shift shift2, and the third shift shift3 can be set to fixed values. For example, the first shift shift1, the second shift shift2, and the third shift shift3 can be set to 7, 4, and 1, respectively. The fourth shift shift4 and the offset offset4 can be derived in the same manner as the example of FIG. 17 . In addition, regardless of the bit depth, the variable mvRefineThres can be set to a fixed value. For example, the variable mvRefineThres can be set to "1<<5".

[0278] FIG. 22 is a view illustrating a method of deriving the gradient, the autocorrelation, and the cross-correlation of the current sub-block according to another embodiment of the present disclosure. FIG. 22 may be an example obtained by modifying the example of FIG. 19 . Accordingly, the description of common parts in FIG. 19 and FIG. 22 will be omitted.

[0279] Compared to the example of FIG. 19 , in the example of FIG. 22In the gradient calculation of Formula (3) to Formula (6), the right shift operation (">>shift1") can not be performed. According to the present embodiment, the gradient can be obtained with higher accuracy by omitting the right shift operation.

[0280] FIG. 23 is an example of deriving a motion refinement (v x ,v y ), deriving a BDOF offset, and generating a predicted sample of a current sub-block according to another embodiment of the present disclosure. FIG. 23 may be an example obtained by modifying FIG. 20 . Thus, FIG. 20 and FIG. 23 will be omitted.

[0281] compared to the example of FIG. 20 , in the calculation of bdofOffset of Formula (3) of FIG. 23 , the right shift operation ">>1" can be changed to ">>shift1". That is, as described above, for the gradient and the normalization of v x and v y , the value obtained by multiplying the gradient by v x and v y can be right shifted by a first shift shift1. In this case, as described above, regardless of the bit depth, shift1 can be set to a fixed value (for example, 7).

[0282] As described above, by modifying the example according to the first embodiment of the present disclosure of FIG. 21 , FIG. 22 and FIG. 23 as shown in FIG. 17 , FIG. 19 and FIG. 20 , the second embodiment of the present disclosure can be implemented.

[0283] According to the second embodiment of the present disclosure, the bit width of predSample and each parameter of BDOF according to BitDepth can be derived as shown in the following table.

[0284] [Table 2]

[0285]

[0286] As shown in Table 2 above, the gradient highly associated with the bit depth changes according to the bit depth. In addition, since the range of the gradient value is determined based on the predSample value, the accuracy of the gradient value can be increased. In addition, regardless of the bit depth, v x and v yThe bit width can have a fixed value. According to Table 2, instead of increasing the range of gradient values, since v x and v y The range of values ​​decreases, so it will not affect the range of bdofOffset values.

[0287] The BDOF processing according to the third embodiment of this disclosure will be described below.

[0288] As described above in the second embodiment of this disclosure, the gradient and v x and v y It can be normalized to a 1-pixel unit value. However, according to FIG. 22 In this implementation method, bit overflow may occur during gradient calculation processing. For example, as... FIG. 22 As shown in equations (3) to (6), when no shift operation is performed in the gradient calculation process, a 32-bit operation can be performed to calculate the gradient. That is, bit overflow may occur when calculating the gradient.

[0289] In the third embodiment of this disclosure, taking this into consideration, gradient normalization is applied when calculating the gradient. That is, normalization can be performed by applying the ">>1" operation when calculating the gradient, thereby preventing bit overflow. According to the third embodiment of this disclosure, the gradient may not exceed 16 bits. Furthermore, a right shift operation can be performed by applying the adjusted shift1 to normalize v. x and v y Normalization. For example, shift1 can be set to a fixed value (e.g., 6) regardless of the bit depth.

[0290] The third embodiment of this disclosure can be improved upon the second embodiment of this disclosure. FIG. 21 and FIG. 22 To achieve this.

[0291] FIG. 24 This is a view illustrating variables for BDOF processing according to another embodiment of this disclosure. FIG. 24 It can be done by modifying FIG. 21 The example obtained from the example. Therefore, FIG. 21 and FIG. 24 The description of the common parts will be omitted.

[0292] like FIG. 24 As shown, in order to perform BDOF processing, the input bit depth bitDepth of the current block can be set to BitDepth. Y In this case, BitDepth YThis can be derived based on information about the bit depth signaled via the bitstream. Furthermore, regardless of the bit depth, the first shift (shift1), second shift (shift2), and third shift (shift3) can be set to fixed values. For example, the first shift (shift1), second shift (shift2), and third shift (shift3) can be set to 6, 4, and 1, respectively. The fourth shift (shift4) and offset (offset4) can be adjusted according to... FIG. 17 The example is derived in the same way. Furthermore, regardless of the bit depth, the variable `mvRefineThres` can be set to a fixed value. For example, the variable `mvRefineThres` can be set to "1<<5".

[0293] FIG. 25 This is a view illustrating a method for deriving the gradient, autocorrelation, and cross-correlation of the current sub-block according to another embodiment of this disclosure. FIG. 25 It can be done by modifying FIG. 22 The example obtained from the example. Therefore, FIG. 22 and FIG. 25 The description of the common parts will be omitted.

[0294] and FIG. 22 Compared to the example, in FIG. 25 In the gradient calculation of equations (3) to (6), a right shift operation (">>1") can be performed. By performing a right shift operation in the gradient calculation process, bit overflow can be prevented.

[0295] In addition, such as FIG. 24 As shown, since shift1 is modified to 6, it is applied to the gradient and v. x and v y The total right shift is "7". Therefore, the total right shift in the third embodiment and the second embodiment of this disclosure can remain equal.

[0296] As described above, through such FIG. 24 and FIG. 25 The modifications shown are based on the second embodiment of this disclosure. FIG. 21 and FIG. 22 Examples of these embodiments can be used to implement the third implementation of this disclosure.

[0297] According to the third embodiment of this disclosure, the bit width of predSample and the various parameters of BDOF based on BitDepth can be derived as shown in the table below.

[0298] [Table 3]

[0299]

[0300] As shown in Table 3 above, the gradient highly correlated with bit depth changes according to the bit depth. Furthermore, since the range of gradient values ​​is determined based on the predSample value, the accuracy of the gradient values ​​can be increased. Additionally, by executing... FIG. 25 The right shift operation in equations (3) to (6) can prevent bit overflow when calculating the gradient. Furthermore, regardless of the bit depth, v, which is not related to the bit depth... x and v y The bit width can have a fixed value.

[0301] The BDOF processing according to the fourth embodiment of this disclosure will be described below.

[0302] As described above in the second embodiment of this disclosure, the gradient and v x and v y It can be normalized to a 1-pixel unit value. However, according to FIG. 22 In this implementation method, bit overflow may occur during gradient calculation processing. For example, as... FIG. 22 As shown in equations (3) to (6), when no shift operation is performed in the gradient calculation process, a 32-bit operation can be performed to calculate the gradient. That is, bit overflow may occur when calculating the gradient.

[0303] In the fourth embodiment of this disclosure, taking this into consideration, bit overflow can be prevented by performing clipping when calculating the gradient. According to the fourth embodiment of this disclosure, the gradient may not exceed 16 bits. Furthermore, the gradient and v can be processed in the same manner as in the second embodiment of this disclosure. x and v y Normalization. This is achieved by multiplying the gradient by v. x and v y The obtained value can be right-shifted by a first shift bit, shift1. For example, regardless of the bit depth, shift1 can be set to a fixed value (e.g., 7).

[0304] The fourth embodiment of this disclosure can be modified from the second embodiment of this disclosure. FIG. 21 and FIG. 22 To achieve this.

[0305] FIG. 26 This is a view illustrating variables for BDOF processing according to another embodiment of this disclosure. FIG. 26 It can be done by modifying FIG. 21 The example obtained from the example. Therefore, FIG. 21 and FIG. 26 The description of the common parts will be omitted.

[0306] like FIG. 26As shown, in order to perform BDOF processing, the input bit depth bitDepth of the current block can be set to BitDepth. Y In this case, BitDepth Y This can be derived based on information about the bit depth signaled via the bitstream. Furthermore, regardless of the bit depth, the first shift (shift1), second shift (shift2), and third shift (shift3) can be set to fixed values. For example, the first shift (shift1), second shift (shift2), and third shift (shift3) can be set to 7, 4, and 1, respectively. The fourth shift (shift4) and offset (offset4) can be configured according to... FIG. 17 The example is derived in the same way. Furthermore, regardless of the bit depth, the variable `mvRefineThres` can be set to a fixed value. For example, the variable `mvRefineThres` can be set to "1 << 5". Additionally, a variable `gradLimit` can be set to specify the limiting range of the gradient values. In this case, `gradLimit` can be set based on the bit depth, and for example, it can be set to "1 < <Max(15,BitDepth+3)”。

[0307] FIG. 27 This is a view illustrating a method for deriving the gradient, autocorrelation, and cross-correlation of the current sub-block according to another embodiment of this disclosure. FIG. 27 It can be done by modifying FIG. 22 The example obtained from the example. Therefore, FIG. 22 and FIG. 27 The description of the common parts will be omitted.

[0308] and FIG. 22 Compared to the example, in FIG. 27 In the gradient calculations of equations (3) to (6), a limiting operation can be performed. That is, the calculated gradient value can be limited to a value within the range specified by gradLimit. For example, the limiting range can be [-gradLimit, gradLimit-1]. According to the third embodiment, by performing a limiting operation in the gradient calculation process, bit overflow can be prevented.

[0309] As described above, through such FIG. 26 and FIG. 27 The modifications shown are based on the second embodiment of this disclosure. FIG. 21 and FIG. 22 Examples of this disclosure can be used to implement the fourth embodiment.

[0310] According to the fourth embodiment of this disclosure, the bit width of predSample and the various parameters of BDOF based on BitDepth can be derived as shown in the table below.

[0311] [Table 4]

[0312]

[0313] As shown in Table 4 above, the gradient, which is highly correlated with bit depth, changes according to the bit depth. Furthermore, since the range of gradient values ​​is determined based on the predSample value, the accuracy of the gradient values ​​can be increased. Additionally, by executing... FIG. 27 The clipping operations in equations (3) to (6) prevent bit overflow during gradient calculation. Furthermore, regardless of the bit depth, v, which is not related to the bit depth... x and v y The bit width can have a fixed value.

[0314] In the following, an implementation of limiting the BDOF offset according to the present disclosure will be described.

[0315] The predicted sample `predSample` generated by interpolation through inter-frame prediction has a range of values ​​determined by the input bit depth and the coefficients of the interpolation filter, and in the worst case, it has a range of values ​​in the range of [-16830, 33150]. In this case, by adding the value -8192 to the `predSample` value to prevent 16-bit overflow, as shown in Table 1, the value of `predSample` can be adjusted to the range of [-25022, 24958].

[0316] Therefore, as shown in Table 1, when BitDepth Y When the value is 8, predSample has a value in the range of 16 bits, the gradient uses 11 bits, and v x and v y Using 6 bits, the final range of bdofOffset is [-49856, 48298]. Therefore, according to various embodiments of this disclosure, when predSample and bdofOffset are added, a 16-bit overflow may occur.

[0317] According to the disclosure, when the value of the refinement prediction sample is calculated by clipping the BDOF offset (bdofOffset) value within a predetermined range, it is possible to prevent a 16-bit overflow from occurring. More specifically, when the bit depth is 8 to 12, predSample has a value in the range of 16 bits, and when the bit depth is 14 and 16, respectively, predSample has a value in the range of 18 bits and 20 bits. Accordingly, it is also possible to define the clipping range of bdofOffset in consideration of the bit depth. For example, a variable OffsetLimit that specifies the clipping range of bdofOffset can be defined based on the bit depth, and the clipping of bdofOffset can be additionally applied to the embodiments of the disclosure.

[0318] For example, in Equation (4) of FIG. 20 For example, in Equation (4) of FIG. 23 In Equation (4) of FIG. 20 For example, in Equation (4) of FIG. 23 In Equation (4) of Y As described above, by applying the clipping to bdofOffset, it is possible to prevent a bit overflow that can occur when predSample and bdofOffset are added.

[0319] Hereinafter, a method of deriving a BDOF offset bdofOffset according to another embodiment of the disclosure will be described.

[0320] According to the present embodiment, it is possible to minimize the right shift operation in the gradient calculation process to improve the accuracy of the BDOF offset. In addition, when the right shift operation is performed in the bdofOffset calculation process, the accuracy of bdofOffset can be further increased by adding the offset value thereto. In this case, the offset can be determined based on the right shift. For example, the present embodiment can be applied to the third embodiment of the disclosure. In this case, FIG. 23 Equation (3) of

[0321] bdofOffset = (vx * (gradientHL0[x+1][y+1] - gradientHL1[x+1][y+1] + offset)) >> shift1 + (vy * (gradientV[x+1][y+1] - gradientV[x+1][y])) >> shift2y (gradientVL0[x+1][y+1] - gradientVL1[x+1][y+1]) + offset) » shift1

[0322] In the above equation, shift1 can be set to "6", and the offset can be set based on shift1, for example, "1 « (shift1 - 1)".

[0323] The modification of equation (3) above is not limited to application to the third embodiment of the present disclosure, and is applicable to other embodiments of the present disclosure. For example, the modification of equation (3) can be applied to the fourth embodiment of the present disclosure. In this case, shift1 can be set to "7", and the offset can be set based on shift1, for example, "1 « (shift1 - 1)".

[0324] According to the present embodiment, by minimizing the right shift operation in the gradient calculation process and adding the offset value when the right shift operation is performed in the bdofOffset calculation process, the accuracy of bdofOffset can be increased.

[0325] Hereinafter, a method of deriving the BDOF offset bdofOffset according to another embodiment of the present disclosure will be described.

[0326] As described above, in the BDOF offset calculation process, various variables are derived using the prediction samples and the gradients. When normalization is performed according to various embodiments of the present disclosure, the bit range of the gradient value can vary if the right shift is set to "1" or the clipping is performed in the gradient calculation process. For example, as shown in Table 1, in the first embodiment of the present disclosure, the gradient in the 11-bit range can be calculated by applying "» 6" to the prediction sample in the 16-bit range. In this case, according to the third or fourth embodiment of the present disclosure, when "» 1" or clipping is applied to the prediction sample in the 16-bit range, the gradient can have a value in the 16-bit range, as shown in Table 3 or Table 4. However, when the range of the gradient value increases, a 16-bit overflow can occur when calculating the variables tempH and tempV based on the gradient calculation. In the present embodiment, in consideration of this, the generation of a bit overflow can be prevented by applying a right shift operation to each term of the gradient when calculating tempH and tempV. For example, FIG. 25 and FIG. 27 Equations (8) and (9) can be modified as follows.

[0327] tempH[x][y] = (gradientHL0[x][y] » shift3) + (gradientHL1[x][y] » shift3) (8)

[0328] tempV[x][y] = (gradientVL0[x][y] » shift3) + (gradientVL1[x][y] » shift3) (9)

[0329] In the modified equations (8) and (9), shift3may be set to Max(1, bitDepth-11) or a fixed value 1. As described above, by modifying the third and fourth embodiments of the present disclosure, it is possible to prevent the generation of bit overflow when calculating tempH and tempV.

[0330] Hereinafter, a method of preventing the generation of bit overflow according to another embodiment of the present disclosure will be described.

[0331] As described above, when "» 1" or clipping is applied to the predicted samples in the 16-bit range according to the third or fourth embodiment of the present disclosure, 16-bit overflow can occur when calculating the variables tempH and tempV. In this case, even in the calculation of intermediate parameters such as sGx2, sGy2, sGxGy, sGxdI, and sGydI calculated based on the variables tempH and tempV, 16-bit overflow can occur. According to the present embodiment, in order to prevent the generation of bit overflow not only in the calculation of the variables tempH and tempV but also in the calculation of the intermediate parameters sGx2, sGy2, sGxGy, sGxdI, and sGydI, the right shift shift3 applied to the modified equations (8) and (9) can be adjusted. For example, in the modification of the equations (8) and (9) as described above FIG. 25 in the third embodiment of the present disclosure, shift3may be set to Max(6, bitDepth-6) or a fixed value 6. Also, in the modification of the equations (8) and (9) as described above FIG. 27 in the fourth embodiment of the present disclosure, shift3may be set to Max(6, bitDepth-6)+1 or a fixed value 7. According to the present embodiment, it is possible to prevent the generation of bit overflow not only in the calculation of the variables tempH and tempV but also in the calculation of the intermediate parameters sGx2, sGy2, sGxGy, sGxdI, and sGydI.

[0332] The embodiments of the present disclosure are not limited to the above-described examples, and the embodiments of the present disclosure can be used in combination with other embodiments or modifications. For example, the fifth embodiment of the present disclosure can be implemented by changing the step FIG. 16 to FIG. 20 in the first embodiment of the present disclosure described above to the step FIG. 17 of the third embodiment of the present disclosure. For example, the fifth embodiment of the present disclosure can be derived by changing the step FIG. 17 of the first embodiment of the present disclosure to the step FIG. 24 of the third embodiment of the present disclosure.

[0333] Although the above-described exemplary methods of the disclosure are represented as a series of operations for the sake of descriptive clarity, the order of execution of the steps is not intended to be limiting, and the steps can be executed simultaneously or in a different order as necessary. To implement the methods according to the disclosure, the described steps can further include other steps, can include the remaining steps in addition to some steps, or can include other additional steps in addition to some steps.

[0334] In the disclosure, an image encoding apparatus or an image decoding apparatus that performs a predetermined operation (step) can perform an operation (step) that confirms an execution condition or situation of the corresponding operation (step). For example, if it is described that a predetermined operation is performed when a predetermined condition is satisfied, the image encoding apparatus or the image decoding apparatus can perform the predetermined operation after determining whether the predetermined condition is satisfied.

[0335] The various embodiments of the disclosure are not a list of all possible combinations and are intended to describe representative aspects of the disclosure, and matters described in the various embodiments can be applied independently or in combination of two or more.

[0336] The various embodiments of the disclosure can be implemented in hardware, firmware, software, or a combination thereof. In the case of implementing the disclosure through hardware, the disclosure can be implemented by an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a general purpose processor, a controller, a microcontroller, a microprocessor, etc.

[0337] In addition, the image decoding apparatus and the image encoding apparatus to which the embodiments of the disclosure are applied can be included in a multimedia broadcast transmission and reception apparatus, a mobile communication terminal, a home theater video apparatus, a digital theater video apparatus, a surveillance camera, a video chat apparatus, a real-time communication apparatus such as video communication, a mobile streaming apparatus, a storage medium, a camcorder, a video on demand (VoD) service providing apparatus, an over the top video (OTT video) apparatus, an Internet streaming service providing apparatus, a three-dimensional (3D) video apparatus, a video phone video apparatus, a medical video apparatus, etc., and can be used to process a video signal or a data signal. For example, the OTT video apparatus can include a game console, a Blu-ray player, an Internet access TV, a home theater system, a smart phone, a tablet PC, a digital video recorder (DVR), etc.

[0338] FIG. 28 is a view showing a content streaming system to which the embodiments of the disclosure can be applied.

[0339] As FIG. 28As illustrated in the middle, a content streaming system to which embodiments of the disclosure are applied can mainly include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0340] The encoding server compresses content input from a multimedia input device such as a smartphone, a camera, a camcorder, or the like into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, when a multimedia input device such as a smartphone, a camera, a camcorder, or the like directly generates a bitstream, the encoding server can be omitted.

[0341] The bitstream can be generated by an image encoding method or an image encoding apparatus to which embodiments of the disclosure are applied, and the streaming server can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0342] The streaming server transmits multimedia data to a user device based on a request of the user through the web server, and the web server serves as a medium to inform a service to the user. When the user requests a desired service to the web server, the web server can deliver it to the streaming server, and the streaming server can transmit multimedia data to the user. In this case, the content streaming system can include a separate control server. In this case, the control server is used to control commands / responses between devices in the content streaming system.

[0343] The streaming server can receive content from the media storage and / or the encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server can store a bitstream for a predetermined time.

[0344] Examples of the user device can include a mobile phone, a smartphone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smart watch, smart glasses, a head-mounted display), a digital TV, a desktop computer, a digital signage, or the like.

[0345] The respective servers in the content streaming system can operate as distributed servers, in which case data received from the respective servers can be distributed.

[0346] The scope of the disclosure includes software or machine-executable commands (e.g., an operating system, an application, firmware, a program, or the like) for enabling operations of the methods according to various embodiments to be performed on a device or a computer, a non-transitory computer-readable medium having such software or commands stored thereon and executable on a device or a computer.

[0347] Industrial applicability

[0348] Embodiments of the present disclosure can be used for encoding or decoding an image.

Claims

1. An image decoding method performed by an image decoding device, the image decoding method comprising the following steps: The L0 and L1 prediction samples of the current block are derived by using the motion information of the current block. Derive the gradient for a predetermined region in the current block, including the current sub-block; The motion offset (v) for the current sub-block is derived using the gradient. x ,v y ); The bidirectional optical flow (BDOF) offset is derived by using the gradient and the motion offset. as well as Predicted samples for the current block are derived using the L0 predicted samples, the L1 predicted samples, and the BDOF offset. The gradient is derived by right-shifting the L0 and L1 prediction samples of the current block by a first shift. Among them, the motion offset (v) is derived. x ,v y The steps include the following: The first intermediate parameter diff is derived by subtracting the L1 prediction sample of the current block from the L0 prediction sample of the current block; and The second intermediate parameters tempH and tempV are derived by shifting the gradient to the right by a third shift. The step of deriving the first intermediate parameter diff includes shifting the L0 prediction sample and the L1 prediction sample of the current block to the right by a second shift. Regardless of the bit depth of the current block, the first shift, the second shift, and the third shift are set to fixed values. The value of the first shift is different from the values ​​of the second and third shifts. Wherein, the value of the second shift is different from the value of the third shift, and The second shift and the third shift are 4 and 1, respectively.

2. The image decoding method according to claim 1, wherein, The first shift is 6.

3. The image decoding method according to claim 1, wherein, The motion offset (v) x ,v y The amplitude is limited to the predetermined range.

4. The image decoding method according to claim 3, wherein, Regardless of the bit depth of the current block, the motion offset (v) x ,v y The predetermined range that is limited is set to a fixed range.

5. The image decoding method according to claim 1, in, The steps of deriving the BDOF offset include shifting the value derived based on the gradient and the motion offset to the right by a predetermined shift, and Regardless of the bit depth of the current block, the predetermined shift is set to a fixed range.

6. The image decoding method according to claim 1, in, The step of deriving the predicted sample for the current block includes limiting the BDOF offset within a predetermined range, and The predetermined range is set based on the bit depth of the current block.

7. An image encoding method performed by an image encoding device, the image encoding method comprising the following steps: The L0 and L1 prediction samples of the current block are derived by using the motion information of the current block. Derive the gradient for a predetermined region in the current block, including the current sub-block; The motion offset (v) for the current sub-block is derived using the gradient. x ,v y ); The bidirectional optical flow (BDOF) offset is derived by using the gradient and the motion offset. as well as Predicted samples for the current block are derived using the L0 predicted samples, the L1 predicted samples, and the BDOF offset. The gradient is derived by right-shifting the L0 and L1 prediction samples of the current block by a first shift. Among them, the motion offset (v) is derived. x ,v y The steps include the following: The first intermediate parameter diff is derived by subtracting the L1 prediction sample of the current block from the L0 prediction sample of the current block; and The second intermediate parameters tempH and tempV are derived by shifting the gradient to the right by a third shift. The step of deriving the first intermediate parameter diff includes shifting the L0 prediction sample and the L1 prediction sample of the current block to the right by a second shift. Regardless of the bit depth of the current block, the first shift, the second shift, and the third shift are set to fixed values. The value of the first shift is different from the values ​​of the second and third shifts. Wherein, the value of the second shift is different from the value of the third shift, and The second shift and the third shift are 4 and 1, respectively.

8. A method for transmitting a bit stream, the method comprising the following steps: Send the bitstream generated by the image encoding method; The image encoding method includes the following steps: The L0 and L1 prediction samples of the current block are derived by using the motion information of the current block. Derive the gradient for a predetermined region in the current block, including the current sub-block; The motion offset (v) for the current sub-block is derived using the gradient. x ,v y ); The bidirectional optical flow (BDOF) offset is derived using the gradient and the motion offset; and Predicted samples for the current block are derived using the L0 predicted samples, the L1 predicted samples, and the BDOF offset. The gradient is derived by right-shifting the L0 and L1 prediction samples of the current block by a first shift. Among them, the motion offset (v) is derived. x ,v y The steps include the following: The first intermediate parameter diff is derived by subtracting the L1 prediction sample of the current block from the L0 prediction sample of the current block; and The second intermediate parameters tempH and tempV are derived by shifting the gradient to the right by a third shift. The step of deriving the first intermediate parameter diff includes shifting the L0 prediction sample and the L1 prediction sample of the current block to the right by a second shift. Regardless of the bit depth of the current block, the first shift, the second shift, and the third shift are set to fixed values. Wherein, the value of the first shift is different from the values ​​of the second shift and the third shift, and Wherein, the value of the second shift is different from the value of the third shift, and The second shift and the third shift are 4 and 1, respectively.

Images