Method and apparatus for sub-picture based image encoding / decoding and method of transmitting bitstream

Abstract

A method and apparatus for image encoding / decoding are provided. An image decoding method according to the disclosure corresponds to an image decoding method performed by an image decoding apparatus, and the method can include the steps of determining whether BDOF or PROF is applied to a current block; when BDOF or PROF is applied to the current block, extracting prediction samples of the current block from a reference picture of the current block based on motion information of the current block; and deriving improved prediction samples of the current block by applying BDOF or PROF to the current block based on the extracted prediction samples.

Description

TECHNICAL FIELD

[0001] The disclosure relates to an image encoding / decoding method and apparatus and a method of transmitting a bitstream, and more particularly, to an image encoding / decoding method and apparatus for performing sub-picture encoding / decoding and a method of transmitting a bitstream generated by the image encoding method / apparatus of the disclosure. BACKGROUND

[0002] Recently, demands for high-resolution and high-quality images, such as high definition (HD) images and ultra-high definition (UHD) images, are increasing in various fields. As the resolution and quality of image data are improved, the amount of information or bits to be transmitted is relatively increased compared to existing image data. The increase in the amount of information or bits to be transmitted results in an increase in transmission cost and storage cost.

[0003] Therefore, an efficient image compression technique is needed to effectively transmit, store, and reproduce information about high-resolution and high-quality images. SUMMARY

[0004] TECHNICAL PROBLEM

[0005] An object of the disclosure is to provide an image encoding / decoding method and apparatus having improved encoding / decoding efficiency.

[0006] Another object of the disclosure is to provide an image encoding / decoding method and apparatus for encoding / decoding an image based on a sub-picture.

[0007] Another object of the disclosure is to provide an image encoding / decoding method and apparatus for performing BDOF or PROF based on whether a current sub-picture is determined as a picture.

[0008] Another object of the disclosure is to provide a method of transmitting a bitstream generated by an image encoding method or apparatus according to the disclosure.

[0009] Another object of the disclosure is to provide a recording medium storing a bitstream generated by an image encoding method or apparatus according to the disclosure.

[0010] Another object of the disclosure is to provide a recording medium storing a bitstream generated by an image encoding method or apparatus according to the disclosure.

[0011] The technical problems addressed by the disclosure are not limited to the above-mentioned technical problems, and other technical problems not described herein will be clearly understood by those skilled in the art from the following description.

[0012] TECHNICAL SOLUTION

[0013] The image decoding method according to an aspect of the disclosure can include determining whether bi-directional optical flow (BDOF) or optical flow prediction refinement (PROF) is applied to a current block; extracting a prediction sample of the current block from a reference picture of the current block based on motion information of the current block based on the BDOF or the PROF being applied to the current block; and deriving a refined prediction sample of the current block by applying the BDOF or the PROF to the current block based on the extracted prediction sample.

[0014] In the image decoding method of the disclosure, the step of extracting the prediction sample of the current block can be performed based on whether a current sub-picture including the current block is treated as a picture.

[0015] In the image decoding method of the disclosure, whether the current sub-picture is treated as a picture can be determined based on flag information signaled through a bitstream.

[0016] In the image decoding method of the disclosure, the flag information can be signaled through a sequence parameter set (SPS).

[0017] In the image decoding method of the disclosure, the step of extracting the prediction sample of the current block can be performed based on a position of the prediction sample to be extracted, and wherein the position of the prediction sample can be clipped within a predetermined range.

[0018] In the image decoding method of the disclosure, based on the current sub-picture being treated as a picture, the predetermined range can be specified by a boundary position of the current sub-picture.

[0019] In the image decoding method of the disclosure, the position of the prediction sample to be extracted can include an x-coordinate and a y-coordinate, the x-coordinate can be clipped within a range of a left boundary position and a right boundary position of the current sub-picture, and the y-coordinate can be clipped within a range of an upper boundary position and a lower boundary position of the current sub-picture.

[0020] In the image decoding method of the disclosure, the left boundary position of the current sub-picture can be derived as a product of position information of a predetermined unit specifying a left position of the current sub-picture and a width of the predetermined unit, the right boundary position of the current sub-picture can be derived by performing a "-1" operation on a product of position information of a predetermined unit specifying a right position of the current sub-picture and a width of the predetermined unit, the upper boundary position of the current sub-picture can be derived as a product of position information of a predetermined unit specifying an upper position of the current sub-picture and a height of the predetermined unit, and the lower boundary position of the current sub-picture can be derived by performing a "-1" operation on a product of position information of a predetermined unit specifying a lower position of the current sub-picture and a height of the predetermined unit.

[0021] In the image decoding method of the disclosure, the predetermined unit can be a grid or a CTU.

[0022] In the image decoding method of the disclosure, the predetermined range can be a range of a current picture including the current block, based on the current sub-picture not being treated as a picture.

[0023] An image decoding apparatus according to another aspect of the disclosure can include a memory and at least one processor. The at least one processor can determine whether bi-directional optical flow (BDOF) or optical flow prediction refinement (PROF) is applied to a current block; extract a prediction sample of the current block from a reference picture of the current block based on motion information of the current block, based on the BDOF or the PROF being applied to the current block; and derive a refined prediction sample of the current block by applying the BDOF or the PROF to the current block based on the extracted prediction sample.

[0024] In the image decoding apparatus of the disclosure, the at least one processor can extract a prediction sample of the current block based on whether a current sub-picture including the current block is treated as a picture.

[0025] An image encoding method according to another aspect of the disclosure can include determining whether bi-directional optical flow (BDOF) or optical flow prediction refinement (PROF) is applied to a current block; extracting a prediction sample of the current block from a reference picture of the current block based on motion information of the current block, based on the BDOF or the PROF being applied to the current block; and deriving a refined prediction sample of the current block by applying the BDOF or the PROF to the current block based on the extracted prediction sample.

[0026] In the image encoding method of the disclosure, the step of extracting a prediction sample of the current block can be performed based on whether a current sub-picture including the current block is treated as a picture.

[0027] A transmission method according to another aspect of the disclosure can transmit a bitstream generated by the image encoding method and / or the image encoding apparatus of the disclosure to an image decoding apparatus.

[0028] Further, a computer-readable recording medium according to another aspect of the disclosure can store a bitstream generated by the image encoding apparatus or the image encoding method of the disclosure.

[0029] The features described above with respect to the brief overview of the disclosure are merely exemplary aspects of the following detailed description of the disclosure and do not limit the scope of the disclosure.

[0030] Advantageous Effects

[0031] According to the disclosure, it is possible to provide an image encoding / decoding method and apparatus having improved encoding / decoding efficiency.

[0032] Further, according to the disclosure, it is possible to provide an image encoding / decoding method and apparatus for encoding / decoding an image based on a sub-picture.

[0033] Also, according to the present disclosure, it is possible to provide an image encoding / decoding method and apparatus for performing BDOF or PROF based on a determination of whether a current sub-picture is treated as a picture.

[0034] Also, according to the present disclosure, it is possible to provide a method of transmitting a bitstream generated by the image encoding method or apparatus according to the present disclosure.

[0035] Also, according to the present disclosure, it is possible to provide a recording medium storing a bitstream generated by the image encoding method or apparatus according to the present disclosure.

[0036] Also, according to the present disclosure, it is possible to provide a recording medium storing a bitstream generated by the image encoding method or apparatus according to the present disclosure.

[0037] Those skilled in the art will appreciate that the effects realized by the present disclosure are not limited to what has been particularly described hereinabove and other advantages of the present disclosure will be more clearly understood from the detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0038] FIG. 1 is a view schematically illustrating a video encoding system to which embodiments of the present disclosure are applicable.

[0039] FIG. 2 is a view schematically illustrating an image encoding apparatus to which embodiments of the present disclosure are applicable.

[0040] FIG. 3 is a view schematically illustrating an image decoding apparatus to which embodiments of the present disclosure are applicable.

[0041] FIG. 4 is a flowchart illustrating a video / image encoding method based on inter prediction.

[0042] FIG. 5 is a view illustrating a configuration of an inter prediction unit 180 according to the present disclosure.

[0043] FIG. 6 is a flowchart illustrating a video / image decoding method based on inter prediction.

[0044] FIG. 7 is a view illustrating a configuration of an inter prediction unit 260 according to the present disclosure.

[0045] FIG. 8 is a view illustrating a motion expressible in an affine mode.

[0046] FIG. 9 is a view illustrating a parametric model of an affine mode.

[0047] FIG. 10is a view illustrating a method of generating an affine merge candidate list.

[0048] FIG. 11 is a view illustrating a CPMV derived from a neighboring block.

[0049] FIG. 12 is a view illustrating a neighboring block for deriving an inherited affine merge candidate.

[0050] FIG. 13 is a view illustrating a neighboring block for deriving a constructed affine merge candidate.

[0051] FIG. 14 is a view illustrating a method of generating an affine MVP candidate list.

[0052] FIG. 15 is a view illustrating a neighboring block for a subblock-based TMVP mode.

[0053] FIG. 16 is a view illustrating a method of deriving a motion vector field according to a subblock-based TMVP mode.

[0054] FIG. 17 is a view illustrating a CU extended to perform BDOF.

[0055] FIG. 18 is a view illustrating a relationship between Δv(i,j), v(i,j), and a subblock motion vector.

[0056] FIG. 19 is a view illustrating an embodiment of a syntax for signaling a subpicture syntax element in an SPS.

[0057] FIG. 20 is a view illustrating an embodiment of an algorithm for deriving a predetermined variable such as SubPicTop.

[0058] FIG. 21 is a view illustrating a method of encoding a picture using subpictures by an encoding device according to an embodiment.

[0059] FIG. 22 is a view illustrating a method of decoding a picture using subpictures by a decoding device according to an embodiment.

[0060] FIG. 23 is a view illustrating a process of deriving prediction samples of a current block by applying BDOF.

[0061] FIG. 24 is a view illustrating input and output of a BDOF process according to an embodiment of the disclosure.

[0062] FIG. 25is a view illustrating variables for BDOF processing according to an embodiment of the present disclosure.

[0063] FIG. 26 is a view illustrating a method of generating prediction samples for each sub-block in a current CU based on whether to apply BDOF according to an embodiment of the present disclosure.

[0064] FIG. 27 is a view illustrating a method of deriving gradients, auto-correlations, and cross-correlations for a current sub-block according to an embodiment of the present disclosure.

[0065] FIG. 28 is a view illustrating a method of deriving motion refinements (vx, vy), deriving BDOF offsets, and generating prediction samples for a current sub-block according to an embodiment of the present disclosure.

[0066] FIG. 29 is a view illustrating a process of deriving prediction samples for a current block by applying PROF.

[0067] FIG. 30 is a view illustrating an example of PROF processing according to the present disclosure.

[0068] FIG. 31 is a view illustrating a case where reference samples are to be extracted across a boundary of a sub-picture.

[0069] FIG. 32 is FIG. 31 an enlarged view of an extraction region of

[0070] FIG. 33 is a view illustrating a reference sample extraction process according to an embodiment of the present disclosure.

[0071] FIG. 34 is a flowchart illustrating a reference sample extraction process according to the present disclosure.

[0072] FIG. 35 is a view illustrating a part of a fractional sample interpolation process according to the present disclosure.

[0073] FIG. 36 is a view illustrating a part of a sbTMVP derivation method according to the present disclosure.

[0074] FIG. 37 is a view illustrating a method of deriving a sub-picture boundary position according to the present disclosure.

[0075] FIG. 38 is a view showing a content streaming system to which an embodiment of the present disclosure is applicable. DETAILED DESCRIPTION

[0076] Hereinafter, detailed descriptions will be given of embodiments of the disclosure with reference to the accompanying drawings so as to be easily carried out by those skilled in the art. However, the disclosure can be implemented in various different forms and is not limited to the embodiments described herein.

[0077] In describing the disclosure, if it is determined that detailed descriptions of relevant known functions or configurations unnecessarily obscure the scope of the disclosure, detailed descriptions thereof will be omitted. In the drawings, parts irrelevant to the description of the disclosure are omitted, and like reference numerals are assigned to like parts.

[0078] In the disclosure, when one component is "connected", "coupled", or "linked" to another component, it can include not only a direct connection relationship but also an indirect connection relationship in which a middle component exists. In addition, when one component "includes" or "has" another component, unless otherwise specified, it means that other components can also be included, rather than excluding other components.

[0079] In the disclosure, the terms first, second, and the like are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of the components, unless otherwise specified. Accordingly, within the scope of the disclosure, a first component in one embodiment can be referred to as a second component in another embodiment, and similarly, a second component in one embodiment can be referred to as a first component in another embodiment.

[0080] In the disclosure, components distinguished from each other are intended to clearly describe each feature, and do not mean that the components must be separated. That is, a plurality of components can be integrated in one hardware or software unit, or one component can be distributed and implemented in a plurality of hardware or software units. Therefore, even if not specifically described, embodiments in which these components are integrated or distributed are included in the scope of the disclosure.

[0081] In the disclosure, components described in each embodiment are not necessarily essential components, and some components can be optional components. Therefore, embodiments consisting of a subset of the components described in the embodiments are also included in the scope of the disclosure. In addition, embodiments including other components in addition to the components described in various embodiments are included in the scope of the disclosure.

[0082] The disclosure relates to encoding and decoding of images, and unless redefined in the disclosure, the terms used in the disclosure can have general meanings commonly used in the technical field to which the disclosure belongs.

[0083] In the disclosure, a "picture" generally refers to a unit representing one image for a specific time period, while a slice / tile is a coding unit constituting a part of a picture, and one picture can be composed of one or more slices / tiles. Further, a slice / tile can include one or more coding tree units (CTU).

[0084] In the disclosure, "pixel" or "pel" can mean a minimum unit constituting one picture (or image). Further, "sample" can be used as a term corresponding to a pixel. One sample can generally represent a pixel or a value of a pixel, and can represent only a pixel / pixel value of a luminance component or only a pixel / pixel value of a chrominance component.

[0085] In the disclosure, "unit" can mean a basic unit of image processing. The unit can include at least one of a specific region of a picture and information related to the region. In some cases, the unit can be used interchangeably with terms such as "sample array", "block", or "region". In general, an MxN block can include a set (or array) of M columns and N rows of samples (or sample array) or transform coefficients.

[0086] In the disclosure, "current block" can mean one of "current coding block", "current coding unit", "coding target block", "decoding target block", or "processing target block". When performing prediction, "current block" can mean "current prediction block" or "prediction target block". When performing transform (inverse transform) / quantization (dequantization), "current block" can mean "current transform block" or "transform target block". When performing filtering, "current block" can mean "filtering target block".

[0087] In the disclosure, the term " / " or "," can be interpreted to mean "and / or". For example, "A / B" and "A,B" can mean "A and / or B". Further, "A / B / C" and "A / B / C" can mean "at least one of A, B, and / or C".

[0088] In the disclosure, the term "or" should be interpreted to mean "and / or". For example, the expression "A or B" can include 1) only "A", 2) only "B", or 3) both "A and B". In other words, in the disclosure, "or" should be interpreted to mean "additionally or alternatively".

[0089] Overview of a video encoding system

[0090] FIG. 1 is a view schematically showing a video coding system according to the disclosure.

[0091] The video encoding system according to the embodiments can include an encoding apparatus 10 and a decoding apparatus 20. The encoding apparatus 10 can deliver encoded video and / or image information or data in the form of a file or a stream to the decoding apparatus 20 via a digital storage medium or a network.

[0092] The encoding apparatus 10 according to the embodiments can include a video source generator 11, an encoding unit 12, and a transmitter 13. The decoding apparatus 20 according to the embodiments can include a receiver 21, a decoding unit 22, and a renderer 23. The encoding unit 12 can be referred to as a video / image encoding unit, and the decoding unit 22 can be referred to as a video / image decoding unit. The transmitter 13 can be included in the encoding unit 12. The receiver 21 can be included in the decoding unit 22. The renderer 23 can include a display and the display can be configured as a separate device or an external component.

[0093] The video source generator 11 can acquire a video / image through a process of capturing, synthesizing, or generating a video / image. The video source generator 11 can include a video / image capturing device and / or a video / image generating device. The video / image capturing device can include, for example, one or more cameras, a video / image archive including previously captured videos / images, etc. The video / image generating device can include, for example, a computer, a tablet, and a smartphone, and can generate a video / image (electronically). For example, a virtual video / image can be generated through a computer, etc. In this case, the video / image capturing process can be replaced by a process of generating related data.

[0094] The encoding unit 12 can encode an input video / image. For compression and encoding efficiency, the encoding unit 12 can perform a series of processes, such as prediction, transformation, and quantization. The encoding unit 12 can output encoded data (encoded video / image information) in the form of a bitstream.

[0095] The transmitter 13 can transmit the encoded video / image information or data output in the form of a bitstream to the receiver 21 of the decoding apparatus 20 in the form of a file or a stream through a digital storage medium or a network. The digital storage medium can include various storage media, such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter 13 can include an element for generating a media file through a predetermined file format and can include an element for transmission through a broadcasting / communication network. The receiver 21 can extract / receive a bitstream from a storage medium or a network and transmit the bitstream to the decoding unit 22.

[0096] The decoding unit 22 can decode a video / image by performing a series of processes, such as dequantization, inverse transformation, and prediction, corresponding to the operations of the encoding unit 12.

[0097] The renderer 23 can render the decoded video / image. The rendered video / image can be displayed through a display.

[0098] Overview of an image encoding device

[0099] FIG. 2 FIG. 1 is a view schematically illustrating an image encoding apparatus to which embodiments of the present disclosure are applicable.

[0100] As shown in FIG. 2 The image encoding apparatus 100 can include an image partitioner 110, a subtractor 115, a transformer 120, a quantizer 130, a dequantizer 140, an inverse transformer 150, an adder 155, a filter 160, a memory 170, an inter prediction unit 180, an intra prediction unit 185, and an entropy encoder 190. The inter prediction unit 180 and the intra prediction unit 185 can be collectively referred to as a "prediction unit". The transformer 120, the quantizer 130, the dequantizer 140, and the inverse transformer 150 can be included in a residue processor. The residue processor can further include the subtractor 115.

[0101] In some embodiments, all or at least some of the plurality of components configuring the image encoding apparatus 100 can be configured by one hardware component (e.g., an encoder or a processor). Also, the memory 170 can include a decoded picture buffer (DPB) and can be configured by a digital storage medium.

[0102] The image partitioner 110 can partition an input image (or picture or frame) input to the image encoding apparatus 100 into one or more processing units. For example, the processing unit can be referred to as a coding unit (CU). The coding unit can be obtained by recursively partitioning a coding tree unit (CTU) or a largest coding unit (LCU) according to a quad tree binary tree ternary (QT / BT / TT) structure. For example, one coding unit can be partitioned into a plurality of coding units of a deeper depth based on a quad tree structure, a binary tree structure, and / or a ternary tree structure. For the partitioning of the coding unit, the quad tree structure can be applied first, and then the binary tree structure and / or the ternary tree structure can be applied. The encoding process according to the present disclosure can be performed based on a final coding unit that is no longer partitioned. The maximum coding unit can be used as the final coding unit, and a coding unit of a deeper depth obtained by partitioning the maximum coding unit can also be used as the final coding unit. Here, the encoding process can include a process of prediction, transform, and reconstruction that will be described later. As another example, the processing unit of the encoding process can be a prediction unit (PU) or a transform unit (TU). The prediction unit and the transform unit can be divided or partitioned from the final coding unit. The prediction unit can be a sample prediction unit, and the transform unit can be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from the transform coefficient.

[0103] The prediction unit (inter prediction unit 180 or intra prediction unit 185) can perform prediction on a block (current block) to be processed and generate a prediction block including predicted samples of the current block. The prediction unit can determine whether to apply intra prediction or inter prediction on a basis of the current block or CU. The prediction unit can generate various information related to prediction of the current block and transmit the generated information to the entropy encoder 190. The information about prediction can be encoded in the entropy encoder 190 and output in the form of a bitstream.

[0104] The intra prediction unit 185 can predict the current block by referring to samples in the current picture. The reference samples can be located in neighbors of the current block or can be placed separately according to an intra prediction mode and / or an intra prediction technique. The intra prediction mode can include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes can include, for example, a DC mode and a planar mode. The directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes according to the level of detail of the prediction direction. However, this is merely an example, and more or less directional prediction modes can be used according to settings. The intra prediction unit 185 can determine a prediction mode applied to the current block by using a prediction mode applied to a neighboring block.

[0105] The inter prediction unit 180 can derive a prediction block of the current block based on a reference block (a reference sample array) specified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of a block, a sub-block, or a sample based on the correlation of the motion information between the neighboring block and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, bi-prediction, etc.) information. In the case of inter prediction, the neighboring block can include a spatial neighboring block present in the current picture and a temporal neighboring block present in a reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block can be the same or different. The temporal neighboring block can be referred to as a collocated reference block, a collocated CU (colCU), etc. The reference picture including the temporal neighboring block can be referred to as a collocated picture (colPic). For example, the inter prediction unit 180 can configure a motion information candidate list based on the neighboring block and generate information indicating which candidate to use to derive a motion vector and / or a reference picture index of the current block. The inter prediction can be performed based on various prediction modes. For example, in the case of a skip mode and a merge mode, the inter prediction unit 180 can use motion information of the neighboring block as motion information of the current block. In the case of the skip mode, unlike the merge mode, a residual signal can not be transmitted. In the case of a motion vector prediction (MVP) mode, a motion vector of the neighboring block can be used as a motion vector predictor, and a motion vector of the current block can be signaled by encoding a motion vector difference and an indicator of the motion vector predictor. The motion vector difference can mean a difference between the motion vector of the current block and the motion vector predictor.

[0106] The prediction unit can generate a prediction signal based on various prediction methods and prediction techniques described below. For example, the prediction unit can not only apply intra prediction or inter prediction, but also simultaneously apply intra prediction and inter prediction to predict the current block. The prediction method of simultaneously applying both intra prediction and inter prediction to predict the current block can be referred to as combined inter and intra prediction (CIIP). In addition, the prediction unit can perform intra block copy (IBC) to predict the current block. Intra block copy can be used for content image / video coding of games, etc., for example, screen content coding (SCC). IBC is a method of predicting a current picture using a reference block previously reconstructed in the current picture at a position separated by a predetermined distance. When IBC is applied, the position of the reference block in the current picture can be coded as a vector (a block vector) corresponding to the predetermined distance.

[0107] The prediction signal generated by the prediction unit can be used to generate a reconstructed signal or to generate a residual signal. The subtractor 115 can generate a residual signal (a residual block or a residual sample array) by subtracting the prediction signal (a prediction block or a prediction sample array) output from the prediction unit from the input image signal (an original block or an original sample array). The generated residual signal can be transmitted to the transformer 120.

[0108] The transformer 120 can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique can include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a karhunen-loève transform (KLT), a graph-based transform (GBT), or a conditional non-linear transform (CNT). Here, the GBT refers to a transform obtained from a graph when relationship information between pixels is represented by a graph. The CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process can be applied to a square pixel block having the same size or can be applied to a block having a variable size other than a square.

[0109] The quantizer 130 can quantize the transform coefficients and transmit them to the entropy encoder 190. The entropy encoder 190 can encode the quantized signal (information about the quantized transform coefficients) and output a bitstream. The information about the quantized transform coefficients can be referred to as residual information. The quantizer 130 can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0110] The entropy encoder 190 can perform various encoding methods, such as exponential golomb, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), and the like. The entropy encoder 190 can encode information (e.g., values of syntax elements, etc.) required for video / image reconstruction other than the quantized transform coefficients together or individually. The encoded information (e.g., encoded video / image information) can be transmitted or stored in the form of a bitstream in units of a network abstraction layer (NAL). The video / image information can further include information about various parameter sets, such as an adaptive parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information can further include general constraint information. The information signaled, transmitted, and / or syntax elements described in the disclosure can be encoded through the above-described encoding process and included in the bitstream.

[0111] The bitstream can be transmitted through a network or can be stored in a digital storage medium. The network can include a broadcasting network and / or a communication network, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that transmits a signal output from the entropy encoder 190 and / or a storage unit (not shown) that stores the signal can be included as an internal / external element of the image encoding apparatus 100. Alternatively, the transmitter can be provided as a component of the entropy encoder 190.

[0112] The quantized transform coefficients output from the quantizer 130 can be used to generate a residual signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients through the dequantizer 140 and the inverse transformer 150.

[0113] The adder 155 adds the reconstructed residual signal to a prediction signal output from the inter prediction unit 180 or the intra prediction unit 185 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array). If there is no residual for a block to be processed, for example, in the case of applying a skip mode, a prediction block can be used as a reconstructed block. The adder 155 can be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal can be used for intra prediction of a next block to be processed in the current picture, and can be used for inter prediction of a next picture by filtering as described below.

[0114] In addition, luma mapping with chroma scaling (LMCS) is applied to the picture encoding process as described below.

[0115] The filter 160 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 160 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 170, specifically, in the DPB of the memory 170. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. The filter 160 can generate various information related to filtering and transmit the generated information to the entropy encoder 190, as described later in the description of each filtering method. The information related to filtering can be encoded by the entropy encoder 190 and output in the form of a bitstream.

[0116] The modified reconstructed picture transmitted to the memory 170 can be used as a reference picture in the inter prediction unit 180. When inter prediction is applied by the image encoding apparatus 100, prediction mismatch between the image encoding apparatus 100 and an image decoding apparatus can be avoided and encoding efficiency can be improved.

[0117] The DPB of memory 170 can store modified reconstructed frames for use as reference frames in inter-frame prediction unit 180. Memory 170 can store motion information of blocks from which motion information in the current frame is derived (or encoded) and / or motion information of already reconstructed blocks in the frame. The stored motion information can be transmitted to inter-frame prediction unit 180 and used as motion information for spatially or temporally neighboring blocks. Memory 170 can store reconstructed samples of reconstructed blocks in the current frame and can transmit the reconstructed samples to intra-frame prediction unit 185.

[0118] Overview of an image decoding device

[0119] FIG. 3 This is a schematic view illustrating an image decoding device to which embodiments of the present disclosure may be applied.

[0120] like FIG. 3 As shown, the image decoding device 200 may include an entropy decoder 210, a dequantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-frame prediction unit 260, and an intra-frame prediction unit 265. The inter-frame prediction unit 260 and the intra-frame prediction unit 265 may be collectively referred to as "prediction units". The dequantizer 220 and the inverse transformer 230 may be included in a residual processor.

[0121] According to an implementation, all or at least some of the components of the image decoding device 200 can be configured by hardware components (e.g., a decoder or a processor). Furthermore, the memory 250 may include a decoded screen buffer (DPB) or may be configured by a digital storage medium.

[0122] The image decoding device 200, having received a bitstream including video / image information, can perform operations related to... FIG. 2 The image is reconstructed by processing corresponding to the processing performed by the image encoding device 100. For example, the image decoding device 200 can perform decoding using a processing unit applied in the image encoding device. Therefore, the decoding processing unit can be, for example, an encoding unit. The encoding unit can be obtained by segmenting a coding tree unit or a maximum coding unit. The reconstructed image signal decoded and output by the image decoding device 200 can be reproduced by a reproduction device (not shown).

[0123] Image decoding device 200 can receive data in bitstream form from... FIG. 2The received signal can be decoded by the entropy decoder 210. For example, the entropy decoder 210 can parse a bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information can further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information can further include general constraint information. The image decoding apparatus can also decode a picture based on the information on the parameter sets and / or the general constraint information. The information and / or the syntax elements described in the disclosure to be signaled / received can be decoded through a decoding process and obtained from the bitstream. For example, the entropy decoder 210 decodes information in the bitstream based on an encoding method such as exponential Golomb coding, CAVLC, or CABAC, and outputs values of syntax elements required for image reconstruction and quantized values of transform coefficients of a residual. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream, determine a context model using information of a decoded target syntax element, decoded information of a neighboring block and a decoded target block, or a symbol / bin decoded in a previous stage, perform arithmetic decoding on the bins by predicting a probability of occurrence of the bins according to the determined context model, and generate a symbol corresponding to a value of each syntax element. In this case, the CABAC entropy decoding method can update the context model by using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information related to prediction among the information decoded by the entropy decoder 210 can be provided to the prediction unit (inter prediction unit 260 and intra prediction unit 265), and residual values, i.e., quantized transform coefficients and related parameter information, on which entropy decoding is performed in the entropy decoder 210 can be input to the dequantizer 220. In addition, information on filtering among the information decoded by the entropy decoder 210 can be provided to the filter 240. Further, a receiver (not shown) for receiving a signal output from the image encoding apparatus can be further configured as an internal / external element of the image decoding apparatus 200, or the receiver can be a component of the entropy decoder 210.

[0124] Further, the image decoding apparatus according to the disclosure can be referred to as a video / image / picture decoding apparatus. The image decoding apparatus can be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder can include the entropy decoder 210. The sample decoder can include at least one of the dequantizer 220, the inverse transformer 230, the adder 235, the filter 240, the memory 250, the inter prediction unit 260, or the intra prediction unit 265.

[0125] The dequantizer 220 can dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 220 can rearrange the quantized transform coefficients in the form of a two-dimensional block. In this case, the rearrangement can be performed based on a coefficient scan order performed in the image encoding apparatus. The dequantizer 220 can perform dequantization on the quantized transform coefficients by using a quantization parameter (e.g., quantization step length information) and obtain the transform coefficients.

[0126] The inverse transformer 230 can inverse-transform the transform coefficients to obtain a residual signal (a residual block, a residual sample array).

[0127] The prediction unit can perform prediction on the current block and generate a prediction block including prediction samples of the current block. The prediction unit can determine whether to apply intra prediction or inter prediction to the current block based on information about prediction output from the entropy decoder 210, and can determine a specific intra / inter prediction mode (prediction technique).

[0128] As the same as described in the prediction unit of the image encoding apparatus 100, the prediction unit can generate a prediction signal based on various prediction methods (techniques) described later.

[0129] The intra prediction unit 265 can predict the current block by referring to samples in the current picture. The description of the intra prediction unit 185 is equally applicable to the intra prediction unit 265.

[0130] The inter prediction unit 260 can derive a prediction block of the current block based on a reference block (a reference sample array) on a reference picture specified by a motion vector. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of a block, a sub-block, or a sample based on the correlation of the motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, bi-prediction, etc.) information. In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter prediction unit 260 can configure a motion information candidate list based on the neighboring blocks, and derive a motion vector and / or a reference picture index of the current block based on received candidate selection information. The inter prediction can be performed based on various prediction modes, and the information about prediction can include information indicating the inter prediction mode of the current block.

[0131] The adder 235 can generate a reconstructed signal (a reconstructed picture, a reconstructed block, a reconstructed sample array) by adding the obtained residual signal and a prediction signal (a prediction block, a prediction sample array) output from the prediction unit (including the inter prediction unit 260 and / or the intra prediction unit 265). The description of the adder 155 is equally applicable to the adder 235.

[0132] In addition, as described below, luminance mapping with chroma scaling (LMCS) is applied to a picture decoding process.

[0133] The filter 240 can improve subjective / objective picture quality by applying filtering to the reconstructed signal. For example, the filter 240 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture and store the modified reconstructed picture in the memory 250, specifically, in the DPB of the memory 250. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc.

[0134] The (modified) reconstructed picture stored in the DPB of the memory 250 can be used as a reference picture in the inter prediction unit 260. The memory 250 can store motion information of a block from which motion information in a current picture is derived (or decoded) and / or motion information of a block in a picture that has been reconstructed. The stored motion information can be transferred to the inter prediction unit 260 to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 250 can store reconstructed samples of a reconstructed block in a current picture and transfer the reconstructed samples to the intra prediction unit 265.

[0135] In the disclosure, the embodiments described in the filter 160, the inter prediction unit 180, and the intra prediction unit 185 of the image encoding apparatus 100 can be equally or correspondingly applied to the filter 240, the inter prediction unit 260, and the intra prediction unit 265 of the image decoding apparatus 200.

[0136] Overview of inter prediction

[0137] The image encoding apparatus / image decoding apparatus can perform inter prediction in a unit of a block to derive a prediction sample. Inter prediction can mean prediction that is derived in a manner of depending on a data element of a picture other than a current picture. When inter prediction is applied to a current block, a prediction block of the current block can be derived based on a reference block on a reference picture specified by a motion vector.

[0138] In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information of the current block can be derived based on the correlation of the motion information between neighboring blocks and the current block, and the motion information can be derived in a unit of a block, a sub-block, or a sample. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction type information. Here, the inter prediction type information can mean direction information of inter prediction. The inter prediction type information can indicate that the current block is predicted using one of L0 prediction, L1 prediction, or bi-prediction.

[0139] When inter prediction is applied to the current block, neighboring blocks of the current block can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference blocks of the current block and the reference picture including the temporal neighboring blocks can be the same or different. The temporal neighboring block can be referred to as a collocated reference block or a collocated CU (colCU), and the reference picture including the temporal neighboring block can be referred to as a collocated picture (colPic).

[0140] Further, a motion information candidate list can be constructed based on the neighboring blocks of the current block, and in this case, flag or index information indicating which candidate to use can be signaled in order to derive the motion vector and / or the reference picture index of the current block.

[0141] According to the inter prediction type, the motion information can include L0 motion information and / or L1 motion information. A motion vector in the L0 direction can be defined as an L0 motion vector or MVL0, and a motion vector in the L1 direction can be defined as an L1 motion vector or MVL1. Prediction based on the L0 motion vector can be defined as L0 prediction, prediction based on the L1 motion vector can be defined as L1 prediction, and prediction based on both the L0 motion vector and the L1 motion vector can be defined as bi-prediction. Here, the L0 motion vector can mean a motion vector associated with a reference picture list L0, and the L1 motion vector can mean a motion vector associated with a reference picture list L1.

[0142] The reference picture list L0 can include pictures preceding the current picture in the output order as reference pictures, and the reference picture list L1 can include pictures succeeding the current picture in the output order. A preceding picture can be defined as a forward (reference) picture, and a succeeding picture can be defined as a backward (reference) picture. Further, the reference picture list L0 can also include pictures succeeding the current picture in the output order as reference pictures. In this case, within the reference picture list L0, the preceding pictures can be indexed first, and then the succeeding pictures can be indexed. The reference picture list L1 can also include pictures preceding the current picture in the output order as reference pictures. In this case, within the reference picture list L1, the succeeding pictures can be indexed first, and then the preceding pictures can be indexed. Here, the output order can correspond to a picture order count (POC) order.

[0143] FIG. 4 is a flowchart illustrating a video / image encoding method based on inter prediction.

[0144] FIG. 5 is a view illustrating a configuration of an inter predictor 180 according to the disclosure.

[0145] FIG. 4 The encoding method of can be performed byFIG. 2 The image encoding device performs. Specifically, step S410 can be performed by the inter predictor 180, and step S420 can be performed by the residual processor. Specifically, step S420 can be performed by the subtractor 115. Step S430 can be performed by the entropy encoder 190. The prediction information of step S430 can be derived by the inter predictor 180, and the residual information of step S430 can be derived by the residual processor. The residual information is information about residual samples. The residual information can include information about quantized transform coefficients for the residual samples. As described above, the residual samples can be derived as transform coefficients by the transformer 120 of the image encoding device, and the transform coefficients can be derived as quantized transform coefficients by the quantizer 130. The information about the quantized transform coefficients can be encoded by the entropy encoder 190 through a residual encoding process.

[0146] The image encoding device can perform inter prediction for the current block (S410). The image encoding device can derive an inter prediction mode and motion information for the current block and generate prediction samples for the current block. Here, the inter prediction mode determination, the motion information derivation, and the prediction sample generation processes can be performed simultaneously or any one of them can be performed before the others. For example, as shown in FIG. 1, the inter prediction unit 180 of the image encoding device can include a prediction mode determination unit 181, a motion information derivation unit 182, and a prediction sample derivation unit 183. The prediction mode determination unit 181 can determine a prediction mode for the current block, the motion information derivation unit 182 can derive motion information for the current block, and the prediction sample derivation unit 183 can derive prediction samples for the current block. For example, the inter prediction unit 180 of the image encoding device can search for a block similar to the current block within a predetermined region (search region) of a reference picture through motion estimation, and derive a reference block whose difference from the current block is equal to or less than a predetermined criterion or minimum value. Based on this, a reference picture index indicating a reference picture in which the reference block is located can be derived, and a motion vector can be derived based on a positional difference between the reference block and the current block. The image encoding device can determine a mode applied to the current block among various inter prediction modes. The image encoding device can compare rate-distortion (RD) costs for various prediction modes, and determine an optimal inter prediction mode for the current block. However, a method of determining an inter prediction mode for the current block by the image encoding device is not limited to the above-described example, and various methods can be used. FIG. 5

[0147] ​For example, the inter prediction mode of the current block can be determined as at least one of a merge mode, a merge skip mode, a motion vector prediction (MVP) mode, a symmetric motion vector difference (SMVD) mode, an affine mode, a subblock-based merge mode, an adaptive motion vector resolution (AMVR) mode, a history-based motion vector predictor (HMVP) mode, a pair-wise average merge mode, a merge mode with motion vector difference (MMVD) mode, a decoder-side motion vector refinement (DMVR) mode, a combined inter and intra prediction (CIIP) mode, or a geometric partition mode (GPM).

[0148] For example, when the skip mode or the merge mode is applied to the current block, the image encoding apparatus can derive merge candidates from neighboring blocks of the current block, and construct a merge candidate list using the derived merge candidates. Also, the image encoding apparatus can derive a reference block whose difference from the current block is equal to or less than a predetermined criterion or a minimum value among reference blocks indicated by the merge candidates included in the merge candidate list. In this case, a merge candidate associated with the derived reference block can be selected, and merge index information indicating the selected merge candidate can be generated and signaled to the image decoding apparatus. Motion information of the selected merge candidate can be used to derive motion information of the current block.

[0149] As another example, when the MVP mode is applied to the current block, the image encoding apparatus can derive motion vector predictor (MVP) candidates from neighboring blocks of the current block, and construct an MVP candidate list using the derived MVP candidates. Also, the image encoding apparatus can use a motion vector of an MVP candidate selected from among the MVP candidates included in the MVP candidate list as an MVP of the current block. In this case, for example, a motion vector indicating a reference block derived through the above-described motion estimation can be used as a motion vector of the current block, an MVP candidate having a motion vector with a minimum difference from a motion vector of the current block among the MVP candidates can be the selected MVP candidate. A motion vector difference (MVD) which is a difference obtained by subtracting the MVP from the motion vector of the current block can be derived. In this case, index information indicating the selected MVP candidate and information about the MVD can be signaled to the image decoding apparatus. Also, when the MVP mode is applied, a value of a reference picture index can be constructed as reference picture index information and separately signaled to the image decoding apparatus.

[0150] The image encoding apparatus can derive residual samples based on the prediction samples (S420). The image encoding apparatus can derive the residual samples through a comparison between original samples of the current block and the prediction samples. For example, the residual samples can be derived by subtracting corresponding prediction samples from the original samples.

[0151] The image encoding apparatus can encode image information including prediction information and residual information (S430). The image encoding apparatus can output the encoded image information in the form of a bitstream. The prediction information can include prediction mode information (e.g., a skip flag, a merge flag, or a mode index, etc.) and information on motion information as information related to a prediction process. Among the prediction mode information, the skip flag indicates whether a skip mode is applied to the current block, and the merge flag indicates whether a merge mode is applied to the current block. Alternatively, the prediction mode information can indicate one of a plurality of prediction modes, e.g., a mode index. When the skip flag and the merge flag are 0, it can be determined that the MVP mode is applied to the current block. The information on motion information can include candidate selection information (e.g., a merge index, an mvp flag, or an mvp index) as information for deriving a motion vector. Among the candidate selection information, the merge index can be signaled when the merge mode is applied to the current block, and can be information for selecting one of merge candidates included in a merge candidate list. Among the candidate selection information, the MVP flag or the MVP index can be signaled when the MVP mode is applied to the current block, and can be information for selecting one of MVP candidates in an MVP candidate list. Specifically, the MVP flag can be signaled using a syntax element mvp_10_flag or mvp_11_flag. In addition, the information on motion information can include information on the above-described MVD and / or reference picture index information. In addition, the information on motion information can include information indicating whether to apply L0 prediction, L1 prediction, or bi-prediction. The residual information is information on residual samples. The residual information can include information on quantized transform coefficients for the residual samples.

[0152] The output bitstream can be stored in a (digital) storage medium and transmitted to an image decoding apparatus or can be transmitted to an image decoding apparatus via a network.

[0153] As described above, the image encoding apparatus can generate a reconstructed picture (a picture including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is to enable the image encoding apparatus to derive the same prediction result as that performed by the image decoding apparatus, thereby improving encoding efficiency. Accordingly, the image encoding apparatus can store the reconstructed picture (or the reconstructed samples and the reconstructed blocks) in a memory and use it as a reference picture for inter prediction. As described above, the in-loop filtering process is also applied to the reconstructed picture.

[0154] FIG. 6 is a flowchart illustrating a video / image decoding method based on inter prediction.

[0155] FIG. 7 is a view illustrating a configuration of the inter prediction unit 260 according to the disclosure.

[0156] The image decoding device can perform operations corresponding to those performed by the image encoding device. The image decoding device can perform prediction for the current block based on the received prediction information and derive prediction samples.

[0157] FIG. 6 The decoding method of the image decoding device can be performed by FIG. 3 Steps S610 to S630 can be performed by the inter prediction unit 260, and the prediction information of step S610 and the residual information of step S640 can be obtained from the bitstream by the entropy decoder 210. The residual processor of the image decoding device can derive the residual samples of the current block based on the residual information (S640). Specifically, the dequantizer 220 of the residual processor can perform dequantization based on the quantized transform coefficients derived from the residual information to derive the transform coefficients, and the inverse transformer 230 of the residual processor can perform inverse transform on the transform coefficients to derive the residual samples of the current block. Step S650 can be performed by the adder 235 or the reconstructor.

[0158] Specifically, the image decoding device can determine the prediction mode of the current block based on the received prediction information (S610). The image decoding device can determine which inter prediction mode is applied to the current block based on the prediction mode information in the prediction information.

[0159] For example, it can be determined whether the skip mode is applied to the current block based on the skip flag. In addition, it can be determined whether the merge mode or the MVP mode is applied to the current block based on the merge flag. Alternatively, one of various inter prediction mode candidates can be selected based on the mode index. The inter prediction mode candidates can include the skip mode, the merge mode, and / or the MVP mode or can include various inter prediction modes to be described below.

[0160] The image decoding device can derive the motion information of the current block based on the determined inter prediction mode (S620). For example, when the skip mode or the merge mode is applied to the current block, the image decoding device can construct a merge candidate list to be described below, and select one of the merge candidates included in the merge candidate list. The selection can be performed based on the above-described candidate selection information (merge index). The motion information of the selected merge candidate can be used to derive the motion information of the current block. For example, the motion information of the selected merge candidate can be used as the motion information of the current block.

[0161] As another example, when the MVP mode is applied to the current block, the image decoding device can construct an MVP candidate list and use the motion vector of the MVP candidate selected from the MVP candidates included in the MVP candidate list as the MVP of the current block. Selection can be performed based on the aforementioned candidate selection information (MVP flag or MVP index). In this case, the MVD of the current block can be derived based on information about the MVD, and the motion vector of the current block can be derived based on the MVP and MVD of the current block. Additionally, the reference frame index of the current block can be derived based on reference frame index information. The frame indicated by the reference frame index in the reference frame list of the current block can be derived as the reference frame to be referenced for inter-frame prediction of the current block.

[0162] The image decoding device can generate a prediction sample for the current block based on the motion information of the current block (S630). In this case, a reference frame can be derived based on the reference frame index of the current block, and the prediction sample for the current block can be derived using samples of the reference block indicated by the motion vector of the current block on the reference frame. In some cases, a prediction sample filtering process can also be performed on all or some of the prediction samples of the current block.

[0163] For example, such as FIG. 7 As shown, the inter-frame prediction unit 260 of the image decoding device may include a prediction mode determination unit 261, a motion information derivation unit 262, and a prediction sample derivation unit 263. In the inter-frame prediction unit 260 of the image decoding device, the prediction mode determination unit 261 can determine the prediction mode of the current block based on the received prediction mode information, the motion information derivation unit 262 can derive the motion information (motion vector and / or reference frame index, etc.) of the current block based on the received motion information, and the prediction sample derivation unit 263 can derive the prediction samples of the current block.

[0164] The image decoding device can generate residual samples for the current block based on the received residual information (S640). The image decoding device can generate reconstructed samples for the current block based on the predicted samples and residual samples, and generate a reconstructed image based on this (S650). Thereafter, the in-loop filtering process is applied to the reconstructed image as described above.

[0165] As described above, the inter-frame prediction process may include the steps of determining an inter-frame prediction mode, deriving motion information based on the determined prediction mode, and performing prediction (generating prediction samples) based on the derived motion information. As described above, the inter-frame prediction process may be performed by an image encoding device and an image decoding device.

[0166] The steps for deriving motion information based on the prediction pattern will be described in more detail below.

[0167] As described above, inter prediction can be performed using motion information of a current block. The image encoding apparatus can derive optimal motion information of the current block through a motion estimation process. For example, the image encoding apparatus can use an original block in an original picture of the current block in a fractional pixel unit to search for a similar reference block having high correlation in a predetermined search range in a reference picture, and use the same to derive motion information. The similarity of the blocks can be calculated based on a sum of absolute differences (SAD) between the current block and the reference block. In this case, the motion information can be derived based on the reference block having the smallest SAD in the search area. The derived motion information can be signaled to the image decoding apparatus according to various methods based on an inter prediction mode.

[0168] When the merge mode is applied to the current block, the motion information of the current block is not directly transmitted, and the motion information of the current block is derived using the motion information of the neighboring block. Accordingly, the motion information of the current prediction block can be indicated by transmitting flag information indicating that the merge mode is used and candidate selection information (e.g., merge index) indicating which neighboring block is used as a merge candidate. In the present disclosure, since the current block is used as the same meaning as the current prediction block, and the neighboring block is used as the same meaning as the neighboring prediction block, the current block can be used as the same meaning as the current prediction block, and the neighboring block can be used as the same meaning as the neighboring prediction block.

[0169] The image encoding apparatus can search for a merge candidate block for deriving the motion information of the current block to perform the merge mode. For example, a maximum of five merge candidate blocks can be used, but is not limited thereto. The maximum number of the merge candidate blocks can be transmitted in a slice header or a tile group header, but is not limited thereto. After finding the merge candidate blocks, the image encoding apparatus can generate a merge candidate list and select a merge candidate block having the smallest RD cost as a final merge candidate block.

[0170] The merge candidate list can use, for example, five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate can be used.

[0171] Overview of affine mode

[0172] Hereinafter, an affine mode as an example of the inter prediction mode will be described in detail. In a conventional video encoding / decoding system, only one motion vector is used to express the motion information of the current block (translation motion model). However, in the conventional method, the optimal motion information is expressed only in units of blocks, but the optimal motion information cannot be expressed in units of pixels. To solve this problem, an affine motion mode of defining the motion information of a block in units of pixels has been proposed. According to the affine mode, two to four motion vectors associated with the current block can be used to determine the motion vector of each pixel and / or sub-block unit of the block.

[0173] In the affine mode, the motion information of each pixel can be expressed using at least one of translation, scaling, rotation, or shearing, compared to the existing motion information expressed using a translation (or displacement) of pixel values.

[0174] FIG. 8 is a view illustrating motion expressible in the affine mode.

[0175] In FIG. 8 Among the motions illustrated in FIG. 1, the affine mode in which the motion information of each pixel is expressed using displacement, scaling, or rotation can be a similar or simplified affine mode. The affine mode in the following description can mean the similar or simplified affine mode.

[0176] The motion information in the affine mode can be expressed using two or more control point motion vectors (CPMVs). The motion vector of a specific pixel position of a current block can be derived using the CPMVs. In this case, a set of motion vectors of each pixel and / or sub-block of the current block can be defined as an affine motion vector field (affine MVF).

[0177] FIG. 9 is a view illustrating a parametric model of the affine mode.

[0178] When the affine mode is applied to a current block, the affine MVF can be derived using one of a 4-parameter model and a 6-parameter model. In this case, the 4-parameter model can mean a model type using two CPMVs, and the 6-parameter model can mean a model type using three CPMVs. FIG. 9 (a) and (b) of FIG. 1 illustrate CPMVs used in the 4-parameter model and the 6-parameter model, respectively. FIG. 9

[0179] When the position of the current block is (x, y), the motion vector according to the pixel position can be derived according to Equation 1 or Equation 2. For example, the motion vector according to the 4-parameter model can be derived according to Equation 1, and the motion vector according to the 6-parameter model can be derived according to Equation 2.

[0180] [Equation 1]

[0181]

[0182] [Equation 2]

[0183]

[0184] ​In the Formula 1 and the Formula 2, mv0={mv_0x,mv_0y} can be a CPMV at the upper left corner position of the current block, mv1={mv_1x,mv_1y} can be a CPMV at the upper right position of the current block, and mv2={mv_2x,mv_2y} can be a CPMV at the lower left position of the current block. In this case, W and H correspond to the width and the height of the current block, respectively, and mv={mv_x,mv_y} can mean a motion vector of a pixel position {x,y}.

[0185] In the encoding / decoding process, the affine MVF can be determined in units of pixels and / or predefined sub-blocks. When the affine MVF is determined in units of pixels, a motion vector can be derived based on each pixel value. Also, when the affine MVF is determined in units of sub-blocks, a motion vector of a corresponding block can be derived based on a center pixel value of the sub-block. The center pixel value can mean a virtual pixel existing at the center of the sub-block or a right lower pixel among four pixels existing at the center. In addition, the center pixel value can be a specific pixel in the sub-block and can be a pixel representing the sub-block. In the present disclosure, a case where the affine MVF is determined in units of 4x4 sub-blocks will be described. However, this is only for convenience of description, and the size of the sub-blocks can be variously changed.

[0186] That is, when the affine prediction is available, a motion model applicable to the current block can include three models, i.e., a translation motion model, a 4-parameter affine motion model, and a 6-parameter affine motion model. Here, the translation motion model can mean a model used by a present block unit motion vector, the 4-parameter affine motion model can mean a model used by two CPMVs, and the 6-parameter affine motion model can mean a model used by three CPMVs. The affine mode can be divided into detailed modes according to a motion information encoding / decoding method. For example, the affine mode can be subdivided into an affine MVP mode and an affine merge mode.

[0187] When the affine merge mode is applied to the current block, the CPMVs can be derived from neighboring blocks of the current block encoded / decoded in the affine mode. The affine merge mode can be applied to the current block when at least one of the neighboring blocks of the current block is encoded / decoded in the affine mode. That is, when the affine merge mode is applied to the current block, the CPMVs of the current block can be derived using the CPMVs of the neighboring blocks. For example, the CPMVs of the neighboring blocks can be determined as the CPMVs of the current block, or the CPMVs of the current block can be derived based on the CPMVs of the neighboring blocks. When the CPMVs of the current block are derived based on the CPMVs of the neighboring blocks, at least one encoding parameter of the current block or the neighboring blocks can be used. For example, the CPMVs of the neighboring blocks can be modified based on the size of the neighboring blocks and the size of the current block and used as the CPMVs of the current block.

[0188] Further, the affine merge in which the MV is derived in a sub-block unit can be referred to as a sub-block merge mode, which can be designated by a merge_subblock_flag having a first value (e.g., 1). In this case, the affine merge candidate list described below can be referred to as a sub-block merge candidate list. In this case, the candidate derived as the SbTMVP described below can be further included in the sub-block merge candidate list. In this case, the candidate derived as the SbTMVP can be used as a candidate of index #0 of the sub-block merge candidate list. In other words, the candidate derived as the SbTMVP can be located in front of the inherited affine candidate and the constructed affine candidate described below in the sub-block merge candidate list.

[0189] For example, an affine mode flag designating whether the affine mode is applicable to the current block can be defined, which can be signaled in at least one higher level (e.g., sequence, picture, slice, tile, tile group, tile group, etc.) of the current block. For example, the affine mode flag can be named sps_affine_enabled_flag.

[0190] When the affine merge mode is applied, the affine merge candidate list can be configured to derive the CPMV of the current block. In this case, the affine merge candidate list can include at least one of an inherited affine merge candidate, a constructed affine merge candidate, or a zero merge candidate. When the neighboring block of the current block is encoded / decoded in the affine mode, the inherited affine merge candidate can mean a candidate derived using the CPMV of the neighboring block. The constructed affine merge candidate can mean a candidate in which the CPMV is derived based on the motion vector of the neighboring block of each control point (CP). Further, the zero merge candidate can mean a candidate consisting of the CPMV having a size of 0. In the following description, the CP can mean a specific position of a block used to derive the CPMV. For example, the CP can be each vertex position of the block.

[0191] FIG. 10 is a view illustrating a method of generating an affine merge candidate list.

[0192] Referring to FIG. 10 The affine merge candidate can be added to the affine merge candidate list in the order of the inherited affine merge candidate (S1210), the constructed affine merge candidate (S1220), and the zero merge candidate (S1230) according to the flowchart of FIG. 12. When the number of candidates included in the candidate list does not satisfy the maximum number of candidates even though all of the inherited affine merge candidate and the constructed affine merge candidate are added to the affine merge candidate list, the zero merge candidate can be added. In this case, the zero merge candidate can be added until the number of candidates of the affine merge candidate list satisfies the maximum number of candidates.

[0193] FIG. 11This is a view illustrating the control point motion vector (CPMV) derived from the neighboring blocks.

[0194] For example, up to two inheritance affine merge candidates can be derived, each of which can be derived based on at least one of the left neighbor block and the upper neighbor block.

[0195] FIG. 12 This is a view that is used to derive neighboring blocks for affine inheritance candidates.

[0196] Based on the inheritance affine merge candidate derived from the left neighbor block. FIG. 12 The affine merge candidate derived from at least one of the neighboring blocks A0 or A1 can be based on the above neighboring blocks. FIG. 12 The candidate is derived from at least one of the neighboring blocks B0, B1, or B2. In this case, the scan order of the neighboring blocks can be A0 to A1 and B0, B1, and B2, but is not limited to this. For each of the left and top neighboring blocks, the inherited affine merge candidate can be derived based on the first neighboring block available in the scan order. In this case, redundancy checks may not be performed between the candidates derived from the left neighboring block and the top neighboring block.

[0197] For example, such as FIG. 11 As shown, when the left neighboring block A is encoded / decoded in an affine mode, at least one of the motion vectors v2, v3, and v4 corresponding to the CP of the neighboring block A can be derived. When the neighboring block A is encoded / decoded using a 4-parameter affine model, the inherited affine merge candidate can be derived using v2 and v3. In contrast, when the neighboring block A is encoded / decoded using a 6-parameter affine model, the inherited affine merge candidate can be derived using v2, v3, and v4.

[0198] FIG. 13 This is an example view used to derive neighboring blocks for constructing affine merge candidates.

[0199] Constructing an affine candidate can mean having a candidate CPMV that is derived using the combined motion information of neighboring blocks. The motion information of each CP can be derived using the spatial or temporal neighboring blocks of the current block. In the following description, CPMVk can mean representing the motion vector of the k-th CP. For example, referring to FIG. 13 CPMV1 can be determined as the first available motion vector among the motion vectors of B2, B3, and A2, and in this case, the scan order can be B2, B3, and A2. CPMV2 can be determined as the first available motion vector among the motion vectors of B1 and B0, and in this case, the scan order can be B1 and B0. CPMV3 can be determined as one of the motion vectors of A1 and A0, and in this case, the scan order can be A1 and A0. When TMVP applies to the current block, CPMV4 can be determined as the motion vector of the time neighboring block T.

[0200] After deriving the four motion vectors of the respective CPs, an affine merge candidate can be derived based thereon. The affine merge candidate can be configured by including at least two motion vectors selected from the four motion vectors of the respective CPs. For example, the affine merge candidate can consist of at least one of {CPMV1,CPMV2,CPMV3}, {CPMV1,CPMV2,CPMV4}, {CPMV1,CPMV3,CPMV4}, {CPMV2,CPMV3,CPMV4}, {CPMV1,CPMV2}, or {CPMV1,CPMV3} in this order. The affine merge candidate consisting of three motion vectors can be a candidate of a 6-parameter affine model. In contrast, the affine merge candidate consisting of two motion vectors can be a candidate of a 4-parameter affine model. To avoid the scaling process of the motion vectors, when the reference picture indices of the CPs are different from each other, the combination of the related CPMVs can be ignored for deriving the affine merge candidate.

[0201] When the affine MVP mode is applied to the current block, the encoding / decoding device can derive two or more CPMV predictors and CPMVs of the current block and derive a CPMV difference based thereon. In this case, the CPMV difference can be signaled from the encoding device to the decoding device. The image decoding device can derive the CPMV predictors of the current block, reconstruct the signaled CPMV difference, and then derive the CPMVs of the current block based on the CPMV predictors and the CPMV difference.

[0202] In addition, when the affine merge mode or the subblock-based TMVP is not applied to the current block (e.g., the value of the affine merge flag or merge_subblock_flag is 0), the affine MVP mode can be applied to the current block. Alternatively, when the value of inter_affine_flag is 1, the affine MVP mode can be applied to the current block. In addition, the affine MVP mode can be denoted as an affine CP MVP mode. The affine mvp candidate list described below can be referred to as a control point motion vector predictor candidate list.

[0203] When the affine MVP mode is applied to the current block, the affine MVP candidate list can be configured to derive the CPMVs of the current block.

[0204] In this case, the affine MVP candidate list can include at least one of an inherited affine MVP candidate, a constructed affine MVP candidate, a translational motion affine MVP candidate, or a zero MVP candidate. For example, the affine MVP candidate list can include up to n (e.g., n = 2) candidates.

[0205] In this case, the inherited affine MVP candidate can mean a candidate derived based on a CPMV of a neighboring block when the neighboring block of the current block is encoded / decoded in an affine mode. The constructed affine MVP candidate can mean a candidate derived by generating a CPMV combination based on motion vectors of CPMs. The zero MVP candidate can mean a candidate consisting of a CPMV having a value of 0. The derivation method and the characteristics of the inherited affine MVP candidate and the constructed affine MVP candidate are the same as those of the inherited affine candidate and the constructed affine candidate described above, and thus the description thereof will be omitted.

[0206] When the maximum number of candidates of the affine MVP candidate list is 2, the constructed affine MVP candidate, the translational motion affine MVP candidate, and the zero MVP candidate can be added when the current number of candidates is less than 2. Specifically, the translational motion affine MVP candidate can be derived in the following order.

[0207] For example, when the number of candidates included in the affine MVP candidate list is less than 2 and CPMV0 of the constructed affine MVP candidate is valid, CPMV0 can be used as an affine MVP candidate. That is, an affine MVP candidate whose all motion vectors of CP0, CP1, and CP2 are CPMV0 can be added to the affine MVP candidate list.

[0208] Next, when the number of candidates of the affine MVP candidate list is less than 2 and CPMV1 of the constructed affine MVP candidate is valid, CPMV1 can be used as an affine MVP candidate. That is, an affine MVP candidate whose all motion vectors of CP0, CP1, and CP2 are CPMV1 can be added to the affine MVP candidate list.

[0209] Next, when the number of candidates of the affine MVP candidate list is less than 2 and CPMV2 of the constructed affine MVP candidate is valid, CPMV2 can be used as an affine MVP candidate. That is, an affine MVP candidate whose all motion vectors of CP0, CP1, and CP2 are CPMV2 can be added to the affine MVP candidate list.

[0210] Irrespective of the above conditions, a temporal motion vector predictor (TMVP) of the current block can be added to the affine MVP candidate list when the number of candidates of the affine MVP candidate list is less than 2. Irrespective of the above, the zero MVP candidate can be added to the affine MVP candidate list when the number of candidates of the affine MVP candidate list is less than 2.

[0211] FIG. 14 is a view illustrating a method of generating an affine MVP candidate list.

[0212] Referring to FIG. 14The flowchart shows how candidates can be added to the affine MVP candidate list in the following order: inheriting affine MVP candidates (S1610), constructing affine MVP candidates (S1620), translating affine MVP candidates (S1630), and zero MVP candidates (S1640). As described above, steps S1620 to S1640 can be performed based on whether the number of candidates included in the affine MVP candidate list in each step is less than 2.

[0213] The scan order of inheriting affine MVP candidates can be equal to the scan order of inheriting affine merge candidates. However, in the case of inheriting affine MVP candidates, only neighboring blocks that reference the same reference frame as the current block can be considered. Redundancy checks can be omitted when an inheriting affine MVP candidate is added to the affine MVP candidate list.

[0214] To derive the construction of affine MVP candidates, we can only consider... FIG. 13 The spatial neighboring blocks are shown. Furthermore, the scan order for constructing affine MVP candidates can be equal to the scan order for constructing affine merge candidates. Additionally, to derive the construction of affine MVP candidates, the reference frame index of neighboring blocks can be checked, and in the scan order, the first neighboring block that is inter-coded and references the same reference frame as the current block can be used.

[0215] Overview of sub-block based temporal motion vector prediction (SbTMVP) mode

[0216] The following section describes in detail the sub-block-based TMVP mode as an example of inter-frame prediction modes. Based on the sub-block-based TMVP mode, the motion vector field (MVF) of the current block can be derived, and motion vectors can be derived on a sub-block basis.

[0217] Unlike the traditional TMVP mode, which executes on a unit-by-unit basis, motion vectors can be encoded / decoded on a unit-by-unit basis for applying the sub-block-based TMVP mode. Furthermore, while temporal motion vectors in the traditional TMVP mode can be derived from juxtaposed blocks in a juxtaposed frame, in the sub-block-based TMVP mode, the motion vector field can be derived from a reference block in the juxtaposed frame, specified by motion vectors derived from neighboring blocks of the current block. Hereinafter, the motion vectors derived from neighboring blocks can be referred to as the motion shift or representative motion vector of the current block.

[0218] FIG. 15 This is a view of neighboring blocks in a TMVP pattern based on sub-blocks.

[0219] When a sub-block-based TMVP pattern is applied to the current block, neighboring blocks can be identified to determine motion shifts. For example, it can be done according to... FIG. 15The order of the blocks of A1, B1, B0, and A0 is scanned for a neighboring block for determining the motion displacement. As another example, the neighboring block for determining the motion displacement can be limited to a specific neighboring block of the current block. For example, the neighboring block for determining the motion displacement can always be determined as the block A1. When the neighboring block has a motion vector referring to the col picture, the corresponding motion vector can be determined as the motion displacement. The motion vector determined as the motion displacement can be referred to as a temporal motion vector. Further, when the motion vector described above cannot be derived from the neighboring block, the motion displacement can be set as (0, 0).

[0220] FIG. 16 is a view illustrating a method of deriving a motion vector field according to a sub-block based TMVP mode.

[0221] Next, a reference block on the collocated picture designated by the motion displacement can be determined. For example, the sub-block based motion information (motion vector or reference picture index) can be obtained from the col picture by adding the motion displacement to the coordinates of the current block. In FIG. 16 In the example shown, it is assumed that the motion displacement is the motion vector of the A1 block. By applying the motion displacement to the current block, a sub-block (col sub-block) in the col picture corresponding to each sub-block configuring the current block can be designated. Thereafter, using the motion information of the corresponding sub-block (col sub-block) in the col picture, the motion information of each sub-block of the current block can be derived. For example, the motion information of the corresponding sub-block can be obtained from the center position of the corresponding sub-block. In this case, the center position can be the position of the right lower sample among four samples located at the center of the corresponding sub-block. When the motion information of a specific sub-block of the col block corresponding to the current block is not available, the motion information of the center sub-block of the col block can be determined as the motion information of the corresponding sub-block. When deriving the motion vector of the corresponding sub-block, similar to the TMVP process described above, a switch to the reference picture index and the motion vector of the current sub-block can be made. That is, when deriving the sub-block based motion vector, scaling of the motion vector can be performed considering the POC of the reference picture of the reference block.

[0222] As described above, the sub-block based TMVP candidate of the current block can be derived using the motion vector field or the motion information of the current block derived based on the sub-block.

[0223] Hereinafter, a merge candidate list configured in a sub-block unit is defined as a sub-block unit merge candidate list. The affine merge candidate and the sub-block based TMVP candidate described above can be merged to configure the sub-block unit merge candidate list.

[0224] In addition, a sub-block based TMVP mode flag specifying whether the sub-block based TMVP mode is applicable to the current block can be defined, which can be signaled in at least one level among higher levels (e.g., sequence, picture, slice, tile, tile group, slice, etc.) of the current block. For example, the sub-block based TMVP mode flag can be named sps_sbtmvp_enabled_flag. When the sub-block based TMVP mode is applicable to the current block, the sub-block based TMVP candidate can be first added to the sub-block unit merge candidate list, and then the affine merge candidate can be added to the sub-block unit merge candidate list. In addition, the maximum number of candidates that can be included in the sub-block unit merge candidate list can be signaled. For example, the maximum number of candidates that can be included in the sub-block unit merge candidate list can be 5.

[0225] The size of the sub-block for deriving the sub-block unit merge candidate list can be signaled or preset as MxN. For example, MxN can be 8x8. Thus, the affine mode or the sub-block based TMVP mode is applicable to the current block only when the size of the current block is 8x8 or more.

[0226] Hereinafter, an embodiment of a prediction performing method of the disclosure will be described. The following prediction performing method can be performed in step S410 of the method of FIG. 4 or step S630 of the method of FIG. 6. FIG. 4 FIG. 6 The following prediction performing method can be performed in step S410 of the method of FIG. 4 or step S630 of the method of FIG. 6.

[0227] A prediction block of the current block can be generated based on the motion information derived according to the prediction mode. The prediction block (block of prediction) can include prediction samples (array of prediction samples) of the current block. When the motion vector of the current block specifies a fractional sample unit, an interpolation process can be performed, and thereby, the prediction samples of the current block can be derived based on the reference samples in a unit of fractional samples within the reference picture. When the affine inter prediction is applied to the current block, the prediction samples can be generated based on the sample / sub-block unit MVs. When the bi-prediction is applied, the prediction samples derived by a weighted sum or a weighted average (according to the phase) of the prediction samples derived based on the L0 prediction (i.e., prediction using MVL0 and a reference picture within a reference picture list L0) and the prediction samples derived based on the L1 prediction (i.e., prediction using MLV1 and a reference picture within a reference picture list L1) can be used as the prediction samples of the current block. When the bi-prediction is applied and the reference picture for the L0 prediction and the reference picture for the L1 prediction are located in different temporal directions with respect to the current picture (i.e., if it corresponds to bi-prediction and bi-directional prediction), this can be referred to as true bi-prediction.

[0228] ​In the image decoding apparatus, the reconstructed samples and the reconstructed picture can be generated based on the derived prediction samples, and then the in-loop filtering process can be performed. Also, in the image encoding apparatus, the residual samples can be derived based on the derived prediction samples, and the encoding of the image information including the prediction information and the residual information can be performed.

[0229] Bi-prediction with CU-level weights (BCW)

[0230] When bi-prediction is applied to the current block as described above, the prediction samples can be derived based on the weighted average. Conventionally, the bi-prediction signal (i.e., bi-prediction sample) can be derived by a simple average of the L0 prediction signal (L0 prediction sample) and the L1 prediction signal (L1 prediction sample). That is, the bi-prediction sample is derived by an average of the L0 prediction sample based on the L0 reference picture and MVL0 and the L1 prediction sample based on the L1 reference picture and MVL1. However, according to the present disclosure, when bi-prediction is applied, the bi-prediction signal (bi-prediction sample) can be derived by a weighted average of the L0 prediction signal and the L1 prediction signal as follows.

[0231] [Equation 3]

[0232] P bi-pred = ((8-w)*P0 + w*P1 + 4) » 3

[0233] In the above Equation 3, P bi-pred denotes the bi-prediction signal (bi-prediction block) derived by the weighted average, P0 and P1 denote the L0 prediction sample (L0 prediction block) and the L1 prediction sample (L1 prediction block), respectively. Also, (8-w) and w denote the weights applied to P0 and P1, respectively.

[0234] When the bi-prediction signal is generated by the weighted average, five weights can be allowed. For example, the weight w can be selected from {-2, 3, 4, 5, 10}. For each bi-prediction CU, the weight w can be determined by one of two methods. As the first method of the two methods, when the current CU is not in the merge mode (non-merge CU), the weight index can be signaled together with the motion vector difference. For example, the bitstream can include the information on the weight index after the information on the motion vector difference. As the second method of the two methods, when the current CU is in the merge mode (merge CU), the weight index can be derived from the neighboring blocks based on the merge candidate index (merge index).

[0235] Generating the bi-prediction signal by weighted averaging can be restricted to be applied only to a CU having a size including 256 or more samples (luma component samples). That is, bi-prediction by weighted averaging can be performed only for a CU for which a product of a width and a height of the current block is 256 or more. In addition, the weight w can be used as one of five weights as described above, and one of different numbers of weights can be used. For example, according to characteristics of a current picture, five weights can be used for a low-delay picture, and three weights can be used for a non-low-delay picture. In this case, the three weights can be {3, 4, 5}.

[0236] By applying the fast search algorithm, the image encoding apparatus can determine the weight index without significantly increasing complexity. In this case, the fast search algorithm can be summarized as follows. Hereinafter, unequal weights can mean that weights applied to P0 and P1 are not equal. In addition, equal weights can mean that weights applied to P0 and P1 can be equal.

[0237] - In the case where the AMVR mode in which the resolution of the motion vector is adaptively changed is applied together, when the current picture is a low-delay picture, unequal weights can be conditionally checked only for each of 1 pel motion vector resolution and 4 pel motion vector resolution.

[0238] - In the case where the affine mode is applied together and the affine mode is selected as the optimal mode of the current block, the image encoding apparatus can perform affine motion estimation (ME) for each unequal weight.

[0239] - When two reference pictures for bi-prediction are equal, unequal weights can be conditionally checked only.

[0240] - When a predetermined condition is satisfied, unequal weights can not be checked. The predetermined condition can be based on a POC distance between the current picture and the reference picture, a quantization parameter (QP), a temporal level, etc.

[0241] The weight index of the BCW can be encoded using one context-coded bin and one or more subsequent bypass-coded bins. The first context-coded bin specifies whether equal weights are used. When unequal weights are used, additional bins can be bypass-coded and signaled. The additional bins can be signaled to specify which weight is used.

[0242] Weighted prediction (WP) is a tool used for efficiently coding images including fade. According to the weighted prediction, a weighted parameter (weight and offset) can be signaled for each reference picture included in the reference picture lists L0 and L1. Then, when performing motion compensation, the weight and offset can be applied to the corresponding reference picture. The weighted prediction and BCW can be used for different types of images. To avoid the interaction between the weighted prediction and BCW, for a CU using the weighted prediction, the BCW weight index can not be signaled. In this case, the weight can be inferred as 4. That is, equal weight can be applied.

[0243] In case of a CU applying merge mode, the weight index can be inferred from neighboring blocks based on the merge candidate index. This can be applied to both general merge mode and inherited affine merge mode.

[0244] In case of constructing affine merge mode, the affine motion information can be configured based on motion information of up to three blocks. The BCW weight index of a CU using the constructing affine merge mode can be set to the BCW weight index of the first CP in the combination. That is, BCW can not be applied to a CU coded in CIIP mode. For example, the BCW weight index of a CU coded in CIIP mode can be set to a value specifying equal weight.

[0245] Bi-directional optical flow (BDOF)

[0246] According to the present disclosure, BDOF can be used to refine bi-prediction signal. When bi-prediction is applied to a current block (e.g., CU), BDOF generates prediction samples by calculating refined motion information. Therefore, the process of calculating refined motion information by applying BDOF can be included in the above-mentioned motion information derivation step.

[0247] For example, BDOF can be applied at a 4x4 sub-block level. That is, BDOF can be performed in units of 4x4 sub-blocks within a current block.

[0248] For example, BODF can be applied to a CU satisfying at least one or all of the following conditions.

[0249] - The CU is coded in true bi-prediction mode, i.e., one of the two reference pictures precedes the current picture in display order and the other follows the current picture in display order

[0250] - The CU is not in affine mode or ATMVP merge mode

[0251] - The CU has more than 64 luma samples

[0252] - The height and width of the CU are 8 or more luma samples

[0253] - BCW weight index specifies equal weights, i.e., equal weights are applied to L0 and L1 prediction samples

[0254] - Weighted prediction (WP) is not applied to the current CU

[0255] - CIIP mode is used for the current CU

[0256] In addition, BDOF can be applied only to the luma component. However, the disclosure is not limited thereto, and BDOF can be applied to the chroma component or both the luma component and the chroma component.

[0257] The BDOF mode is based on the concept of optical flow. That is, it is assumed that the motion of an object is smooth. When BDOF is applied, for each 4x4 subblock, motion refinements (v x ,v y ) can be calculated. The motion refinements can be calculated by minimizing the difference between L0 and L1 prediction samples. The motion refinements can be used to adjust the bi-prediction sample values within the 4x4 subblock.

[0258] Hereinafter, the process of performing BDOF will be described in more detail.

[0259] First, horizontal gradients and vertical gradients of two prediction signals can be calculated. In this case, k can be 0 or 1. The gradients can be calculated by directly calculating the difference between two adjacent samples. For example, the gradients can be calculated as follows.

[0260] [Equation 4]

[0261]

[0262] In Equation 4 above, I (k) (i,j) denotes a sample value of a coordinate (i,j) of a prediction signal in a list k (k=0,1). For example, I (0) (i,j) can denote a sample value at a position (i,j) in an L0 prediction block, I (1) (i,j) can denote a sample value at a position (i,j) in an L1 prediction block. In Equation 4 above, a first shift shift1 can be determined based on the bit depth of the luma component. For example, when the bit depth of the luma component is bitDepth, shift1 can be determined as max(6,bitDepth-6).

[0263] As described above, after the gradients are calculated, the autocorrelations and cross-correlations S1, S2, S3, S5, and S6 between the gradients can be calculated as follows.

[0264] [Equation 5]

[0265] S1=∑(i,j)∈Ω Abs(ψ x (i,j)), S3=∑ (i,j)∈Ω θ(i,j)·Sign(ψ x (i,j))

[0266]

[0267] in

[0268]

[0269] θ(i,j)=(I (1) (i,j)>>n b )-(I (0) (i,j)>>n b )

[0270] Here, Ω is the 6×6 window surrounding the 4×4 sub-block.

[0271] In equation 5 above, n a and n b It can be set to min(1, bitDepth-11) and min(4, bitDepth-8) respectively.

[0272] Motion refinement (v) x ,v y The above autocorrelation and cross-correlation between gradients can be used to derive the following.

[0273] [Formula 6]

[0274]

[0275] in S_(2,s)=S_2&(2^(n_(S_2))-1), th′ BIO =2 13-BD .and It is the floor function.

[0276] In equation 6 above, n S2 It can be 12. Based on the derived motion refinement and gradient, the following adjustments can be performed on each sample in the 4×4 sub-block.

[0277] [Formula 7]

[0278]

[0279] Finally, the predicted samples pred of the CU with BDOF can be calculated by adjusting the double predicted samples of the CU as follows. BDOF .

[0280] [Formula 8]

[0281] pred BDOF (x,y) = (I (0) (x,y) + I (1) (x,y) + b(x,y) + o offset ) » shift

[0282] In the above equation, n a , n b and n S2 may be 3, 6 and 12, respectively. These values can be selected such that the multiplier does not exceed 15 bits and the bit width of the intermediate parameter is maintained within 32 bits in the BDOF process.

[0283] To derive gradient values, prediction samples I (k) (i,j) existing outside the current CU in the list k (k = 0, 1) can be generated. FIG. 17 is a view illustrating a CU to which the extension is performed to perform BDOF.

[0284] As shown in FIG. 17 , to perform BDOF, rows / columns extended around the boundary of the CU can be used. To control the computational complexity of generating prediction samples outside the boundary, prediction samples in the white region in the extension region ( FIG. 17 ) can be generated using a bilinear filter, and prediction samples in the gray region in the CU ( FIG. 17 ) can be generated using a normal 8-tap motion compensation interpolation filter. Sample values at the extension positions can be used only for gradient calculation. When sample values and / or gradient values located outside the CU boundary are needed to perform the remaining steps of the BDOF process, the nearest neighboring sample values and / or gradient values can be padded (repeated) and used.

[0285] When the width and / or height of the CU is greater than 16 luma samples, the corresponding CU can be divided into sub-blocks having a width and / or height of 16 luma samples. The boundaries of the sub-blocks can be treated in the same way as the above-described CU boundaries in the BDOF process. The maximum unit size to perform the BDOF process can be limited to 16x16.

[0286] For each sub-block, it can be determined whether to perform BDOF. That is, the BDOF process for each sub-block can be skipped. For example, when the SAD value between the initial L0 prediction sample and the initial L1 prediction sample is less than a predetermined threshold, the BDOF process can not be applied to the corresponding sub-block. In this case, when the width and height of the corresponding sub-block are W and H, the predetermined threshold can be set to (8*W*(H>>1). Considering the complexity of additional SAD calculation, the SAD between the initial L0 prediction sample and the initial L1 prediction sample calculated in the DMVR process can be reused.

[0287] BDOF can not be applied when BCW is available for the current block, e.g., when the BCW weight index specifies unequal weights. Similarly, BDOF can not be applied when WP is available for the current block, e.g., when luma_weight_lx_flag of at least one of the two reference pictures is 1. In this case, luma_weight_lx_flag can be information specifying whether a weighting factor for the luma component of lx prediction (x is 0 or 1) exists in the bitstream or information specifying whether WP is applied to the luma component of lx prediction. BDOF can not be applied when a CU is coded in a symmetric MVD (SMVD) mode or a CIIP mode.

[0288] Optical flow prediction refinement (PROF)

[0289] Hereinafter, a method of refining a subblock-based affine motion compensation predicted block by applying optical flow will be described. The predicted samples generated by performing subblock-based affine motion compensation can be refined based on a difference derived by an optical flow equation. The refinement of these predicted samples in the present disclosure can be referred to as optical flow prediction refinement (PROF). By PROF, inter prediction at a pixel level granularity can be achieved without increasing the bandwidth of memory access.

[0290] The parameters of the affine motion model can be used to derive the motion vector of each pixel in a CU. However, since pixel-based affine motion compensation prediction results in high complexity and an increase in memory access bandwidth, subblock-based affine motion compensation prediction can be performed. When subblock-based affine motion compensation prediction is performed, a CU can be divided into 4x4 subblocks, and a motion vector can be determined for each subblock. In this case, the motion vector of each subblock can be derived from the CPMV of the CU. Subblock-based affine motion compensation has a trade-off relationship between encoding efficiency and complexity and memory access bandwidth. Since the motion vector is derived in units of subblocks, complexity and memory access bandwidth are reduced, but prediction accuracy is reduced.

[0291] Therefore, by applying optical flow to subblock-based affine motion compensation prediction, motion compensation at a refined granularity can be achieved by refinement.

[0292] As described above, the luma prediction samples can be refined by adding a difference derived by an optical flow equation after performing subblock-based affine motion compensation. More specifically, PROF can be performed in the following four steps.

[0293] Step 1) A predicted subblock I(i,j) is generated by performing subblock-based affine motion compensation.

[0294] Step 2) Calculate spatial gradient g of the predictor block at each sample position x (i,j) and g y (i,j). In this case, a 3-tap filter can be used, and filter coefficients can be [-1, 0, 1]. For example, the spatial gradient can be calculated as follows.

[0295] [Equation 9]

[0296] g x (i,j) = I(i+1,j) - I(i-1,j)

[0297] g y (i,j) = I(i,j+1) - I(i,j-1)

[0298] To calculate the gradient, the predictor block can be extended by one pixel on each side. In this case, to reduce memory bandwidth and complexity, the pixels at the extended boundary can be copied from the nearest integer pixels in the reference picture. Thus, additional interpolation for the padded region can be skipped.

[0299] Step 3) The luminance prediction refinement (ΔI(i,j)) can be calculated by the optical flow equation. For example, the following equation can be used.

[0300] [Equation 10]

[0301] ΔI(i,j) = gx(i,j) * Δv x (i,j) + g y (i,j) * Δv y (i,j)

[0302] In the above equation, Δv(i,j) denotes the difference between the pixel motion vector (pixel MV, v(i,j)) calculated at the sample position (i,j) and the sub-block MV of the sub-block to which the sample (i,j) belongs.

[0303] FIG. 18 is a view illustrating the relationship between Δv(i,j), v(i,j), and the sub-block motion vector.

[0304] In FIG. 18 the example shown, for example, the difference between the motion vector v(i,j) at the top-left sample position of the current sub-block and the motion vector v SB of the current sub-block can be denoted by the thick dashed arrow, and the vector denoted by the thick dashed arrow can correspond to Δv(i,j).

[0305] The affine model parameters and pixel positions from the center of the sub-block do not change. Thus, Δv(i,j) can be calculated only for the first sub-block and reused for other sub-blocks in the same CU. Assuming the horizontal and vertical offsets from the pixel position to the center of the sub-block are x and y, respectively, Δv(x,y) can be derived as follows.

[0306] [Equation 11]

[0307]

[0308] For the 4-parameter affine model,

[0309]

[0310] For the 6-parameter affine model,

[0311]

[0312] In the above, (v 0x ,v 0y ), (v 1x ,v 1y ) and (v 2x ,v 2y ) correspond to the top-left CPMV, top-right CPMV and bottom-left CPMV, respectively, and w and h represent the width and height of the CU, respectively.

[0313] Step 4) Finally, the final prediction block I'(i,j) can be generated based on the calculated luminance prediction refinement ΔI(i,j) and the prediction sub-block I(i,j). For example, the final prediction block I' can be generated as follows.

[0314] [Equation 12]

[0315] I'(i,j) = I(i,j) + ΔI(i,j)

[0316] Overview of sub-pictures

[0317] As described above, quantization and dequantization of the luminance component and the chrominance component can be performed based on a quantization parameter. In addition, a picture to be encoded can be divided in units of multiple CTUs, slices, tiles or tiles, and further, the picture can be divided in units of multiple sub-pictures.

[0318] Within a picture, a sub-picture can be encoded or decoded regardless of whether or not its preceding sub-pictures are encoded or decoded. For example, different quantization or different resolutions can be applied to multiple sub-pictures.

[0319] In addition, a sub-picture can be processed like a separate picture. For example, a picture to be encoded can be a projected picture or a packed picture in a 360-degree image / video or an omnidirectional image / video.

[0320] In this embodiment, a part of a picture can be rendered or displayed based on a viewport of a user terminal (e.g., a head-mounted display). Thus, to achieve low latency, at least one sub-picture covering the viewport among sub-pictures constructing one picture can be encoded or decoded preferentially or independently from the remaining sub-pictures.

[0321] An encoding result of a sub-picture can be referred to as a sub-bitstream, a sub-stream, or simply a bitstream. A decoding device can decode a sub-picture from a sub-bitstream, a sub-stream, or a bitstream. In this case, high-level syntax (HLS) such as PPS, SPS, VPS, and / or DPS (Decoding Parameter Set) can be used to encode / decode a sub-picture.

[0322] In the disclosure, high-level syntax (HLS) can include at least one of APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, or slice header syntax. For example, APS (APS syntax) or PPS (PPS syntax) can include information / parameters commonly applicable to one or more slices or pictures. SPS (SPS syntax) can include information / parameters commonly applicable to one or more sequences. VPS (VPS syntax) can include information / parameters commonly applicable to multiple layers. DPS (DPS syntax) can include information / parameters commonly applicable to an entire video. For example, DPS can include information / parameters related to a concatenation of coded video sequences (CVS).

[0323] Definition of sub-pictures

[0324] A sub-picture can construct a rectangular region of an encoded picture. The size of a sub-picture can be set differently within a picture. For all pictures belonging to one sequence, the size and position of a certain individual sub-picture can be equally set. Individual sub-picture sequences can be decoded independently. Tiles and slices (and CTBs) can be restricted not to cross sub-picture boundaries. To this end, an encoding device can perform encoding such that sub-pictures are decoded independently. To this end, semantic constraints in a bitstream can be required. In addition, for each picture in one sequence, the arrangement of tiles, slices, and tiles in a sub-picture can be differently constructed.

[0325] Purpose of sub-picture design

[0326] Subpicture design aims at abstraction or encapsulation of a range smaller than the picture level or larger than the slice or tile group level. Thus, VCL NAL units of a subset of a motion constant tile set (MCTS) can be extracted from one VVC bitstream and redirected to another VVC bitstream without the difficulty of modification such as VCL level. Here, MCTS is a coding technique that allows spatial and temporal independence between tiles. When MCTS is applied, information about tiles not included in the MCTS to which the current tile belongs cannot be referenced. When an image is divided into MCTS and coded, independent transmission and decoding of MCTS can be made.

[0327] The advantage of this subpicture design is to change the viewing direction in a mixed resolution viewport dependent 360° streaming scheme.

[0328] Sub-picture use cases

[0329] The use of subpictures is required in a viewport dependent 360° scheme providing extended real spatial resolution on the viewport. For example, a scheme in a tile covering a viewport derived from a 6K (6144x3072) ERP (equirectangular projection) picture or a cubemap projection (CMP) resolution with equivalent 4K decoding performance (HEVC level 5.1) is included in sections D.6.3 and D.6.4 of OMAF and adopted in the VR Industry Forum guideline. This resolution is known to be suitable for a head-mounted display using a quad HD (2560x1440) display panel.

[0330] Encoding: The content can be encoded with two spatial resolutions including a resolution with 1656x1536 cubemap face size and a resolution with 768x768 cubemap face size. In both bitstreams, a 6x4 tile grid can be used and MCTSs can be encoded at respective tile positions.

[0331] Streaming MCTS selection: 12 MCTSs can be selected from the high resolution bitstream and 12 additional MCTSs can be obtained from the low resolution bitstream. Thus, a hemisphere (180°x180°) of the streaming content can be generated from the high resolution bitstream.

[0332] MCTS and bitstream merge decoding: MCTSs of a single time instance are received, which can be merged into an encoded picture with a 1920x4608 resolution conforming to HEVC level 5.1. In another option of the merged picture, four tile columns have a width value of 768, two tile columns have a width value of 384, three tile rows have a height value of 768, thereby constructing a picture consisting of 3840x2304 luma samples. Here, the width and height units can be units of luma sample number.

[0333] Sub-picture signaling

[0334] Signaling of sub-pictures can be performed at SPS level as shown in FIG. 19 FIG. 19 A syntax for signaling sub-picture syntax elements in SPS is shown. In the following, the syntax elements of FIG. 19

[0335] The syntax element pic_width_max_in_luma_samples can specify the maximum width of each decoded picture in units of luma samples with reference to the SPS. The value of pic_width_max_in_luma_samples is greater than 0 and can have a value that is an integer multiple of andMinCbSizeY. Here, MinCbSizeY is a variable that specifies the minimum size of a coding block for a luma component.

[0336] The syntax element pic_height_max_in_luma_samples can specify the maximum height of each decoded picture in units of luma samples with reference to the SPS. pic_height_max_in_luma_samples is greater than 0 and can have a value that is an integer multiple of MinCbSizeY.

[0337] The syntax element subpic_grid_col_width_minus1 can be used to specify the width of each element of the sub-picture identifier grid. For example, subpic_grid_col_width_minus1 can specify the width of each element of the sub-picture identifier grid in units of 4 samples, and a value obtained by adding 1 to subpic_grid_col_width_minus1 can specify the width of each element of the sub-picture identifier grid in units of 4 samples. The length of the syntax element can be Ceil(Log2(pic_width_max_in_luma_samples / 4)) bits long.

[0338] Accordingly, a variable NumSubPicGridCols that specifies the number of columns in the sub-picture grid can be derived as follows.

[0339] NumSubPicGridCols = (pic_width_max_in_luma_samples + subpic_grid_col_width_minus1 * 4 + 3) / (subpic_grid_col_width_minus1 * 4 + 4)

[0340] ​​The syntax element subpic_grid_row_height_minusl can be used to specify the height of each element of the subpicture identifier grid. For example, subpic_grid_row_height_minusl can specify the height of each element of the subpicture identifier grid in units of 4 samples. The value obtained by adding 1 to subpic_grid_row_height_minusl can specify the height of each element of the subpicture identifier grid in units of 4 samples. The length of the syntax element can be Ceil( Log2( pic_height_max_in_luma_samples / 4 ) ) bits long.

[0341] Thus, the variable NumSubPicGridRows, which specifies the number of rows in the subpicture grid, can be derived as follows.

[0342] NumSubPicGridRows = ( pic_height_max_in_luma_samples + subpic_grid_row_height_minusl * 4 + 3 ) / ( subpic_grid_row_height_minusl * 4 + 4 )

[0343] The syntax element subpic_grid_idx[ i ][ j ] can specify the subpicture index for grid location ( i, j ). The length of the syntax element can be Ceil( Log2( max_subpics_minusl + 1 ) ) bits.

[0344] The variables SubPicTop[ subpic_grid_idx[ i ][ j ] ], SubPicLeft[ subpic_grid_idx[ i ][ j ] ], SubPicWidth[ subpic_grid_idx[ i ][ j ] ], SubPicHeight[ subpic_grid_idx[ i ][ j ] ], and NumSubPics can be derived as in the algorithm of FIG. 20

[0345] ​The syntax element subpic_treated_as_pic_flag[ i ] can specify whether the subpicture is treated as a normal picture in decoding processes other than in-loop filtering. For example, a first value (e.g., 0) of subpic_treated_as_pic_flag[ i ] can specify that the i-th subpicture of each coded picture in the CVS is not treated as a picture in decoding processes other than in-loop filtering. A second value (e.g., 1) of subpic_treated_as_pic_flag[ i ] can specify that the i-th subpicture of each coded picture in the CVS is treated as a picture in decoding processes other than in-loop filtering. When the value of subpic_treated_as_pic_flag[ i ] is not obtained from the bitstream, the value of subpic_treated_as_pic_flag[ i ] can be derived to be the first value (e.g., 0).

[0346] The syntax element loop_filter_across_subpic_enabled_flag[ i ] can specify whether in-loop filtering is performed across the boundary of the i-th subpicture of each coded picture belonging to the CVS. For example, a first value (e.g., 0) of loop_filter_across_subpic_enabled_flag[ i ] can specify that in-loop filtering is not performed across the boundary of the i-th subpicture of each coded picture belonging to the CVS. A second value (e.g., 1) of loop_filter_across_subpic_enabled_flag[ i ] can specify that in-loop filtering can be performed across the boundary of the i-th subpicture of each coded picture belonging to the CVS. When the value of loop_filter_across_subpic_enabled_flag[ i ] is not obtained from the bitstream, the value of loop_filter_across_subpic_enabled_flag[ i ] can be derived to be the second value.

[0347] Furthermore, for bitstream conformance, the following constraint can be applied. For any two subpictures subpicA and subpicB, when the index of subpicA is less than the index of subpicB, all coded NAL units of subpicA can have lower decoding order than all coded NAL units of subpicB. Alternatively, after performing decoding, the shape of a subpicture needs to have a perfect left boundary and a perfect top boundary that construct a picture boundary or the boundary of a previously decoded subpicture.

[0348] Overview of sub-picture based encoding and decoding

[0349] The following disclosure relates to encoding / decoding of the above-described picture and / or sub-pictures. An encoding apparatus can encode a current picture based on a sub-picture structure. Alternatively, the encoding apparatus can encode at least one sub-picture constructing the current picture and output a (sub-)bitstream including (encoding) information about the at least one sub-picture.

[0350] A decoding apparatus can decode at least a sub-picture belonging to a current picture based on a (sub-)bitstream including (encoding) information about at least one sub-picture.

[0351] FIG. 21 is a view illustrating a method of encoding an image using sub-pictures by an encoding apparatus according to an embodiment. The encoding apparatus can divide an input picture into a plurality of sub-pictures (S2110). The encoding apparatus can encode at least one sub-picture using information about the sub-pictures (S2120). For example, each sub-picture can be independently separated and encoded using information about the sub-pictures. Next, the encoding apparatus can output a bitstream by encoding image information including information about the sub-pictures (S2130). Here, the bitstream of the sub-pictures can be referred to as a sub-stream or a sub-bitstream.

[0352] The information about the sub-pictures will be variously described in the disclosure, for example, there can be information about whether in-loop filtering can be performed across a boundary of a sub-picture, information about a sub-picture region, information about using a grid pitch of a sub-picture, etc.

[0353] FIG. 22 is a view illustrating a method of decoding an image using sub-pictures by a decoding apparatus according to an embodiment. The decoding apparatus can obtain information about sub-pictures from a bitstream (S2210). Next, the decoding apparatus can derive at least one sub-picture (S2220) and decode the at least one sub-picture (S2230).

[0354] As such, the decoding apparatus can decode at least one sub-picture, and thus output at least one decoded sub-picture or a current picture including at least one sub-picture. The bitstream can include a sub-stream or a sub-bitstream of the sub-pictures.

[0355] As described above, the information about the sub-pictures can be constructed in an HLS of the bitstream. The decoding apparatus can derive at least one sub-picture based on the information about the sub-pictures. The decoding apparatus can decode the sub-pictures based on a CABAC method, a prediction method, a residual processing method (transform, quantization), an in-loop filtering method, etc.

[0356] When the decoded sub-pictures are output, the decoded sub-pictures can be output together in the form of an OPS (Output Sub-Picture Set). For example, when the current picture is related to a 360° or omnidirectional image / video and is partially rendered, only some sub-pictures can be decoded, and some or all of the decoded sub-pictures can be rendered according to a viewport of a user.

[0357] When the information specifying whether in-loop filtering across a sub-picture boundary is available specifies availability, the decoding device can perform in-loop filtering (e.g., a deblocking filter) on a sub-picture boundary located between two sub-pictures. In addition, when the sub-picture boundary is equal to a picture boundary, in-loop filtering processing on the sub-picture boundary can not be applied.

[0358] The present disclosure relates to sub-picture-based encoding / decoding. Hereinafter, BDOF and PROF to which embodiments of the present disclosure are applicable will be described in more detail.

[0359] As described above, by applying BDOF in inter prediction processing to refine reference samples in motion compensation processing, the compression performance of an image can be increased. BDOF can be performed in a normal mode. That is, BDOF is not performed in the case of an affine mode, a GPM mode, or a CIIP mode.

[0360] FIG. 23 is a view illustrating a process of deriving a prediction sample of a current block by applying BDOF.

[0361] FIG. 23 The BDOF-based inter prediction process of can be performed by an image encoding device and an image decoding device.

[0362] First, in step S2310, motion information of a current block can be derived. The motion information of the current block can be derived by various methods described in the present disclosure. For example, the motion information of the current block can be derived by a regular merge mode, an MMVD mode, or an AMVP mode. The motion information can include bi-prediction motion information (L0 motion information and L1 motion information). For example, the L0 motion information can include MVL0 (L0 motion vector) and refIdxL0 (L0 reference picture index), and the L1 motion information can include MVL1 (L1 motion vector) and refIdxL1 (L1 reference picture index).

[0363] Thereafter, prediction samples of the current block can be derived based on the derived motion information of the current block (S2320). Specifically, L0 prediction samples of the current block can be derived based on the L0 motion information. In addition, L1 prediction samples of the current block can be derived based on the L1 motion information.

[0364] Thereafter, the BDOF offset can be derived based on the derived prediction samples (S2330). The BDOF of step S2330 can be performed according to the methods described in the present disclosure. For example, the BDOF offset can be derived based on the gradient (according to the phase) of the L0 prediction samples and the gradient (according to the phase) of the L1 prediction samples.

[0365] Thereafter, based on the LX (X = 0 or 1) prediction samples and the BDOF offset, the refined prediction samples of the current block can be derived (S2340). The refined prediction samples can be used to generate the final prediction block of the current block.

[0366] The image encoding apparatus can generate the prediction samples of the current block based on the method according to FIG. 23 The residual samples can be derived by comparing the prediction samples of the current block generated according to the method according to FIG. 23 to the original samples. As described above, the information about the residual samples (residual information) can be included in the image / video information and encoded and outputted in the form of a bitstream. In addition, as described above, the image decoding apparatus can generate the reconstructed current block based on the prediction samples of the current block generated according to the method according to

[0367] FIG. 24 is a view illustrating input and output of the BDOF process according to an embodiment of the present disclosure.

[0368] As illustrated in FIG. 24 , the input of the BDOF process can include the width nCbW and the height CbH of the current block, the prediction sub-blocks predSamplesL0 and predSamplesL1 of the boundary region expansion predetermined length (e.g., 2), the prediction direction indexes predFlagL0 and predFlagL1, and the reference picture indexes refIdxL0 and refIdxL1. In addition, the input of the BDOF process can further include the BDOF utilization flag bdofUtilizationFlag. In this case, the BDOF utilization flag can be an input that specifies whether the BDOF is applied to the corresponding sub-block in units of the sub-block within the current block.

[0369] In addition, the BDOF process can generate the refined prediction block pbSamples by applying the BDOF based on the input information.

[0370] FIG. 25 is a view illustrating variables used for the BDOF process according to an embodiment of the present disclosure. FIG. 25 may be FIG. 24 the process thereafter.

[0371] As illustrated in FIG. 25As shown, in order to perform BDOF processing, the input bit depth bitDepth of the current block can be set to BitDepth. Y In this case, BitDepth Y It can be derived based on information about the bit depth signaled via the bit stream. Furthermore, various right shifts can be set based on the bit depth. For example, the first shift (shift1), second shift (shift2), third shift (shift3), and fourth shift (shift4) can be set as follows: FIG. 24 The derivation shown is based on bit depth. Additionally, the offset 4 can be set based on shift4. Furthermore, the variable mvRefineThres, used to specify the limiting range for motion refinement, can be set based on bit depth. This will be described below. FIG. 24 The uses of the various variables described in the text.

[0372] FIG. 26 This is a view illustrating a method for generating prediction samples of each sub-block in the current CU based on whether or not BDOF is applied, according to an embodiment of the present disclosure. FIG. 26 It can be FIG. 25 Subsequent processing.

[0373] It can execute on each sub-block in the current CU. FIG. 26 The processing is shown, and in this case, the size of the sub-block can be 4×4. When the BDOF utilization flag bdofUtilizationFlag of the current sub-block is the first value (false, "0"), BDOF may not be applied to the current sub-block. In this case, the predicted samples of the current sub-block are derived from the weighted sum of the L0 and L1 predicted samples, and in this case, the weights applied to the L0 predicted samples and the weights applied to the L1 predicted samples can be the same. FIG. 26 The shift4 and offset4 used in equation (1) can be FIG. 17 The value set in [the document]. When the BDOF utilization flag bdofUtilizationFlag for the current sub-block is the second value (true, "1"), BDOF can be applied to the current sub-block. In this case, a prediction sample for the current sub-block can be generated by the BDOF processing according to this disclosure.

[0374] FIG. 27 This is a view illustrating a method for deriving the gradient, autocorrelation, and cross-correlation of the current sub-block according to an embodiment of this disclosure. FIG. 27 It can be FIG. 26 Subsequent processing.

[0375] Execute on each sub-block in the current CU FIG. 27 The processing shown is as follows, and in this case, the size of the sub-block can be 4×4.

[0376] according to FIG. 27 Based on equations (1) and (2), the positions (h) of each sample position (x, y) in the current sub-block can be derived. x ,h y ). Then, the horizontal and vertical gradients at each sample location can be derived according to equations (3) to (6). Then, the variables used to derive autocorrelation and cross-correlation (the first intermediate parameter diff and the second intermediate parameters tempH and tempV) can be derived according to equations (7) to (9). For example, the first intermediate parameter diff can be derived using the values ​​obtained by applying a right shift of shift2 to the predicted samples predSamplesL0 and predSamplesL1 of the current block. For example, the second intermediate parameters tempH and tempV can be derived by applying a right shift of shift3 to the sum of the gradients in the L0 direction and the L1 direction as shown in equations (8) and (9). Then, autocorrelation and cross-correlation can be derived based on the derived first and second intermediate parameters according to equations (10) to (16).

[0377] FIG. 28 This illustrates the derivation of motion refinement based on embodiments of the present disclosure (v x ,v y A view of the method for deriving BDOF offsets and generating predicted samples for the current sub-block. FIG. 28 It can be FIG. 27 Subsequent processing.

[0378] Execute on each sub-block in the current CU FIG. 28 The processing shown is as follows, and in this case, the size of the sub-block can be 4×4.

[0379] according to FIG. 28 The motion refinement (v) can be derived from equations (1) and (2). x ,v y Motion refinement can be limited to the range specified by mvRefineThres. Furthermore, based on the motion refinement and gradient, the BDOF offset bdofOffset can be derived according to equation (3). The derived BDOF offset can be used to generate prediction samples pbSamples for the current sub-block according to equation (4).

[0380] By continuously executing the reference FIG. 24 to FIG. 28 The described method can implement BDOF processing according to the first embodiment of this disclosure. In accordance with... FIG. 24 to FIG. 28In an embodiment of the disclosure, the first shift shift1 is set as Max(6, bitDepth-6), and the mvRefineThres is set as 1 « Max(, bitDepth-7). Thus, the bit width of predSample and each parameter of BDOF according to BitDepth can be derived as shown in the following table.

[0381] [Table 1]

[0382]

[0383] As described above, by applying BDOF in inter prediction processing to refine reference samples in motion compensation processing, the compression performance of an image can be increased. BDOF can be performed in a normal mode. That is, BDOF is not performed in the case of an affine mode, a GPM mode, or a CIIP mode.

[0384] As a method similar to BDOF, PROF can be performed on a block coded in an affine mode. As described above, by refining reference samples in each 4x4 sub-block via PROF, the compression performance of an image can be increased.

[0385] According to the disclosure, the above-described affine motion (sub-block motion) information of a current block can be derived, and the affine motion information can be refined or the prediction sample derived based on the affine motion information can be refined through the above-described PROF processing.

[0386] FIG. 29 is a view illustrating a process of deriving a prediction sample of a current block by applying PROF.

[0387] FIG. 29 The PROF-based inter prediction process of the disclosure can be performed by an image encoding apparatus and an image decoding apparatus.

[0388] First, in step S2910, motion information of a current block can be derived. The motion information of the current block can be derived through various methods described in the disclosure. For example, the motion information of the current block can be derived through the methods described in the above-described affine mode or sub-block-based TMVP mode. The motion information can include sub-block motion information of the current block. The sub-block motion information can include bi-predictive sub-block motion information (L0 sub-block motion information and L1 sub-block motion information). For example, the L0 sub-block motion information can include sbMVL0 (L0 sub-block motion vector) and refIdxL0 (L0 reference picture index), and the L1 sub-block motion information can include sbMVL1 (L1 sub-block motion vector) and refIdxL1 (L1 reference picture index).

[0389] Thereafter, prediction samples of the current block can be derived based on the derived motion information of the current block (S2920). Specifically, L0 prediction samples of each subblock of the current block can be derived based on the L0 subblock motion information. In addition, L1 prediction samples of each subblock of the current block can be derived based on the L1 subblock motion information.

[0390] Thereafter, a PROF offset can be derived based on the derived prediction samples (S2930). The PROF of step S2930 can be performed according to the methods described in the present disclosure. For example, a difference motion vector diffMv and a gradient of the LX (X=0 or 1) prediction samples can be calculated, and based on these, a PROF offset dl or DI can be derived according to the methods described in the present disclosure. Various examples of the present disclosure are related to difference motion vector derivation, gradient derivation, and / or PROF offset derivation.

[0391] Thereafter, based on the LX (X=0 or 1) prediction samples and the PROF offset, refined prediction samples of the current block can be derived (S2940). The refined prediction samples can be used to generate a final prediction block of the current block. For example, the final prediction block of the current block can be generated by weighted summing the refined L0 prediction samples and the refined L1 prediction samples.

[0392] The image encoding apparatus can generate prediction samples of the current block based on the method according to FIG. 29 The residual samples are derived by comparison with the original samples based on the prediction samples of the current block generated according to the method according to FIG. 29 The image decoding apparatus can generate the reconstructed current block based on the prediction samples of the current block generated according to the method according to

[0393] FIG. 30 is a view illustrating an example of PROF processing according to the present disclosure.

[0394] According to the example according to FIG. 30 The PROF processing can be performed using, as inputs, the width sbWidth, the height sbHeight of the current subblock, the prediction subblock predSamples extended by the predetermined length of the border region borderExtention, and the difference motion vector diffMv. In this case, for example, the prediction subblock can be a prediction subblock generated by performing affine motion compensation. As a result of performing the PROF processing, a refined prediction subblock pbSamples can be generated.

[0395] To perform the PROF processing, a predetermined first shift shift1 can be calculated. The first shift can be based on the bit depth BitDepthY For example, the first shift can be derived as the maximum of 6 and (BitDepth Y - 6).

[0396] Thereafter, a horizontal gradient gradientH, g x and a vertical gradient gradientV, g y may be calculated for each sample position (x, y) of the input predictor block. The horizontal gradient and the vertical gradient can be calculated according to equation (1) and equation (2), respectively. FIG. 30

[0397] Thereafter, based on the horizontal gradient, the vertical gradient, and the differential motion vector diffM v , a PROF offset di or AI for each sample position can be calculated. For example, the PROF offset can be calculated according to equation (3). FIG. 30 In equation (3), the differential motion vector diffM v used for calculating the PROF offset can mean AV described with reference to FIG. 18 In this case, diffMv can be clipped by dmvLimit as follows, and dmvLimit can be calculated based on BitDepth Y as follows.

[0398] [Equation 13]

[0399] dmvLimit = 1 « Max(5, BitDepthY - 7),

[0400] diffMv[x][y][i] = Clip3(-dmvLimit, dmvLimit - 1, diffMv[x][y][i])

[0401] Thereafter, a refined predictor block pbSamples can be derived based on the calculated PROF offset and the predictor block predSamples. For example, the refined predictor block can be derived according to equation (4). FIG. 30

[0402] According to an example of FIG. 30 , the first shift shift1) can be set as Max(6, bitDepth - 6), and dmvLimit can be set as 1 « Max(5, bitDepth - 7). In addition, diffMv can be clipped within a range of [-dmvLimit, dmvLimit - 1]. Accordingly, the bit width of predSample and each parameter of the PROF according to BitDepth Y may be derived as shown in the following table. ​​

[0403] [Table 2]

[0404]

[0405] The present disclosure can provide various embodiments of a case where a subpicture is treated as a picture (e.g., subpic_treated_as_pic_flag == 1) when performing subpicture-based encoding / decoding. For example, in BDOF or PROF processing, the reference sample extraction process can be constrained not to refer to a reference sample included in a subpicture different from a subpicture to which a current block belongs. In addition, in the temporal motion vector predictor subderivation process, the bilinear interpolation process of luma samples, the 8-tap interpolation filter process of luma samples, and / or the chroma sample interpolation process, a constraint according to a subpicture being treated as a picture can be added.

[0406] As described above, in BDOF and / or PROF processing, in order to calculate a gradient, a reference sample extending a predetermined length beyond a boundary of a current block can be used. However, when a current subpicture to which a current block belongs is treated as a picture, the range of the reference sample for calculating the gradient needs to be limited within the same subpicture as the current subpicture.

[0407] As described above, a prediction sample can be modified by bdofOffset in the case of BDOF and by dl in the case of PROF. bdofOffset and dl can be obtained based on a gradient. The gradient can be derived based on a difference between reference samples in a reference picture. In order to derive the gradient, a reference sample extraction process of extracting a reference sample from a reference picture and an 8-tap interpolation filter process can be performed. The output of the reference sample extraction process can be a luma sample at an integer pixel position.

[0408] FIG. 31 is a view illustrating a case where a reference sample is to be extracted across a boundary of a subpicture.

[0409] In FIG. 31 , an area to be extracted (an extraction area) in a reference picture can be specified according to a motion vector of a current block included in a current subpicture [1] in a current picture. In this case, the extraction area can exist across a boundary of the subpicture [1] in the reference picture. That is, the reference sample to be extracted can be included in a subpicture different from the current subpicture [1].

[0410] FIG. 32 is an enlarged view of the extraction area of FIG. 31 .

[0411] As FIG. 32As shown, the reference samples to be extracted can be included in a subpicture (subpicture[0], subpicture[2], subpicture[3]) different from the current subpicture [1]. Consider FIG. 32 As shown, when a subpicture is treated as a picture, the range of the reference samples to be extracted needs to be limited within the same subpicture as the current subpicture.

[0412] FIG. 33 is a view illustrating a reference sample extraction process according to an embodiment of the disclosure.

[0413] In the reference sample extraction process of FIG. 33 , information about a motion vector of the current block and a reference picture refPicLX L may be input. The information about the motion vector of the current block specifies an extraction region and can be an integer sample position (xInt L , yInt L ) derived from the motion vector of the current block.

[0414] FIG. 33 The output of the reference sample extraction process of

[0415] As shown in FIG. 33 , it can be determined whether the current subpicture is treated as a picture. For example, when subpic_treated_as_pic_flag is 1, it can be determined that the current subpicture is treated as a picture and the extraction region can be limited within the same subpicture as the current subpicture.

[0416] When the current subpicture is treated as a picture, as shown in FIG. 33 , an x-coordinate xInt specifying a position of a sample to be extracted can be clamped within a range of [SubPicLeftBoundaryPos, SubPicRightBoundaryPos] as shown in Equation (1). SubPicLeftBoundaryPos can specify a position of a left boundary of the current subpicture. In addition, SubPicRightBoundaryPos can specify a position of a right boundary of the current subpicture. According to the above Equation (1), since the x-coordinate of the reference sample to be extracted exists within the range of SubPicLeftBoundaryPos to SubPicRightBoundaryPos, the reference sample located outside the left or right boundary of the subpicture is not extracted. A method of deriving SubPicLeftBoundaryPos and SubPicRightBoundaryPos will be described later.

[0417] Similarly, when the current sub-screen is treated as a screen, such as FIG. 33 As shown in Equation (2), the y-coordinate yInt of the location of the sample to be extracted can be limited to the range [SubPicTopBoundaryPos, SubPicBotBoundaryPos]. SubPicTopBoundaryPos can specify the position of the upper boundary of the current sub-screen. In addition, SubPicBotBoundaryPos can specify the position of the lower boundary of the current sub-screen. According to Equation (2) above, the y-coordinate of the reference sample to be extracted exists within the range of SubPicTopBoundaryPos to SubPicBotBoundaryPos, and reference samples existing outside the upper or lower boundary of the sub-screen are not extracted. The method of deriving SubPicTopBoundaryPos and SubPicBotBoundaryPos will be described later.

[0418] When the current sub-picture is not treated as a picture, for example, when subpic_treated_as_pic_flag is 0, the coordinates of the reference sample to be extracted can be derived according to equations (3) and (4). According to equations (3) and (4), the coordinates of the reference sample to be extracted are not limited by the boundary position of the sub-picture. According to equations (3) and (4), the coordinates of the reference sample to be extracted can be limited to the range of the current picture.

[0419] Subsequently, according to equation (5), the extraction of reference samples from the reference image can be performed based on the coordinates (xInt, yInt) of the reference sample to be extracted.

[0420] FIG. 34 This is a flowchart illustrating the extraction process of a reference sample according to this disclosure.

[0421] FIG. 33 and FIG. 34 The reference sample extraction process can be performed by image encoding and image decoding devices used to perform BDOF and / or PROF.

[0422] Reference FIG. 34 First, it can be determined whether the current sub-pic is treated as a pic (S3410). The determination of step S3410 can be performed based on subpic_treated_as_pic_flag.

[0423] When the current sub-picture is treated as a picture (e.g., subpic_treated_as_pic_flag == 1), the extraction position can be derived (S3420), and the derived extraction position can be limited (S3430). The derivation and limiting of the extraction position can be based on... FIG. 33The limiting in step S3430 can be performed by using equations (1) and (2). When the derived extraction position is outside the boundary of the current sub-screen, the corresponding extraction position can be changed to a position in the current sub-screen (e.g., the boundary position of the current sub-screen).

[0424] When the current sub-picture is not treated as a picture (e.g., subpic_treated_as_pic_flag == 0), the extraction position can be deduced (S3440). For example, the extraction position can be deduced based on... FIG. 33 Equations (3) and (4) are used to execute the procedure.

[0425] Subsequently, reference sample extraction can be performed based on the extraction location derived in step S3430 or step S3440 (S3450). For example, reference sample extraction can be based on... FIG. 33 The formula (5) is used to execute the operation.

[0426] according to FIG. 34 In the implementation shown, when a sub-picture is treated as a picture, reference samples outside the boundary of the current sub-picture may not be extracted during the reference sample extraction process. That is, reference samples in the reference picture that belong to a sub-picture different from the current sub-picture may not be referenced.

[0427] The fractional sample interpolation process according to another embodiment of this disclosure will now be described.

[0428] As described above, when the motion vector of the current block specifies a fractional sample unit, an interpolation process can be performed, and the predicted sample of the current block can be derived accordingly based on the reference sample of the fractional sample unit in the reference frame.

[0429] FIG. 35 This is a view illustrating a portion of the fractional sample interpolation process according to this disclosure.

[0430] To perform fractional sample interpolation, the variables fRefWidth and fRefHeight can be derived. For example... FIG. 35 As shown, fRefWidth and fRefHeight can be derived differently depending on whether the current subpic is treated as a pic (subpic_treated_as_pic_flag).

[0431] When the current subpic is treated as a picture, for example, when subpic_treated_as_pic_flag is 1, fRefWidth and fRefHeight can be derived as follows.

[0432] fRefWidth=(SubPicWidth[SubPicIdx]*(subpic_grid_col_width_minus1+1)*4)

[0433] fRefHeight=(SubPicHeight[SubPicIdx]*(subpic_grid_row_height_minus1+1)*4)

[0434] In the above text, `SubPicWidth[SubPicIdx]` and `SubPicHeight[SubPicIdx]` can refer to the width and height of the current subpicture, respectively. In this case, the width and height of the subpicture can be represented by the number of grid cells configured in the subpicture. For example, a subpicture width of 4 can mean that the corresponding subpicture includes four grid cells in the horizontal direction. Additionally, `subpic_grid_col_width_minus1` and `subpic_grid_row_height_minus1` can refer to the width and height of the grid cells configured in the subpicture, respectively. In this case, the width and height of the grid cells can be represented in units of 4 pixels. For example, a grid width of 4 can mean that the grid width is 16 (4×4) pixels.

[0435] When the current subpic is not treated as a picture, for example, when subpic_treated_as_pic_flag is 0, fRefWidth and fRefHeight can be derived as the width PicOutputWidthL and height PicOutputHeightL of the output image of the reference picture, respectively.

[0436] The fRefWidth and fRefHeight derived as described above can be used based on FIG. 35 Equations (1) and (2) are used to derive the scaling factors hori_scale_fp and vert_scale_fp. Subsequently, based on equations (3) to (6), the position (refx) of the reference sample specified by the motion vector can be derived. L ,refy L Based on the position derived from the reference sample, the integer position (xInt) can be derived. L ,yInt L ) and fractional position (xFrac) L ,yFrac L The aforementioned reference sample extraction or 8-tap interpolation filtering can be performed based on the derived integer and / or fractional positions.

[0437] The following describes another embodiment of the sbTMVP derivation method according to this disclosure.

[0438] For reference FIG. 16 The described motion shift can be derived and applied to the current block, thereby specifying the sub-blocks (col sub-blocks) in the col frame that correspond to the various sub-blocks configured in the current block. Subsequently, using the motion information of the corresponding sub-blocks (col sub-blocks) in the col frame, the motion information of each sub-block in the current block can be derived. Based on the derived motion information of the sub-blocks, sbTMVP can be derived.

[0439] FIG. 36 This is a view illustrating a portion of the sbTMVP derivation method according to this disclosure.

[0440] FIG. 36 This illustrates a portion of the processing following the derivation of the motion shift of the current block during the sbTMVP derivation process. Specifically, FIG. 36 The process shown includes deriving the position of the corresponding col sub-block for each sub-block configured in the current block.

[0441] according to FIG. 37 First, based on equations (1) and (2), the position (xSb, ySb) of the lower right center sample of the current sub-block can be derived. Then, the position (xColSb, yColSb) of the col sub-block can be derived based on whether the current sub-picture is treated as a picture (e.g., subpic_treated_as_pic_flag).

[0442] Specifically, when subpic_treated_as_pic_flag is 1, the y-coordinate yColSb of the col sub-block can be derived according to equation (3). In this case, yColSb derived from ySb and motion shift tempMv can be bounded to a position specified by SubPicTopBoundaryPos, SubPicBotBoundaryPos, and the y-coordinate yCtb of the current CTB. Through equation (3), the y-coordinate of the col sub-block exists in the current CTB and the current sub-picture. Additionally, when subpic_treated_as_pic_flag is 0, the y-coordinate yColSb of the col sub-block can be derived according to equation (4). In this case, yColSb derived from ySb and motion shift tempMv can be bounded to a position specified by the y-coordinate yCtb of the current CTB and the height of the current picture. Through equation (4), the y-coordinate of the col sub-block exists in the current CTB and the current picture.

[0443] Similarly, when subpic_treated_as_pic_flag is 1, the x-coordinate xColSb of the col sub-block can be derived according to equation (5). In this case, xColSb, derived from xSb and the motion shift tempMv, can be bounded to a position specified by SubPicLeftBoundaryPos, SubPicRightBoundaryPos, and the x-coordinate xCtb of the current CTB. Through equation (5), the x-coordinate of the col sub-block exists in the current CTB and the current sub-picture. Additionally, when subpic_treated_as_pic_flag is 0, the x-coordinate yColSb of the col sub-block can be derived according to equation (6). In this case, xColSb, derived from xSb and the motion shift tempMv, can be bounded to a position specified by the x-coordinate xCtb of the current CTB and the width of the current picture. Through equation (6), the x-coordinate of the col sub-block exists in the current CTB and the current picture.

[0444] According to this disclosure, when the current sub-screen is treated as a screen, the col sub-block used to derive sbTMVP for the current sub-block exists in the same sub-screen as the current sub-screen.

[0445] The following describes a method for deriving the sub-screen boundary position according to another embodiment of the present disclosure.

[0446] FIG. 38 This is a view illustrating a method for deriving the position of a sub-screen boundary according to this disclosure.

[0447] According to this disclosure, the positions of the left boundary of a sub-screen, SubPicLeftBoundaryPos, the right boundary of a sub-screen, SubPicRightBoundaryPos, the top boundary of a sub-screen, SubPicTopBoundaryPos, and the bottom boundary of a sub-screen, SubPicBotBoundaryPos, can be derived. In this disclosure, SubPicIdx can be an index used to identify each sub-screen in the current screen.

[0448] When the current sub-picture specified by SubPicIdx is treated as a picture, SubPicLeftBoundaryPos can be derived based on the position information SubPicLeft of the predetermined cell specifying the left position of the current sub-picture and the width information subpic_grid_col_width_minus1 of the corresponding cell. The predetermined cell can be a grid. However, this disclosure is not limited to this; for example, the predetermined cell can be a CTU. When the predetermined cell is a CTU, the left position of the current sub-picture can be derived as the product of the position information of the CTU cell specifying the left position of the current sub-picture and the size of the CTU.

[0449] When the current sub-screen is treated as a screen, SubPicRightBoundaryPos can be derived based on the position information of a predetermined cell specifying the left position of the current sub-screen, the width information of a predetermined cell specifying the width of the current sub-screen, and the width information of the corresponding cell. For example, the position information of a predetermined cell specifying the right position of the current sub-screen can be derived by adding the left position and the width of the current sub-screen. SubPicRightBoundaryPos can be derived by performing a "-1" operation on the final calculated value. The predetermined cell can be a grid. However, this disclosure is not limited to this; for example, the predetermined cell can be a CTU. When the predetermined cell is a CTU, the right position of the current sub-screen can be derived by performing a "-1" operation on the product of the position information of the CTU cell specifying the right position of the current sub-screen and the size of the CTU. The position information of the CTU cell specifying the right position of the current sub-screen can be derived as the sum of the position information of the CTU cell specifying the left position of the current sub-screen and the width information of the CTU cell specifying the width of the current sub-screen.

[0450] Similarly, SubPicTopBoundaryPos can be derived based on the position information SubPicTop of a predetermined cell specifying the upper position of the current sub-screen and the height information subpic_grid_low_height_minus1 of the corresponding cell. The predetermined cell can be a grid. However, this disclosure is not limited thereto; for example, the predetermined cell can be a CTU. When the predetermined cell is a CTU, the upper position of the current sub-screen can be derived as the product of the position information of the CTU cell specifying the upper position of the current sub-screen and the size of the CTU.

[0451] When the current sub-screen is treated as a screen, SubPicBotBoundaryPos can be derived based on the position information of a predetermined cell at the upper position of the current sub-screen, the height information of a predetermined position at the height of the current sub-screen, and the height of the corresponding cell. For example, the position information of a predetermined cell at the lower position of the current sub-screen can be derived by adding the upper position of the current sub-screen to the height of the current sub-screen. SubPicBotBoundaryPos can be derived by performing a "-1" operation on the final calculated value. The predetermined cell can be a grid. However, this disclosure is not limited to this; for example, the predetermined cell can be a CTU. When the predetermined cell is a CTU, the lower position of the current sub-screen can be derived by performing a "-1" operation on the product of the position information of the CTU cell at the lower position of the current sub-screen and the size of the CTU. The position information of the CTU cell at the lower position of the current sub-screen can be derived as the sum of the position information of the CTU cell at the upper position of the current sub-screen and the height information of the CTU cell at the height of the current sub-screen.

[0452] The various embodiments described in this disclosure can be implemented individually or in combination with other embodiments. Alternatively, some embodiments can be added to another embodiment, or some embodiments can be replaced by other embodiments.

[0453] Although the exemplary methods of this disclosure described above are represented as a series of operations for clarity of description, they are not intended to limit the order in which the steps are performed, and these steps may be performed simultaneously or in different orders if necessary. To implement the method according to the invention, the described steps may further include other steps, including steps in addition to some steps, or may include additional steps in addition to some steps.

[0454] In this disclosure, the image encoding device or image decoding device that performs a predetermined operation (step) can perform an operation (step) that confirms the execution conditions or circumstances of the corresponding operation (step). For example, if it is described that a predetermined operation is performed when predetermined conditions are met, the image encoding device or image decoding device can perform the predetermined operation after determining whether the predetermined conditions are met.

[0455] The various embodiments of this disclosure are not a list of all possible combinations and are intended to describe representative aspects of this disclosure; the matters described in the various embodiments may be applied independently or in combination of two or more.

[0456] Various embodiments of this disclosure can be implemented in hardware, firmware, software, or a combination thereof. When this disclosure is implemented in hardware, it can be implemented using application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, etc.

[0457] Furthermore, the image decoding and image encoding devices applying the embodiments of this disclosure can be included in multimedia broadcasting transmission and receiving devices, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video chat devices, real-time communication devices such as video communication, mobile streaming devices, storage media, cameras, video-on-demand (VoD) service providers, OTT (over-the-top) video devices, internet streaming service providers, three-dimensional (3D) video devices, video telephony devices, medical video devices, etc., and can be used to process video signals or data signals. For example, OTT video devices can include game consoles, Blu-ray players, internet access televisions, home theater systems, smartphones, tablet PCs, digital video recorders (DVRs), etc.

[0458] FIG. 38 This is a view illustrating a content streaming system to which embodiments of the present disclosure can be applied.

[0459] like ​ As shown, the content streaming system applying the embodiments of this disclosure may mainly include an encoding server, a streaming server, a network server, a media storage device, a user device, and a multimedia input device.

[0460] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream and then sends the bitstream to the streaming server. As another example, when multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server can be omitted.

[0461] The bitstream can be generated by an image encoding method or image encoding device applying the embodiments of this disclosure, and the stream server can temporarily store the bitstream during the sending or receiving of the bitstream.

[0462] A streaming server sends multimedia data to a user's device based on a request from a web server, and the web server acts as a medium for informing the user of the service. When a user requests a service from the web server, the web server can deliver it to the streaming server, and the streaming server can send the multimedia data to the user. In this scenario, the content streaming system may include a separate control server. In this case, the control server is used to control the commands / responses between devices in the content streaming system.

[0463] A streaming server can receive content from media storage devices and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a predetermined period of time.

[0464] Examples of user devices may include mobile phones, smartphones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head-mounted displays), digital televisions, desktop computers, digital signage, etc.

[0465] In a content streaming system, each server can operate as a distributed server, in which case the data received from each server can be distributed.

[0466] The scope of this disclosure includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) for enabling the operation of methods according to various embodiments to be executed on a device or computer, and non-transitory computer-readable media having such software or commands stored thereon and executable on a device or computer.

[0467] Industrial applicability

[0468] The embodiments disclosed herein can be used to encode or decode images.

Claims

1. An image decoding method performed by an image decoding device, the image decoding method comprising the following steps: Determine whether bidirectional optical flow (BDOF) or optical flow prediction refinement (PROF) is applied to the current block; Based on BDOF or PROF applied to the current block, a predicted sample of the current block is generated from a reference image of the current block based on the motion information of the current block; as well as By applying BDOF or PROF to the current block based on the generated prediction samples, refined prediction samples for the current block are derived. The application of BDOF or PROF to the current block is performed on a sub-block basis. The step of generating the predicted sample for the current block is based on whether the current sub-image including the current block is treated as an image. Whether the current sub-image is treated as an image is determined based on the flag information notified by the sequence parameter set SPS; When the value of the flag information is 1, the current sub-image is determined to be treated as an image; The step of generating the predicted sample for the current block is performed based on the position within the reference image used to generate the predicted sample. The location within the reference image is limited to a predetermined range. Wherein, based on the current sub-image being regarded as the image, the predetermined range is specified by the boundary position of the current sub-image, wherein the position within the reference image includes x-coordinates and y-coordinates, wherein the x-coordinate is limited to the range of the left and right boundary positions of the current sub-image, and wherein the y-coordinate is limited to the range of the upper and lower boundary positions of the current sub-image. The left boundary position of the current sub-image is derived as the product of the position information of a predetermined unit specifying the left position of the current sub-image and the width of the predetermined unit. The right boundary position of the current sub-image is derived by performing a "-1" operation on the product of the position information of a predetermined unit specifying the right position of the current sub-image and the width of the predetermined unit. Wherein, the upper boundary position of the current sub-image is derived as the product of the position information of a predetermined unit specifying the upper position of the current sub-image and the height of the predetermined unit, and The lower boundary position of the current sub-image is derived by performing a "-1" operation on the product of the position information of the predetermined unit specifying the lower position of the current sub-image and the height of the predetermined unit.

2. An image encoding method performed by an image encoding device, the image encoding method comprising the following steps: Determine whether bidirectional optical flow (BDOF) or optical flow prediction refinement (PROF) is applied to the current block; Based on BDOF or PROF applied to the current block, a predicted sample of the current block is generated from a reference image of the current block based on the motion information of the current block; as well as By applying BDOF or PROF to the current block based on the generated prediction samples, refined prediction samples for the current block are derived. The application of BDOF or PROF to the current block is performed on a sub-block basis. The step of generating the predicted sample for the current block is based on whether the current sub-image including the current block is treated as an image. Whether the current sub-image is treated as an image is determined based on the flag information notified by the sequence parameter set SPS; When the value of the flag information is 1, the current sub-image is determined to be treated as an image; The step of generating the predicted sample for the current block is performed based on the position within the reference image used to generate the predicted sample. The location within the reference image is limited to a predetermined range. Wherein, based on the current sub-image being regarded as the image, the predetermined range is specified by the boundary position of the current sub-image, wherein the position within the reference image includes x-coordinates and y-coordinates, wherein the x-coordinate is limited to the range of the left and right boundary positions of the current sub-image, and wherein the y-coordinate is limited to the range of the upper and lower boundary positions of the current sub-image. The left boundary position of the current sub-image is derived as the product of the position information of a predetermined unit specifying the left position of the current sub-image and the width of the predetermined unit. The right boundary position of the current sub-image is derived by performing a "-1" operation on the product of the position information of a predetermined unit specifying the right position of the current sub-image and the width of the predetermined unit. Wherein, the upper boundary position of the current sub-image is derived as the product of the position information of a predetermined unit specifying the upper position of the current sub-image and the height of the predetermined unit, and The lower boundary position of the current sub-image is derived by performing a "-1" operation on the product of the position information of the predetermined unit specifying the lower position of the current sub-image and the height of the predetermined unit.

3. A method for transmitting a bit stream, comprising: Perform the image encoding method according to claim 2 to generate the bitstream; as well as Send the bit stream.

Images