Method for encoding images, method for decoding images, and non-volatile machine-readable storage media

Abstract

FIELD: image encoding and decoding.SUBSTANCE: determining whether bidirectional optical flow (BDOF) or prediction refinement using optical flow (PROF) is applied to the current block when BDOF or PROF is applied to the current block, sampling a prediction sample of the current block from a reference frame of the current block based on motion information of the current block; and extracting an improved prediction sample for the current block by applying BDOF or PROF to the current block based on the sampled prediction sample.EFFECT: ensuring efficient encoding and decoding of images.7 cl, 38 dwg, 2 tbl

Description

Field of technology to which the invention relates

[0001] The present disclosure relates to an image encoding method and an image decoding method, and more particularly to an image encoding method and an image decoding method using filtering, as well as to a non-volatile computer-readable storage medium.Background of the invention

[0002] Recently, the demand for high-quality, high-resolution images, such as high definition (HD) images and ultra-high definition (UHD) images, is growing in various technical fields.

[0003] As the resolution and quality of image data increases, the amount of information or bits transmitted increases relatively compared to the existing image data.

[0004] Increasing the amount of information or bits transmitted results in increased transmission costs and storage costs.

[0005] Accordingly, there is a need for a highly efficient image compression technology for efficiently transmitting, storing and reproducing information of relatively high-quality, high-resolution images.Essence of the inventionTechnical problem

[0006] The purpose of the present disclosure is to provide an image encoding method and an image decoding method with improved encoding / decoding efficiency.

[0007] Another object of the present disclosure is to provide an image encoding method and an image decoding method for encoding / decoding an image based on a subframe.

[0008] Another object of the present disclosure is to provide an image coding method and an image decoding method for performing BDOF or PROF based on determining whether the current subframe is processed as a frame.

[0009] Another object of the present disclosure is to provide a non-volatile computer-readable storage medium that stores a computer program that, when executed on a processor, implements the image encoding method according to the present disclosure.

[0010] The technical problems solved by the present disclosure are not limited to the above technical problems, and other technical problems that are not described herein should become apparent to those skilled in the art from the following description. Technical Solution

[0011] In a first aspect of the present invention, there is provided a method of decoding images performed by an image decoding device, wherein the method of decoding images includes: determining whether bidirectional optical flow (BDOF) or prediction refinement with optical flow (PROF) is applied to a current block; based on the BDOF or PROF applied to the current block, generating a prediction sample of the current block from a reference frame of the current block based on motion information of the current block; and deriving a refined prediction sample for the current block by applying the BDOF or PROF to the current block based on the generated prediction sample, wherein generating the prediction sample of the current block is performed based on whether the current subframe containing the current block is processed as a frame, wherein generating the prediction sample of the current block includes obtaining a reference sample indicated by a position within the reference frame,and shifting to the left the value of the reference sample, wherein the position within the reference frame is clipped in a predetermined range, wherein based on the current subframe processed as a frame, the predetermined range is indicated by the position of the boundary of the current subframe, wherein the position within the reference frame includes an x ​​coordinate and a y coordinate, wherein the x coordinate is clipped in a range from the left boundary position to the right boundary position of the current subframe, wherein the y coordinate is clipped in a range from the upper boundary position to the lower boundary position of the current subframe, wherein the left boundary position of the current subblock is output as a product of the position information of a predetermined unit indicating the left position of the current subframe and the width of the predetermined unit, wherein the right boundary position of the current subframe is output by performing an operation "-1" with respect to the product of the position information of the predetermined unit,indicating the right position of the current subframe and the width of the predetermined unit, wherein the upper position of the boundary of the current subframe is output as the product of the position information of the predetermined unit indicating the upper position of the current subframe and the height of the predetermined unit, and wherein the lower position of the boundary of the current subframe is output by performing the operation "-1" with respect to the product of the position information of the predetermined unit indicating the lower position of the current subframe and the height of the predetermined unit.

[0012] In a preferred embodiment, whether the current subframe is processed as a frame is determined based on flag information transmitted via a bit stream.

[0013] In a preferred embodiment, the flag information is transmitted via a sequence parameter set (SPS).

[0014] In a preferred embodiment, the predetermined unit is a grid or CTU.

[0015] In a preferred embodiment, based on the current subframe that is not processed as a frame, the predetermined range is a range of the current frame containing the current block.

[0016] In a second aspect of the present invention, there is provided an image coding method performed by an image coding device, wherein the image coding method includes: determining whether bidirectional optical flow (BDOF) or prediction refinement with optical flow (PROF) is applied to a current block; based on the BDOF or PROF applied to the current block, generating a prediction sample of the current block from a reference frame of the current block based on motion information of the current block; and deriving a refined prediction sample for the current block by applying the BDOF or PROF to the current block based on the generated prediction sample, wherein generating the prediction sample of the current block is performed based on whether the current subframe containing the current block is processed as a frame, wherein generating the prediction sample of the current block includes obtaining a reference sample indicated by a position within the reference frame,and shifting to the left the value of the reference sample, wherein the position within the reference frame is clipped in a predetermined range, wherein based on the current subframe processed as a frame, the predetermined range is indicated by the position of the boundary of the current subframe, wherein the position within the reference frame includes an x ​​coordinate and a y coordinate, wherein the x coordinate is clipped in a range from the left boundary position to the right boundary position of the current subframe, wherein the y coordinate is clipped in a range from the upper boundary position to the lower boundary position of the current subframe, wherein the left boundary position of the current subblock is output as a product of the position information of a predetermined unit indicating the left position of the current subframe and the width of the predetermined unit, wherein the right boundary position of the current subframe is output by performing an operation "-1" with respect to the product of the position information of the predetermined unit,indicating the right position of the current subframe and the width of the predetermined unit, wherein the upper position of the boundary of the current subframe is output as the product of the position information of the predetermined unit indicating the upper position of the current subframe and the height of the predetermined unit, and wherein the lower position of the boundary of the current subframe is output by performing the operation "-1" with respect to the product of the position information of the predetermined unit indicating the lower position of the current subframe and the height of the predetermined unit.

[0017] In a third aspect of the present invention, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed on a processor, implements the above-mentioned image encoding method.

[0018] The features briefly summarized above with respect to the present disclosure are merely exemplary aspects of the following detailed description of the present disclosure and do not limit the scope of the present disclosure. Advantages of the invention

[0019] According to the present disclosure, it is possible to provide an image encoding method and an image decoding method with improved encoding / decoding efficiency.

[0020] Furthermore, according to the present disclosure, it is possible to provide an image encoding method and an image decoding method for encoding / decoding an image based on a subframe.

[0021] Furthermore, according to the present disclosure, it is possible to provide an image encoding method and an image decoding method for performing BDOF or PROF based on determining whether or not the current subframe is treated as a frame.

[0022] Furthermore, according to the present disclosure, it is possible to provide a non-volatile computer-readable storage medium storing a bit stream generated by the image encoding method according to the present disclosure.

[0023] Those skilled in the art will appreciate that the advantages that can be achieved through the present disclosure are not limited to what is specifically described above, and other advantages of the present disclosure will be more clearly understood from the detailed description.Brief Description of the Drawings

[0024] Fig. 1 is a view schematically illustrating a video coding system to which an embodiment of the present disclosure is applicable.

[0025] Fig. 2 is a view schematically illustrating image encoding equipment to which an embodiment of the present disclosure is applicable.

[0026] Fig. 3 is a view schematically illustrating image decoding equipment to which an embodiment of the present disclosure is applicable.

[0027] Fig. 4 is a flowchart illustrating a video / image coding method based on inter prediction.

[0028] Fig. 5 is a view illustrating the configuration of the inter-prediction module 180 according to the present disclosure.

[0029] Fig. 6 is a flowchart illustrating a video / image decoding method based on inter prediction.

[0030] Fig. 7 is a view illustrating the configuration of the inter-prediction module 260 according to the present disclosure.

[0031] Fig. 8 is a view illustrating the motion expressible in the affine mode.

[0032] Fig. 9 is a view illustrating a parametric model of the affine mode.

[0033] Fig. 10 is a view illustrating a method for forming a list of affine possible combination options.

[0034] Fig. 11 is a view illustrating CPMV being extracted from an adjacent block.

[0035] Fig. 12 is a view illustrating neighboring blocks for extracting an inherited affine merge candidate.

[0036] Fig. 13 is a view illustrating neighboring blocks for extracting the constructed affine candidate union option.

[0037] Fig. 14 is a view illustrating a method for forming a list of affine possible MVP options.

[0038] Fig. 15 is a view illustrating a neighboring block of the TMVP mode based on sub-blocks.

[0039] Fig. 16 is a view illustrating a method for extracting a motion vector field according to the TMVP mode based on sub-blocks.

[0040] Fig. 17 is a view illustrating a CU extended with the ability to perform BDOF.

[0041] Fig. 18 is a view illustrating the relationship between Δv(i, j), v(i, j) and the sub-block motion vector.

[0042] Fig. 19 is a view illustrating an embodiment of syntax for transmitting in signaling a syntax element for subframes in an SPS.

[0043] Fig. 20 is a view illustrating an embodiment of an algorithm for extracting a predetermined variable such as SubPicTop.

[0044] Fig. 21 is a view illustrating a method for encoding an image using a subframe by encoding equipment according to an embodiment.

[0045] Fig. 22 is a view illustrating a method for decoding an image using a subframe by decoding equipment according to an embodiment.

[0046] Fig. 23 is a view illustrating the process of extracting a prediction sample of the current block by applying BDOF.

[0047] Fig. 24 is a view illustrating the input and output of a BDOF process according to an embodiment of the present disclosure.

[0048] Fig. 25 is a view illustrating variables used for a BDOF process according to an embodiment of the present disclosure.

[0049] Fig. 26 is a view illustrating a method for generating a predictive sample for each sub-block in a current CU based on whether or not to apply BDOF, according to an embodiment of the present disclosure.

[0050] Fig. 27 is a view illustrating a method for extracting a gradient, autocorrelation, and cross-correlation of a current sub-block according to an embodiment of the present disclosure.

[0051] Fig. 28 is a view illustrating a method for extracting motion detail (vx, vy), extracting BDOF offset, and generating a prediction sample of a current sub-block, according to an embodiment of the present disclosure.

[0052] Fig. 29 is a view illustrating the process of extracting a prediction sample of the current block by applying PROF.

[0053] Fig. 30 is a view illustrating an example of a PROF process according to the present disclosure.

[0054] Fig. 31 is a view illustrating a case in which a reference sample to be sampled crosses a subframe boundary.

[0055] Fig. 32 is an enlarged view of the sampling region of Fig. 31.

[0056] Fig. 33 is a view illustrating a process of sampling reference samples according to an embodiment of the present disclosure.

[0057] Fig. 34 is a flow chart of a method illustrating a process for sampling reference samples according to the present disclosure.

[0058] Fig. 35 is a view illustrating a portion of a fractional sample interpolation procedure according to the present disclosure.

[0059] Fig. 36 is a view illustrating a part of the sbTMVP extraction method according to the present disclosure.

[0060] Fig. 37 is a view illustrating a method for extracting a boundary position of a subframe according to the present disclosure.

[0061] Fig. 38 is a view showing a content streaming system to which an embodiment of the present disclosure is applicable. Optimal mode for carrying out the invention

[0062] Hereinafter, this document describes embodiments of the present disclosure in detail with reference to the accompanying drawings so that they can be easily implemented by those skilled in the art. However, the present disclosure can be implemented in various forms and is not limited to the embodiments described herein.

[0063] When describing the present disclosure, if it is determined that a detailed description of a related known function or structure would unnecessarily ambiguously define the scope of the present disclosure, the detailed description thereof is omitted. In the drawings, parts not related to the description of the present disclosure are omitted, and like reference numerals are attached to like parts.

[0064] In the present disclosure, when a component is "connected," "coupled," or "linked" to another component, this may include not only a direct connection-based relationship, but also an indirect connection-based relationship in which an intermediate component is present. In addition, when a component "includes" or "has" other components, this means that other components may be additionally included, rather than excluding the other components, unless otherwise specified.

[0065] In the present disclosure, the terms "first," "second," etc. may be used only for the purpose of distinguishing one component from other components and do not limit the order or importance of the components unless otherwise specified. Accordingly, within the scope of the present disclosure, the first component in one embodiment may be referred to as the "second component" in another embodiment, and similarly, the second component in one embodiment may be referred to as the "first component" in another embodiment.

[0066] In the present disclosure, components that differ from each other are intended to clearly describe each feature and do not necessarily mean that the components are separated. Thus, multiple components may be integrated and implemented in a single hardware or software module, or a single component may be distributed and implemented in multiple hardware or software modules. Therefore, even if not otherwise indicated, such embodiments in which components are integrated or a component is distributed are also included within the scope of the present disclosure.

[0067] In the present disclosure, the components described in various embodiments do not necessarily mean essential components, and some components may be optional components. Accordingly, an embodiment consisting of a subset of the components described in an embodiment is also included within the scope of the present disclosure. Furthermore, embodiments including other components in addition to the components described in the various embodiments are included within the scope of the present disclosure.

[0068] The present disclosure relates to image encoding and decoding, and the terms used in the present disclosure may have a general meaning widely used in the technical field to which the present disclosure belongs, unless defined for the first time in the present disclosure.

[0069] In the present disclosure, a "frame" generally refers to a unit representing one image in a specific period of time, and a slice / tile refers to a coding unit that constitutes a portion of a frame. One frame may consist of one or more slices / tiles. In addition, a slice / tile may include one or more coding tree units (CTUs).

[0070] In this disclosure, "pixel" or "pel" may refer to the smallest unit constituting one frame (or image). Furthermore, "sample" may be used as a term corresponding to a pixel. A sample, in general, may represent a pixel or a pixel value, and may represent only the pixel / pixel value of the luminance signal component or only the pixel / pixel value of the chrominance signal component.

[0071] In the present disclosure, a "unit" may represent a basic unit of image processing. A unit may include at least one of a specific region of a frame and information associated with the region. A unit may be used interchangeably with terms such as "sample array," "block," or "zone" in some cases. In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients from M columns and N rows.

[0072] In the present disclosure, the "current block" may refer to one of "the current coding block," "the current coding unit," "the coding target block," "the decoding target block," or "the processing target block." When prediction is performed, the "current block" may refer to "the current prediction block" or "the prediction target block." When transforming (inverse transforming) / quantizing (dequantizing) is performed, the "current block" may refer to "the current transform block" or "the transform target block." When filtering is performed, the "current block" may refer to "the filter target block."

[0073] In this disclosure, the terms " / " and "" should be interpreted as indicating "and / or." For example, the expressions "A / B" and "A, B" may mean "A and / or B." Additionally, "A / B / C" and "A / B / C" may mean "at least one of A, B, and / or C."

[0074] In this disclosure, the term "or" shall be interpreted as indicating "and / or." For example, the expression "A or B" may contain 1) only "A," 2) only "B," and / or 3) "both A and B." In other words, in this disclosure, the term "or" shall be interpreted as indicating "additionally or alternatively."

[0075] Overview of video coding system

[0076] Fig. 1 is a view showing a video coding system according to the present disclosure.

[0077] A video coding system according to an embodiment may include coding equipment 10 and decoding equipment 20. Coding equipment 10 may deliver encoded information or video and / or image data to decoding equipment 20 in the form of a file or streaming via a digital storage medium or a network.

[0078] The encoding equipment 10 according to the embodiment may include a video source generator 11, an encoding module 12, and a transmitter 13. The decoding equipment 20 according to the embodiment may include a receiver 21, a decoding module 22, and a rendering module 23. The encoding module 12 may be called a "video / image encoding module," and the decoding module 22 may be called a "video / image decoding module." The transmitter 13 may be included in the encoding module 12. The receiver 21 may be included in the decoding module 22. The rendering module 23 may include a display, and the display may be configured as a separate device or an external component.

[0079] The video source generator 11 can obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source generator 11 can include a video / image capturing device and / or a video / image generating device. The video / image capturing device can include, for example, one or more cameras, video / image archives including previously captured video / images, and the like. The video / image generating device can include, for example, computers, tablet computers, and smartphones and can (electronically) generate video / images. For example, a virtual video / image can be generated via a computer, etc. In this case, the process of capturing video / images can be replaced by a process of generating associated data.

[0080] Encoding module 12 can encode the input video / image. Encoding module 12 can perform a series of procedures, such as prediction, transformation, and quantization, to achieve compression and encoding efficiency. Encoding module 12 can output encoded data (encoded video / image information) in the form of a bitstream.

[0081] The transmitter 13 can transmit encoded information or video / image data output in the form of a bit stream to the receiver 21 of the decoding equipment 20 via a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter 13 may include an element for generating a multimedia file via a predetermined file format and may include an element for transmitting via a broadcast network / communication network. The receiver 21 can extract / receive the bit stream from the storage medium or network and transmit the bit stream to the decoding module 22.

[0082] The decoding unit 22 may decode a video / image by executing a series of procedures such as dequantization, inverse transform, and prediction corresponding to the operation of the encoding unit 12.

[0083] Rendering module 23 can render decoded video / images. The rendered video / images can be displayed on a monitor.

[0084] Overview of image coding equipment

[0085] Fig. 2 is a view schematically showing image encoding equipment to which an embodiment of the present disclosure is applicable.

[0086] As shown in Fig. 2, the image encoding equipment 100 may include an image segmentation unit 110, a subtractor 115, a converter 120, a quantizer 130, a dequantizer 140, an inverse converter 150, an adder 155, a filter 160, a memory 170, an inter-prediction unit 180, an intra-prediction unit 185, and an entropy encoder 190. The inter-prediction unit 180 and the intra-prediction unit 185 may be collectively referred to as a "prediction unit." The converter 120, the quantizer 130, the dequantizer 140, and the inverse converter 150 may be included in a residual processor. The residual processor may further include a subtractor 115.

[0087] All or at least some of the multiple components configuring the image encoding equipment 100 may be configured by a single hardware component (e.g., an encoder or a processor) in some embodiments. In addition, the memory device 170 may include a decoded frame buffer (DPB) and may be configured by a digital storage medium.

[0088] The image segmentation module 110 may segment an input image (or frame or movie frame) input to the image encoding equipment 100 into one or more processing units. For example, a processing unit may be referred to as a "coding unit (CU)". The coding unit may be obtained by recursively segmenting a coding tree unit (CTU) or a largest coding unit (LCU) according to a quadtree, binary tree, and ternary tree (QT / BT / TT) structure. For example, one coding unit may be segmented into a plurality of coding units of greater depth based on a quadtree structure, a binary tree structure, and / or a ternary structure. To segment a coding unit, a quadtree structure may be applied first, and a binary tree structure and / or a ternary structure may be applied subsequently.The encoding procedure according to the present disclosure can be performed based on a final coding unit that is no longer segmented. The largest coding unit can be used as the final coding unit, or a coding unit of greater depth obtained by segmenting the largest coding unit can be used as the final coding unit. Here, the encoding procedure can include a prediction, transformation, and restoration procedure, which is described below. As another example, the processing unit of the encoding procedure can be a prediction unit (PU) or a transformation unit (TU). The prediction unit and transformation unit can be split or segmented from the final coding unit.The prediction unit may be a sample prediction unit, and the transformation unit may be a unit for extracting a transformation coefficient and / or a unit for extracting a residual signal from the transformation coefficient.

[0089] The prediction module (inter-prediction module 180 or intra-prediction module 185) can perform prediction regarding the block to be processed (the current block) and generate a predicted block including prediction samples for the current block. The prediction module can determine whether intra-prediction or inter-prediction is applied based on the current block or CU. The prediction module can generate various information related to the prediction of the current block and transmit the generated information to the entropy encoder 190. The prediction information can be encoded in the entropy encoder 190 and output in the form of a bitstream.

[0090] The intra-prediction module 185 can predict the current block by referring to samples in the current frame. The samples for reference may be located in the neighborhood of the current block or may be arranged with space according to the intra-prediction mode and / or the intra-prediction technology. The intra-prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional mode may include, for example, the DC mode and the planar mode. The directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes according to the granularity of the prediction direction. However, this is merely an example; a greater or lesser number of directional prediction modes may be used depending on the setting.The intra prediction module 185 may determine a prediction mode applied to the current block by using a prediction mode applied to a neighboring block.

[0091] The inter-prediction unit 180 may extract a predicted block for the current block based on a reference block (an array of reference samples) indicated by a motion vector for the reference frame. In this case, in order to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between the neighboring block and the current block. The motion information may include a motion vector and a reference frame index. The motion information may further include inter-prediction direction information (L0 prediction, L1 prediction, bi-prediction, etc.). In the case of inter-prediction, the neighboring block may include a spatial neighboring block present in the current frame and a temporal neighboring block present in the reference frame.A reference frame including a reference block and a reference frame including a temporal neighboring block may be identical or different. A temporal neighboring block may be referred to as a "collocated reference block," "collocated CU (colCU)," and the like. A reference frame including a temporal neighboring block may be referred to as a "collocated frame (colPic)." For example, inter-prediction module 180 may configure a list of candidate motion information options based on neighboring blocks and generate information indicating which candidate is used to extract a motion vector and / or a reference frame index of the current block. Inter-prediction may be performed based on various prediction modes. For example, in the case of the skip mode and the fusion mode, inter-prediction module 180 may use the motion information of the neighboring block as the motion information of the current block.In the skip mode, unlike the fusion mode, the residual signal may not be transmitted. In the motion vector prediction (MVP) mode, the motion vector of an adjacent block can be used as a motion vector predictor, and the motion vector of the current block can be transmitted in signaling by encoding the motion vector difference and an indicator for the motion vector predictor. The motion vector difference can represent the difference between the motion vector of the current block and the motion vector predictor.

[0092] The prediction module can generate a prediction signal based on various prediction methods and prediction technologies described below. For example, the prediction module can not only apply intra-prediction or inter-prediction, but also simultaneously apply both intra-prediction and inter-prediction to predict the current block. The prediction method that simultaneously applies both intra-prediction and inter-prediction to predict the current block can be called "combined inter- and intra-prediction (CIIP)". In addition, the prediction module can perform intra-block copying (IBC) to predict the current block. Intra-block copying can be used for coding image / video content of a game, etc., such as screen content coding (SCC).IBC is a method for predicting the current frame using a previously reconstructed reference block in the current frame at a location separated from the current block by a predetermined distance. When IBC is applied, the reference block's location in the current frame can be encoded as a vector (block vector) corresponding to the predetermined distance.

[0093] The prediction signal generated by the prediction module can be used to generate a reconstructed signal or generate a residual signal. Subtractor 115 can generate a residual signal (residual block or residual sample array) by subtracting the prediction signal (predicted block or prediction sample array) output from the prediction module from the input image signal (original block or original sample array). The generated residual signal can be transmitted to converter 120.

[0094] The converter 120 can generate transform coefficients by applying a transform technology to the residual signal. For example, the transform technology can include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally nonlinear transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is represented by a graph. CNT refers to a transform obtained based on a predicted signal generated using all previously reconstructed pixels. In addition, the transform process can be applied to square pixel blocks of identical size or can be applied to blocks of variable size rather than square.

[0095] Quantizer 130 can quantize transform coefficients and transmit them to entropy encoder 190. Entropy encoder 190 can encode a quantized signal (information regarding the quantized transform coefficients) and output a bitstream. Information regarding the quantized transform coefficients can be called "residual information." Quantizer 130 can rearrange the quantized transform coefficients in block form into a one-dimensional vector form based on the scan order of the coefficients and generate information regarding the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0096] The entropy encoder 190 may implement various encoding methods, such as, for example, exponential Golomb coding, context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), and the like. The entropy encoder 190 may encode information necessary for restoring video / images, other than quantized transform coefficients (e.g., values ​​of syntax elements, etc.), together or separately. The encoded information (e.g., encoded video / image information) may be transmitted or stored in units of network abstraction layers (NAL) in the form of a bit stream. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a frame parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS).In addition, the video / image information may further include general constraint information. The information transmitted in the service signals, the transmitted information, and / or the syntax elements described in this disclosure may be encoded using the above-described encoding procedure and included in the bitstream.

[0097] The bit stream can be transmitted over a network or can be stored on a digital storage medium. The network can include a broadcast network and / or a communication network, and the digital storage medium can include various storage media, such as USB, SD, CD, DVD, Blu-Ray, HDD, SSD, etc. A transmitter (not shown) that transmits a signal output from the entropy encoder 190 and / or a storage module (not shown) that stores the signal can be included as an internal / external element of the encoding equipment 100, and alternatively, the transmitter can be included in the entropy encoder 190.

[0098] The quantized transform coefficients output from quantizer 130 can be used to generate a residual signal. For example, the residual signal (residual block or residual samples) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients via dequantizer 140 and inverse transformator 150.

[0099] Adder 155 adds the reconstructed residual signal to the prediction signal output from inter-prediction unit 180 or intra-prediction unit 185 to generate a reconstructed signal (reconstructed frame, reconstructed block, reconstructed sample array). If there is no residual for the block to be processed, for example, in the case in which the skip mode is applied, the predicted block can be used as the reconstructed block. Adder 155 can be called a "reconstruction unit" or a "reconstructed block generator." The generated reconstructed signal can be used for intra-prediction of the next block to be processed in the current frame and can be used for inter-prediction of the next frame through filtering, as described below.

[0100] Meanwhile, as described below, luminance signal conversion with chrominance signal scaling (LMCS) is applicable in the frame encoding process.

[0101] The filter 160 can improve the subjective / objective quality of images by applying filtering to the reconstructed signal. For example, the filter 160 can generate a modified reconstructed frame by applying various filtering methods to the reconstructed frame and store the modified reconstructed frame in the memory 170, namely, in the DPB of the memory 170. The various filtering methods can include, for example, deblocking filtering, filtering based on sampled adaptive offset, an adaptive loop filter, a bilateral filter, and the like. The filter 160 can generate various information related to filtering and transmit the generated information to the entropy encoder 190, as described below in the description of each filtering method. The information related to filtering can be encoded by the entropy encoder 190 and output in the form of a bit stream.

[0102] The modified reconstructed frame transmitted to the memory device 170 can be used as a reference frame in the inter-prediction unit 180. When inter-prediction is applied via the image encoding equipment 100, the prediction mismatch between the image encoding equipment 100 and the image decoding equipment can be eliminated, and the encoding efficiency can be improved.

[0103] The DPB of the memory 170 may store the modified reconstructed frame for use as a reference frame in the inter-prediction unit 180. The memory 170 may store the motion information of the block from which the motion information in the current frame is extracted (or encoded) and / or the motion information of the blocks in the frame that have already been reconstructed. The stored motion information may be transmitted to the inter-prediction unit 180 and used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory 170 may store the reconstructed samples of the reconstructed blocks in the current frame and may transmit the reconstructed samples to the intra-prediction unit 185.

[0104] Overview of image decoding equipment

[0105] Fig. 3 is a view schematically showing image decoding equipment to which an embodiment of the present disclosure is applicable.

[0106] As shown in Fig. 3, the image decoding equipment 200 may include an entropy decoder 210, a dequantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-prediction unit 260, and an intra-prediction unit 265. The inter-prediction unit 260 and the intra-prediction unit 265 may be collectively referred to as a "prediction unit." The dequantizer 220 and the inverse transformer 230 may be included in a residual processor.

[0107] All or at least some of the plurality of components configuring the image decoding equipment 200 may be configured by a hardware component (e.g., a decoder or a processor) according to an embodiment. In addition, the memory device 250 may include a decoded frame buffer (DPB) or may be configured by a digital storage medium.

[0108] The image decoding equipment 200, which receives a bitstream including video / image information, can reconstruct an image by performing a process corresponding to the process performed by the image encoding equipment 100 in FIG. 2. For example, the image decoding equipment 200 can perform decoding using a processing unit applied in the image encoding equipment. Thus, the decoding processing unit can be, for example, a coding unit. The coding unit can be obtained by segmenting a coding tree unit or the largest coding unit. The reconstructed image signal decoded and output by the decoding equipment 200 can be reproduced by the playback equipment.

[0109] The image decoding equipment 200 may receive a signal output from the image encoding equipment of FIG. 2 in the form of a bit stream. The received signal may be decoded via the entropy decoder 210. For example, the entropy decoder 210 may parse the bit stream to extract information (e.g., video / image information) necessary for image reconstruction (or frame reconstruction). The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a frame parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The image decoding equipment may further decode a frame based on the information regarding the parameter set and / or the general constraint information.The transmitted / received information and / or syntax elements described in this disclosure may be decoded via a decoding procedure and obtained from a bitstream. For example, entropy decoder 210 decodes information in the bitstream based on a coding method, such as Exponential Golomb, CAVLC, or CABAC coding, and the output values ​​of the syntax elements required for image reconstruction and the quantized values ​​of the transform coefficients for the residual.More specifically, the entropy CABAC decoding method can receive a resolution unit corresponding to each syntax element in a bitstream, determine a context model using the information of the target syntax element, the decoding information of the neighboring block and the target decoding block, or the information of the symbol / resolution unit decoded in the previous stage, and perform arithmetic decoding for the resolution unit by predicting the probability of the occurrence of the resolution unit according to the determined context model and generate a symbol corresponding to the value of each syntax element. In this case, the entropy CABAC decoding method can update the context model by using the information of the decoded symbol / resolution unit for the context model of the next symbol / resolution unit after determining the context model.Information related to prediction from the information decoded by the entropy decoder 210 may be supplied to the prediction module (the inter-prediction module 260 and the intra-prediction module 265), and the residual value for which the entropy decoding is performed in the entropy decoder 210, i.e., the quantized transform coefficients and the related parameter information, may be input to the dequantizer 220. In addition, information regarding filtering from the information decoded by the entropy decoder 210 may be supplied to the filter 240. Meanwhile, a receiver (not shown) for receiving a signal output from the image encoding equipment may be further configured as an internal / external element of the image decoding equipment 200, or the receiver may be a component of the entropy decoder 210.

[0110] Meanwhile, the image decoding equipment according to the present disclosure may be called "video / image / frame decoding equipment." The image decoding equipment may be classified into an information decoder (video / image / frame information decoder) and a sample decoder (video / image / frame sample decoder). The information decoder may include an entropy decoder 210. The sample decoder may include at least one of a dequantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-prediction unit 260, or an intra-prediction unit 265.

[0111] The dequantizer 220 can dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 220 can rearrange the quantized transform coefficients into a two-dimensional block. In this case, the rearrangement can be performed based on the scan order of the coefficients performed in the image coding equipment. The dequantizer 220 can perform dequantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) and obtain the transform coefficients.

[0112] The inverse transformer 230 can inversely transform the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0113] The prediction module can perform prediction for the current block and can generate a predicted block including prediction samples for the current block. The prediction module can determine whether intra-prediction or inter-prediction is applied to the current block based on the prediction information output from the entropy decoder 210 and can determine a specific intra / inter-prediction mode.

[0114] Identical to what is described in the prediction module of the image coding equipment 100, the prediction module can generate a prediction signal based on various prediction methods (technologies), which are described below.

[0115] Intra-prediction module 265 can predict the current block by referring to samples in the current frame. The description of intra-prediction module 185 applies equally to intra-prediction module 265.

[0116] The inter-prediction unit 260 may extract a predicted block for the current block based on a reference block (an array of reference samples) indicated by a motion vector for the reference frame. In this case, in order to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between the neighboring block and the current block. The motion information may include a motion vector and a reference frame index. The motion information may further include inter-prediction direction information (L0 prediction, L1 prediction, bi-prediction, etc.). In the case of inter-prediction, the neighboring block may include a spatial neighboring block present in the current frame and a temporal neighboring block present in the reference frame.For example, inter-prediction module 260 may configure a list of motion information candidates based on neighboring blocks and extract the motion vector of the current block and / or the reference frame index based on the received candidate selection information. Inter-prediction may be performed based on various prediction modes, and the prediction information may include information indicating the inter-prediction mode for the current block.

[0117] Adder 235 can generate a reconstructed signal (a reconstructed frame, a reconstructed block, an array of reconstructed samples) by adding the obtained residual signal with a prediction signal (a predicted block, an array of predicted samples) output from the prediction module (including inter-prediction module 260 and / or intra-prediction module 265). The description of adder 155 is equally applicable to adder 235.

[0118] Meanwhile, as described below, luminance signal conversion with chrominance signal scaling (LMCS) is applicable in the frame decoding process.

[0119] The filter 240 can improve the subjective / objective quality of images by applying filtering to the reconstructed signal. For example, the filter 240 can generate a modified reconstructed frame by applying various filtering methods to the reconstructed frame and store the modified reconstructed frame in the memory 250, namely, in the DPB of the memory 250. The various filtering methods can include, for example, deblocking filtering, filtering based on sampled adaptive offset, an adaptive loop filter, a bilateral filter, and the like.

[0120] The (modified) reconstructed frame stored in the DPB of the memory 250 can be used as a reference frame in the inter-prediction unit 260. The memory 250 can store the motion information of the block from which the motion information in the current frame is extracted (or decoded) and / or the motion information of the blocks in the frame that have already been reconstructed. The stored motion information can be transmitted to the inter-prediction unit 260, so that it is used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory 250 can store the reconstructed samples of the reconstructed blocks in the current frame and transmit the reconstructed samples to the intra-prediction unit 265.

[0121] In the present disclosure, the embodiments described in the filter 160, the inter-prediction unit 180, and the intra-prediction unit 185 of the image coding equipment 100 can be equally or correspondingly applied to the filter 240, the inter-prediction unit 260, and the intra-prediction unit 265 of the image decoding equipment 200.

[0122] General Introduction to Mutual Forecasting

[0123] The image coding equipment / image decoding equipment can perform inter-prediction in block units to extract a prediction sample. Inter-prediction may refer to a prediction extracted in a manner that depends on the data elements of a frame(s) other than the current frame. When inter-prediction is applied to the current block, the predicted block for the current block can be extracted based on a reference block indicated by a motion vector for the reference frame.

[0124] In this case, to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information of the current block can be extracted based on the correlation of motion information between the neighboring block and the current block, and the motion information can be extracted in units of blocks, sub-blocks, or samples. The motion information may include a motion vector and a reference frame index. The motion information may further include inter-prediction type information. Here, the inter-prediction type information may refer to inter-prediction direction information. The inter-prediction type information may indicate that the current block is predicted using one of L0 prediction, L1 prediction, or bi-prediction.

[0125] When applying inter-prediction to the current block, the current block's neighboring block may include a spatial neighboring block present in the current frame and a temporal neighboring block present in the reference frame. The reference frame including the reference block for the current block and the reference frame including the temporal neighboring block may be identical or different. The temporal neighboring block may be referred to as a "co-located reference block" or "co-located CU (colCU)," and the reference frame including the temporal neighboring block may be referred to as a "co-located frame (colPic)."

[0126] Meanwhile, a list of motion information candidate variants may be constructed based on neighboring blocks relative to the current block, and in this case, flag or index information indicating which candidate variant is used may be transmitted in service signals so as to extract the motion vector of the current block and / or the index of the reference frame.

[0127] The motion information may include L0 motion information and / or L1 motion information according to the inter-prediction type. The motion vector in the L0 direction may be defined as the L0 motion vector or MVL0, and the motion vector in the L1 direction may be defined as the L1 motion vector or MVL1. Prediction based on the L0 motion vector may be defined as L0 prediction, prediction based on the L1 motion vector may be defined as L1 prediction, and prediction based on both the L0 motion vector and the L1 motion vector may be defined as biprediction. Here, the L0 motion vector may mean a motion vector associated with the L0 reference frame list, and the L1 motion vector may mean a motion vector associated with the L1 reference frame list.

[0128] The L0 reference frame list may include the frames before the current frame in the output order as reference frames, and the L1 reference frame list may include the frames after the current frame in the output order. The previous frames may be specified as forward (reference) frames, and the subsequent frames may be specified as backward (reference) frames. Meanwhile, the L0 reference frame list may additionally include the frames after the current frame in the output order as reference frames. In this case, in the L0 reference frame list, the previous frames may be indexed first, and the subsequent frames may then be indexed. The L1 reference frame list may additionally include the frames before the current frame in the output order as reference frames. In this case, in the L1 reference frame list, the subsequent frames may be indexed first, and the previous frames may then be indexed. Here, the output order may correspond to the order of the frame numbers in the sequence (POC).

[0129] Fig. 4 is a flowchart illustrating a video / image coding method based on inter prediction.

[0130] Fig. 5 is a view illustrating the configuration of the inter-prediction module 180 according to the present disclosure.

[0131] The encoding method of Fig. 6 can be performed by the image encoding equipment of Fig. 2. Specifically, step S410 can be performed by the inter-prediction unit 180, and step S420 can be performed by the residual processor. Specifically, step S420 can be performed by the subtractor 115. Step S430 can be performed by the entropy encoder 190. The prediction information of step S630 can be extracted by the inter-prediction unit 180, and the residual information of step S630 can be extracted by the residual processor. The residual information is information regarding residual samples. The residual information can include information regarding quantized transform coefficients for the residual samples.As described above, the residual samples can be extracted as transform coefficients through the converter 120 of the image encoding equipment, and the transform coefficient can be extracted as quantized transform coefficients through the quantizer 130. Information regarding the quantized transform coefficients can be encoded by the entropy encoder 190 through the residual encoding procedure.

[0132] The image coding equipment can perform inter-prediction with respect to the current block (S410). The image coding equipment can extract the inter-prediction mode and motion information of the current block and generate prediction samples of the current block. Here, the procedures of determining the inter-prediction mode, extracting motion information, and generating prediction samples can be simultaneously performed, or any of them can be performed before the other procedures. For example, as shown in FIG. 5 , the inter-prediction unit 180 of the image coding equipment 100 can include a prediction mode determination unit 181, a motion information extraction unit 182, and a prediction sample extraction unit 183.The prediction mode determining unit 181 can determine the prediction mode of the current block, the motion information extracting unit 182 can extract the motion information of the current block, and the prediction sample extracting unit 183 can extract the prediction samples of the current block. For example, the inter-prediction unit 180 of the image coding equipment can search for a block similar to the current block in a predetermined zone (search zone) of reference frames through motion estimation, and extract a reference block whose difference with the current block is equal to or less than a predetermined criterion or minimum. Based on this, the reference frame index indicating the reference frame in which the reference block is located can be extracted, and the motion vector can be extracted based on the position difference between the reference block and the current block.The image coding equipment can determine the mode applied to the current block from among various inter-prediction modes. The image coding equipment can compare the bit rate distortion (RD) cost functions for different prediction modes and determine the optimal inter-prediction mode for the current block. However, the method for determining the inter-prediction mode for the current block by the image coding equipment is not limited to the above example, and various methods can be used.

[0133] For example, the inter prediction mode of the current block may be defined as at least one of a pooling mode, a skip pooling mode, a motion vector prediction (MVP) mode, a symmetric motion vector difference (SMVD) mode, an affine mode, a sub-block pooling mode, an adaptive motion vector resolution (AMVR) mode, a history-based motion vector predictor (HMVP) mode, a pairwise average pooling mode, a pooling mode using motion vector differences (MMVD), a decoder-side motion vector granularity (DMVR) mode, a combined inter- and intra-prediction (CIIP) mode, or a geometric segmentation (GPM) mode.

[0134] For example, when the skip mode or the combining mode is applied to the current block, the image coding equipment may extract combining candidate options from neighboring blocks relative to the current block and construct a combining candidate list using the extracted combining candidate options. In addition, the image coding equipment may extract a reference block whose difference with the current block is equal to or less than a predetermined criterion or minimum from the reference blocks indicated by combining candidate options included in the combining candidate list. In this case, a combining candidate associated with the extracted reference block may be selected, and combining index information indicating the selected combining candidate may be generated and transmitted in service signals to the image decoding equipment.The motion information of the current block can be extracted using the motion information of the selected possible combination option.

[0135] As another example, when the MVP mode is applied to the current block, the image coding equipment may extract motion vector predictor (MVP) candidates from neighboring blocks relative to the current block and construct a list of MVP candidates using the extracted MVP candidates. In addition, the image coding equipment may use the motion vector of the MVP candidate selected from among the MVP candidates included in the MVP candidate list as the MVP of the current block. In this case, for example, the motion vector indicating the reference block extracted through the above-described motion estimation may be used as the motion vector of the current block, and the MVP candidate with the motion vector having the smallest difference from the motion vector of the current block among the MVP candidates may be the selected MVP candidate.The motion vector difference (MVD), which is the difference obtained by subtracting the MVP from the motion vector of the current block, can be extracted. In this case, index information indicating the selected MVP candidate and information regarding the MVD can be transmitted in signaling to the image decoding equipment. Furthermore, when using the MVP mode, the reference frame index value can be constructed as reference frame index information and separately transmitted in signaling to the image decoding equipment.

[0136] The image coding equipment can extract residual samples based on the predicted samples (S420). The image coding equipment can extract residual samples by comparing the original samples of the current block with the predicted samples. For example, a residual sample can be extracted by subtracting the corresponding predicted sample from the original sample.

[0137] The image encoding equipment may encode image information including prediction information and residual information (S430). The image encoding equipment 100 may output the encoded image information in the form of a bit stream. The prediction information may include prediction mode information (for example, a skip flag, a combining flag, or a mode index, etc.) and motion information as information related to the prediction procedure. Of the prediction mode information, the skip flag indicates whether the skip mode is applied to the current block, and the combining flag indicates whether the combining mode is applied to the current block. Alternatively, the prediction mode information may indicate one of a plurality of prediction modes, for example, a mode index. When the skip flag and the combining flag are both 0, it may be determined that the MVP mode is applied to the current block.The motion information may include candidate selection information (e.g., a combination index, MVP flag, or MVP index), which is information for extracting a motion vector. Of the candidate selection information, the combination index may be signaled when the combination mode is applied to the current block and may be information for selecting one of the possible combination options included in the list of possible combination options. Of the candidate selection information, the MVP flag or MVP index may be signaled when the MVP mode is applied to the current block and may be information for selecting one of the possible MVP options in the list of possible MVP options. In particular, the MVP flag may be signaled using the syntax element mvp_10_flag or mvp_11_flag.In addition, the motion information information may include information regarding the above-described MVD and / or reference frame index information. In addition, the motion information information may include information indicating whether L0 prediction, L1 prediction, or biprediction should be used. Residual information represents information regarding residual samples. The residual information may include information regarding quantized transform coefficients for the residual samples.

[0138] The output bit stream may be stored on a (digital) storage medium and transmitted to image decoding equipment, or may be transmitted to image decoding equipment via a network.

[0139] As described above, the image coding equipment can generate a reconstructed frame (a frame including reconstructed samples and a reconstructed block) based on reference samples and residual samples. This serves to enable the image coding equipment to extract a prediction result identical to the prediction performed by the image decoding equipment, thereby improving coding efficiency. Accordingly, the image coding equipment can store the reconstructed frame (or reconstructed samples and a reconstructed block) in a memory device and use it as a reference frame for inter-prediction. As described above, an in-loop filtering procedure can additionally be applied to the reconstructed frame.

[0140] Fig. 6 is a flowchart illustrating a video / image decoding method based on inter prediction.

[0141] Fig. 7 is a view illustrating the configuration of the inter-prediction module 260 according to the present disclosure.

[0142] The image decoding equipment can perform an operation corresponding to the operation performed by the image coding equipment. The image decoding equipment can perform a prediction regarding the current block based on the received prediction information and extract the prediction samples.

[0143] The decoding method of Fig. 6 can be performed by the image decoding equipment of Fig. 3. Steps S610-S630 can be performed by the inter-prediction unit 260, and the prediction information of step S610 and the residual information of step S640 can be obtained from the bitstream by the entropy decoder 210. The residual processor of the image decoding equipment can extract residual samples for the current block based on the residual information (S640). Specifically, the dequantizer 220 of the residual processor can perform dequantization based on the quantized transform coefficients extracted based on the residual information to extract transform coefficients, and the inverse transformer 230 of the residual processor can perform the inverse transform with respect to the transform coefficients to extract residual samples for the current block.Step S650 may be performed by means of adder 235 or recovery module.

[0144] Specifically, the image decoding equipment may determine the prediction mode of the current block based on the received prediction information (S610). The image decoding equipment may determine the inter-prediction mode to be applied to the current block based on the prediction mode information in the prediction information.

[0145] For example, it may determine whether the skip mode applies to the current block based on the skip flag. Furthermore, it may determine whether the merge mode or MVP mode applies to the current block based on the merge flag. Alternatively, one of various possible inter-prediction mode options may be selected based on the mode index. Possible inter-prediction mode options may include the skip mode, the merge mode, and / or the MVP mode, or may include various inter-prediction modes, which are described below.

[0146] The image decoding equipment may extract motion information of the current block based on a determined inter-prediction mode (S620). For example, when the skip mode or the combination mode is applied to the current block, the image decoding equipment may construct a combination candidate list described below and select one of the combination candidates included in the combination candidate list. The selection may be performed based on the combination candidate selection information (combination index) described above. The motion information of the current block may be extracted using the motion information of the selected combination candidate. For example, the motion information of the selected combination candidate may be used as the motion information of the current block.

[0147] As another example, when the MVP mode is applied to the current block, the image decoding equipment may construct a candidate MVP list and use the motion vector of the candidate MVP selected from among the candidate MVPs included in the candidate MVP list as the MVP of the current block. The selection may be performed based on the candidate selection information (MVP flag or MVP index) described above. In this case, the MVD of the current block may be extracted based on the information regarding the MVD, and the motion vector of the current block may be extracted based on the MVP and MVD of the current block. In addition, the reference frame index of the current block may be extracted based on the reference frame index information. The frame indicated by the reference frame index in the reference frame list of the current block may be extracted as the reference frame referenced for inter-prediction of the current block.

[0148] The image decoding equipment may generate predictive samples of the current block based on the motion information of the current block (S630). In this case, a reference frame may be extracted based on the reference frame index of the current block, and the predictive samples of the current block may be extracted using the samples of the reference block specified by the motion vector of the current block for the reference frame. In some cases, a filtering procedure for the predictive samples may be additionally performed on all or some of the predictive samples of the current block.

[0149] For example, as shown in Fig. 7, the inter-prediction unit 260 of the image decoding equipment may include a prediction mode determination unit 261, a motion information extraction unit 262, and a prediction sample extraction unit 263. In the inter-prediction unit 260 of the image decoding equipment, the prediction mode determination unit 261 may determine the prediction mode of the current block based on the received prediction mode information, the motion information extraction unit 262 may extract motion information (motion vector and / or reference frame index, etc.) of the current block based on the received motion information, and the prediction sample extraction unit 263 may extract prediction samples of the current block.

[0150] The image decoding equipment may generate residual samples of the current block based on the received residual information (S640). The image decoding equipment may generate reconstructed samples of the current block based on the predicted samples and residual samples and generate a reconstructed frame based on this (S650). Afterwards, the in-loop filtering procedure may be applied to the reconstructed frame as described above.

[0151] As described above, the inter-prediction procedure may include a step of determining an inter-prediction mode, a step of extracting motion information according to the determined prediction mode, and a step of performing prediction (generating prediction samples) based on the extracted motion information. The inter-prediction procedure may be performed by image coding equipment and image decoding equipment as described above.

[0152] The step of extracting motion information according to the prediction mode is described in more detail in the following document.

[0153] As described above, inter-prediction can be performed using the motion information of the current block. The image coding equipment can extract optimal motion information of the current block through a motion estimation procedure. For example, the image coding equipment can search for a similar reference block with a high correlation in a predetermined search range in a reference frame using the source block in the source frame for the current block in a sub-pixel unit and extract motion information using it. The block similarity can be calculated based on the sum of absolute differences (SAD) between the current block and the reference block. In this case, motion information can be extracted based on the reference block with the smallest SAD in the search area. The extracted motion information can be transmitted in service signals to the image decoding equipment according to various methods based on the inter-prediction mode.

[0154] When the combining mode is applied to the current block, the motion information of the current block is not directly transmitted, and the motion information of the current block is derived using the motion information of the neighboring block. Accordingly, the motion information of the current prediction block can be indicated by transmitting flag information indicating that the combining mode is used and candidate selection information (for example, a combining index) indicating which neighboring block is used as a candidate for combining. In the present disclosure, since the current block represents a unit of prediction performance, the current block can be used with a meaning identical to the meaning of the current prediction block, and the neighboring block can be used with a meaning identical to the meaning of the neighboring prediction block.

[0155] The image coding equipment can search for possible fusion block variants used to extract motion information of the current block for fusion mode. For example, up to five possible fusion block variants can be used without limitation. The maximum number of possible fusion block variants can be transmitted in the slice header or tile group header without limitation. After finding the possible fusion block variants, the image coding equipment can generate a list of possible fusion block variants and select the possible fusion block variant with the lowest RD cost as the final possible fusion block variant.

[0156] The list of possible merge options may use, for example, five possible merge block options. For example, four spatial merge options and one temporal merge option may be used.

[0157] General Introduction to Affine Mode

[0158] This paper further describes the affine mode, which is an example of an inter-prediction mode. In a traditional video encoding / decoding system, only one motion vector is used to express the motion information of the current block (a model of translational motion in space). However, in the traditional method, optimal motion information is expressed only in block units, but optimal motion information cannot be expressed in pixel units. To solve this problem, an affine motion mode is proposed, which specifies block motion information in pixel units. According to the affine mode, the motion vector for each pixel and / or sub-block unit of the block can be determined using two to four motion vectors associated with the current block.

[0159] Compared with the existing motion information expressed using spatial translation (or displacement) of a pixel value, in the affine mode, motion information for each pixel can be expressed using at least one of spatial translation, scaling, rotation, or translation.

[0160] Fig. 8 is a view illustrating a motion expressible in affine mode.

[0161] Among the motions shown in Fig. 8, the affine mode in which the motion information for each pixel is expressed using translation, scaling, or rotation may be a simplified affine mode (based on similarity). The affine mode in the description below may refer to the simplified affine mode (based on similarity).

[0162] Motion information in affine mode can be expressed using two or more control point-based motion vectors (CPMVs). The motion vector of a specific pixel position of the current block can be extracted using CPMVs. In this case, the set of motion vectors for each pixel and / or sub-block of the current block can be defined as an affine motion vector field (affine MVF).

[0163] Fig. 9 is a view illustrating a parametric model of the affine mode.

[0164] When the affine mode is applied to the current block, the affine MVF can be extracted using one of the 4-parameter model and the 6-parameter model. In this case, the 4-parameter model can refer to a model type that uses two CPMVs, and the 6-parameter model can refer to a model type that uses three CPMVs. Figs. 9(a) and 9(b) show the CPMVs used in the 4-parameter model and the 6-parameter model, respectively.

[0165] When the position of the current block is (x, y), the motion vector according to the pixel position can be extracted according to Equation 1 or 2 below. For example, the motion vector according to the 4-parameter model can be extracted according to Equation 1, and the motion vector according to the 6-parameter model can be extracted according to Equation 2.

[0166] Equation 1

[0167] Equation 2

[0168] In equations 1 and 2, mv0={mv_0x, mv_0y} can represent the CPMV at the upper-left corner position of the current block, mv1={mv_1x, mv_1y} can represent the CPMV at the upper-right position of the current block, and mv2={mv_2x, mv_2y} can represent the CPMV at the lower-left position of the current block. In this case, W and H correspond respectively to the width and height of the current block, and mv={mv_x, mv_y} can mean the motion vector of the pixel position {x, y}.

[0169] During the encoding / decoding process, the affine MVF can be determined in units of pixels and / or predetermined sub-blocks. When the affine MVF is determined in units of pixels, the motion vector can be extracted based on each pixel value. Meanwhile, when the affine MVF is determined in units of sub-blocks, the motion vector of the corresponding block can be extracted based on the central pixel value of the sub-block. The central pixel value can mean a virtual pixel present in the center of the sub-block or the lower right pixel of four pixels present in the center. In addition, the central pixel value can represent a specific pixel in the sub-block and can represent a pixel representing the sub-block. In the present disclosure, a case is further described in which the affine MVF is determined in units of 4×4 sub-blocks.However, this is for convenience of description only, and the subblock size can be changed in various ways.

[0170] Thus, when affine prediction is available, the motion model applicable to the current block may include three models: a spatial translational motion model, a 4-parameter affine motion model, and a 6-parameter affine motion model. Here, the spatial translational motion model may represent the model used by the existing motion vector in block units, the 4-parameter affine motion model may represent the model used by two CPMVs, and the 6-parameter affine motion model may represent the model used by three CPMVs. The affine mode may be divided into detailed modes according to the encoding / decoding method of motion information. For example, the affine mode may be divided into an affine MVP mode and an affine fusion mode.

[0171] When the affine combining mode is applied to the current block, the CPMV can be extracted from the neighboring blocks of the current block encoded / decoded in the affine mode. When at least one of the neighboring blocks of the current block is encoded / decoded in the affine mode, the affine combining mode can be applied to the current block. Thus, when the affine combining mode is applied to the current block, the CPMV of the current block can be extracted using the CPMV of the neighboring blocks. For example, the CPMV of the neighboring blocks can be determined as the CPMV of the current block, or the CPMV of the current block can be extracted based on the CPMV of the neighboring blocks. When the CPMV of the current block is extracted based on the CPMV of the neighboring blocks, at least one of the coding parameters of the current block or the neighboring blocks can be used.For example, the CPMVs of neighboring blocks can be modified based on the size of neighboring blocks and the size of the current block and used as the CPMV of the current block.

[0172] Meanwhile, the affine merge in which the MV is extracted in units of sub-blocks may be called the "sub-block-based merge mode," which may be specified by the merge_subblock_flag having a first value (e.g., 1). In this case, the affine merge candidate list described below may be called the "sub-block-based merge candidate list." In this case, the candidate extracted as the SbTMVP described below may additionally be included in the sub-block-based merge candidate list. In this case, the candidate extracted as the sbTMVP may be used as the candidate index #0 of the sub-block-based merge candidate list.In other words, the candidate extracted as sbTMVP may be located before the inherited affine candidate candidates and the constructed affine candidate candidates described below in the list of candidate candidates of the sub-block-based merge.

[0173] For example, an affine mode flag may be set to indicate whether or not the affine mode is applicable to the current block, which may be signaled in at least one of the upper layers of the current block, such as a sequence, a frame, a slice, a tile, a tile group, a brick, etc. For example, the affine mode flag may be named "sps_affine_enabled_flag".

[0174] When the affine combining mode is applied, the affine combining candidate list may be configured to extract the CPMV of the current block. In this case, the affine combining candidate list may include at least one of an inherited affine combining candidate, a constructed affine combining candidate, or a zero combining candidate. An inherited affine combining candidate may refer to a candidate extracted using the CPMV of neighboring blocks when neighboring blocks relative to the current block are encoded / decoded in the affine mode. A constructed affine combining candidate may refer to a candidate having each CPMV extracted based on the motion vectors of neighboring blocks of each control point (CP). Meanwhile, a zero combining candidate may refer to a candidate consisting of CPMVs having a size of 0.In the description below, CP may refer to a specific block position used to extract CPMV. For example, CP may represent each block vertex position.

[0175] Fig. 10 is a view illustrating a method for forming a list of affine possible combination options.

[0176] Referring to the flowchart of the method in Fig. 10, affine combination candidates may be added to the affine combination candidate list in the order of the inherited affine combination candidate (S1210), the constructed affine combination candidate (S1220), and the zero combination candidate (S1230). The zero combination candidate may be added when the number of candidates included in the choice list does not satisfy the maximum number of candidates, even if all of the inherited affine combination candidates and the constructed affine combination candidates are added to the affine combination candidate list. In this case, the zero combination candidate may be added until the number of candidates of the affine combination candidate list satisfies the maximum number of candidates.

[0177] Fig. 11 is a view illustrating a control point-based motion vector (CPMV) extracted from a neighboring block.

[0178] For example, a maximum of two inherited affine merge candidate options can be retrieved, each of which can be retrieved based on at least one of the left neighboring blocks and the upper neighboring blocks.

[0179] Fig. 12 is a view illustrating neighboring blocks for extracting an inherited affine merge candidate.

[0180] The inherited affine merge candidate retrieved based on the left neighboring block is retrieved based on at least one of the neighboring blocks A0 or A1 in Fig. 12, and the inherited affine merge candidate retrieved based on the upper neighboring block may be retrieved based on at least one of the neighboring blocks B0, B1 or B2 in Fig. 12. In this case, the scanning order of the neighboring blocks may be A0-A1 and B0, B1 and B2, but is not limited to this. For each of the left and upper, the inherited affine merge candidates may be retrieved based on the available first neighboring block in the scanning order. In this case, redundancy check may not be performed between the candidates retrieved from the left neighboring block and the upper neighboring block.

[0181] For example, as shown in Fig. 11, when the left neighboring block A is encoded / decoded in the affine mode, at least one of the motion vectors v2, v3, and v4 corresponding to the CP of the neighboring block A can be extracted. When the neighboring block A is encoded / decoded through the 4-parameter affine model, the inherited affine merge candidate can be extracted using v2 and v3. In contrast, when the neighboring block A is encoded / decoded through the 6-parameter affine model, the inherited affine merge candidate can be extracted using v2, v3, and v4.

[0182] Fig. 13 is a view illustrating neighboring blocks for extracting the constructed affine candidate union option.

[0183] The constructed affine candidate may mean a candidate having a CPMV extracted using a combination of common motion information of neighboring blocks. The motion information for each CP may be extracted using spatial neighboring blocks or temporal neighboring blocks relative to the current block. In the description below, CPMVk may mean a motion vector representing the k-th CP. For example, referring to FIG. 13, CPMV1 may be defined as the available first motion vector of motion vectors B2, B3, and A2, and in this case, the scanning order may be B2, B3, and A2. CPMV2 may be defined as the available first motion vector of motion vectors B1 and B0, and in this case, the scanning order may be B1 and B0. CPMV3 may be defined as representing one of motion vectors A1 and A0, and in this case, the scanning order may be A1 and A0.When TMVP is applicable to the current block, CPMV4 can be defined as a motion vector T, which represents a temporal neighboring block.

[0184] After the four motion vectors for each CP are extracted, a constructed affine combination candidate can be extracted based on this. The constructed affine combination candidate can be configured by including at least two motion vectors selected from among the extracted four motion vectors for each CP. For example, the constructed affine combination candidate can consist of at least one of {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, or {CPMV1, CPMV3}, in this order. The constructed affine candidate consisting of three motion vectors can be a candidate for the 6-parameter affine model. In contrast, the constructed affine candidate consisting of two motion vectors can represent a candidate for the 4-parameter affine model.To avoid the motion vector scaling process when the indices of the CP reference frames are different from each other, the combination of related CPMVs can be ignored without being used to extract the constructed affine candidate.

[0185] When the affine MVP mode is applied to the current block, the encoding / decoding equipment can extract two or more CPMV predictors and CPMV for the current block and extract CPMV differences based on them. In this case, the CPMV differences can be transmitted in signaling from the encoding equipment to the decoding equipment. The image decoding equipment can extract the CPMV predictor for the current block, reconstruct the CPMV difference signaled in the signaling, and then extract the CPMV of the current block based on the CPMV predictor and CPMV difference.

[0186] Meanwhile, when the affine merge mode or subblock-based TMVP is not applied to the current block (e.g., the value of the affine merge flag or merge_subblock_flag is 0), the affine MVP mode can be applied to the current block. Alternatively, when the value of inter_affine_flag is 1, the affine MVP mode can be applied to the current block. Meanwhile, the affine MVP mode can be expressed as the affine CPMVP mode. The list of affine MVP candidate variants described below can be referred to as the "list of control point-based motion vector predictor candidate variants."

[0187] When the affine MVP mode is applied to the current block, the list of affine possible MVP options may be configured to retrieve CPMV for the current block. In this case, the list of affine possible MVP options may include at least one of an inherited affine possible MVP option, a constructed affine possible MVP option, an affine possible MVP option of translation in space, or a zero possible MVP option. For example, the list of affine possible MVP options may include a maximum of n (e.g., n=2) possible options.

[0188] In this case, the inherited affine MVP candidate may refer to a candidate extracted based on the CPMVs of neighboring blocks when the neighboring blocks relative to the current block are encoded / decoded in the affine mode. The constructed affine MVP candidate may refer to a candidate extracted by forming a CPMV combination based on the motion vector of the neighboring CP block. The zero MVP candidate may refer to a candidate consisting of CPMVs having the value 0. The extraction method and characteristics of the inherited affine MVP candidate and the constructed affine MVP candidate are identical to the above-described inherited affine candidate and the constructed affine candidate, and therefore their description is omitted.

[0189] When the maximum number of possible options of the affine possible MVP option list is 2, the constructed affine possible MVP option, the affine possible MVP option of translational motion in space, and the null possible MVP option can be added when the current number of possible options is less than 2. Specifically, the affine possible MVP option of translational motion in space can be extracted in the following order.

[0190] For example, when the number of possible options included in the list of affine possible MVP options is less than 2, and CPMV0 of the constructed affine possible MVP option is valid, CPMV0 can be used as the affine possible MVP option. Thus, affine possible MVP options with all motion vectors CP0, CP1, CP2 representing CPMV0 can be added to the list of affine possible MVP options.

[0191] Then, when the number of possible variants in the affine possible MVP variant list is less than 2, and CPMV1 of the constructed affine possible MVP variant is valid, CPMV1 can be used as the affine possible MVP variant. Thus, affine possible MVP variants with all motion vectors CP0, CP1, CP2 representing CPMV1 can be added to the affine possible MVP variant list.

[0192] Then, when the number of possible variants in the affine possible MVP variant list is less than 2, and CPMV2 of the constructed affine possible MVP variant is valid, CPMV2 can be used as an affine possible MVP variant. Thus, affine possible MVP variants with all motion vectors CP0, CP1, CP2 representing CPMV2 can be added to the affine possible MVP variant list.

[0193] Notwithstanding the above conditions, when the number of possible options in the affine possible MVP list is less than 2, the current block's temporal motion vector predictor (TMVP) may be added to the affine possible MVP list. Notwithstanding the above, when the number of possible options in the affine possible MVP list is less than 2, the zeroth possible MVP may be added to the affine possible MVP list.

[0194] Fig. 14 is a view illustrating a method for forming a list of affine possible MVP options.

[0195] Referring to the flowchart of the method in Fig. 14, the candidate variants may be added to the affine MVP candidate list in the order of the inherited affine MVP candidate (S1610), the constructed affine MVP candidate (S1620), the affine MVP candidate of translation in space (S1630), and the zero MVP candidate (S1640). As described above, steps S1620-S1640 may be performed depending on whether the number of candidate variants included in the affine MVP candidate list is less than or not 2 at each step.

[0196] The scan order of inherited affine MVP candidates may be equal to the scan order of the inherited affine union candidates. However, in the case of an inherited affine MVP candidate, only adjacent blocks that reference the same reference frame as the current block's reference frame may be considered. When an inherited affine MVP candidate is added to the affine MVP candidate list, redundancy check may not be performed.

[0197] To extract the constructed affine MVP candidate, only the spatial neighboring blocks shown in Fig. 13 can be considered. In addition, the scanning order of the constructed affine MVP candidates can be equal to the scanning order of the constructed affine union candidates. In addition, to extract the constructed affine MVP candidate, the reference frame index of the neighboring block can be checked, and in the scanning order, the first neighboring block that is mutually encoded and references the same reference frame with the reference frame of the current block can be used.

[0198] General Introduction of Sub-Block-Based Temporal Motion Vector Prediction Mode (SbTMVP)

[0199] This document further describes the sub-block-based TMVP mode, which is an example of an inter-prediction mode. According to the sub-block-based TMVP mode, the motion vector field (MVF) for the current block can be extracted, and the motion vector can be extracted in sub-block units.

[0200] Unlike the traditional TMVP mode, which is implemented in units of coding units, for a coding unit to which the sub-block-based TMVP mode is applied, the motion vector can be encoded / decoded in units of sub-coding units. In addition, according to the traditional TMVP mode, the temporal motion vector can be extracted from a co-located block in a co-located frame. However, in the sub-block-based TMVP mode, the motion vector field can be extracted from a reference block in a co-located frame, indicated by a motion vector extracted from the neighboring block of the current block. Hereinafter, the motion vector extracted from the neighboring block may be referred to as the "motion offset" or "feature motion vector" of the current block.

[0201] Fig. 15 is a view illustrating adjacent blocks of the TMVP mode based on sub-blocks.

[0202] When the sub-block-based TMVP mode is applied to the current block, a neighboring block for determining the motion shift may be determined. For example, scanning for a neighboring block for determining the motion shift may be performed in the order of blocks A1, B1, B0, and A0 in Fig. 15. As another example, the neighboring block for determining the motion shift may be limited to a specific neighboring block of the current block. For example, the neighboring block for determining the motion shift may always be determined as block A1. When the neighboring block has a motion vector referring to a co-located frame, the corresponding motion vector may be determined as the motion shift. The motion vector determined as the motion shift may be called a "temporal motion vector". Meanwhile, when the above-described motion vector cannot be extracted from neighboring blocks, the motion shift may be set to (0, 0).

[0203] Fig. 16 is a view illustrating a method for extracting a motion vector field according to the TMVP mode based on sub-blocks.

[0204] Next, a reference block in the co-located frame indicated by the motion offset can be determined. For example, sub-block-based motion information (motion vector or reference frame index) can be obtained from the co-located frame by adding the motion offset to the coordinates of the current block. In the example shown in Fig. 16, it is assumed that the motion offset is the motion vector A1 of the block. By applying the motion offset to the current block, a sub-block in the co-located frame (co-located sub-block) corresponding to each sub-block configuring the current block can be indicated. After that, using the motion information of the corresponding sub-block in the co-located frame (co-located sub-block), the motion information of each sub-block of the current block can be extracted. For example, the motion information of the corresponding sub-block can be obtained from the center position of the corresponding sub-block.In this case, the central position may be the position of the lower-right sample of four samples located at the center of the corresponding sub-block. When motion information of a specific sub-block of the co-located block corresponding to the current block is not available, the motion information of the central sub-block of the co-located block may be determined as the motion information of the corresponding sub-block. When the motion vector of the corresponding sub-block is extracted, it can be switched to the reference frame index and the motion vector of the current sub-block, similar to the TMVP process described above. Thus, when the sub-block-based motion vector is extracted, motion vector scaling can be performed taking into account the POC of the reference frame of the reference block.

[0205] As described above, a sub-block-based TMVP candidate for the current block can be extracted using the motion vector field or motion information of the current block extracted based on the sub-block.

[0206] Hereinafter in this document, the list of possible merge options configured in sub-block units is defined as a list of possible merge options based on sub-block units. The above-described affine possible merge option and the sub-block-based TMVP possible merge option can be combined to configure a list of possible merge options based on sub-block units.

[0207] Meanwhile, a sub-block-based TMVP mode flag may be set, indicating whether the sub-block-based TMVP mode is applicable to the current block, which may be signaled in at least one layer of the upper layers of the current block, such as a sequence, frame, slice, tile, tile group, brick, etc. For example, the sub-block-based TMVP mode flag may be named "sps_sbtmvp_enabled_flag". When the sub-block-based TMVP mode is applicable to the current block, the sub-block-based TMVP candidate may first be added to the sub-block-based combination candidate list, and then the affine combination candidate may be added to the sub-block-based combination candidate list. Meanwhile, the maximum number of possible options that can be included in the list of possible options for combining based on sub-block units can be transmitted in service signals.For example, the maximum number of possible options that can be included in the list of possible options for a sub-block-based combination could be 5.

[0208] The sub-block size used to retrieve the list of possible sub-block-based merge options can be signaled or preset to MxN. For example, MxN can be 8x8. Accordingly, only when the current block size is 8x8 or larger is the affine mode or sub-block-based TMVP mode applicable to the current block.

[0209] Hereinafter, this document describes an embodiment of the method for performing prediction of the present disclosure. The following method for performing prediction may be performed in step S410 of Fig. 4 or in step S630 of Fig. 6.

[0210] A predicted block for the current block can be generated based on the motion information extracted according to the prediction mode. The predicted block (prediction block) may include prediction samples (an array of prediction samples) of the current block. When the motion vector of the current block specifies a fractional sample unit, an interpolation procedure can be performed, and through it, the prediction samples of the current block can be extracted based on the reference samples in units of fractional samples in the reference frame. When affine inter-prediction is applied to the current block, the prediction samples can be generated based on the MV in units of samples / sub-blocks. When bi-prediction is applied, the prediction samples extracted through the weighted sum or weighted average (according to phase) of the prediction samples extracted based on L0 prediction (i.e.prediction samples extracted based on L1 prediction (i.e., prediction using MLV1 and the reference frame in the L0 reference frame list) and prediction samples extracted based on L1 prediction (i.e., prediction using MLV1 and the reference frame in the L1 reference frame list) can be used as the prediction samples of the current block. When biprediction is applied, both the reference frame used for L0 prediction and the reference frame used for L1 prediction are located in different temporal directions relative to the current frame (i.e., if it corresponds to biprediction and bidirectional prediction), it can be called "true biprediction".

[0211] In image decoding equipment, reconstructed samples and a reconstructed frame can be generated based on the extracted prediction samples, and then the in-loop filtering procedure can be performed. In addition, in image encoding equipment, residual samples can be extracted based on the extracted prediction samples, and image information including prediction information and residual information can be encoded.

[0212] Bi-forecasting using CU, BCW level weighting coefficients

[0213] When bi-prediction is applied to the current block as described above, the prediction samples can be extracted based on a weighted average. Conventionally, the bi-prediction signal (i.e., bi-prediction samples) can be extracted through a simple average of the prediction L0 signal (prediction L0 samples) and the prediction L1 signal (prediction L1 samples). That is, the bi-prediction samples are extracted through the average of the prediction L0 samples based on the reference L0 frame and MVL0 and the prediction L1 samples based on the reference L1 frame and MVL1. However, according to the present disclosure, when bi-prediction is applied, the bi-prediction signal (bi-prediction samples) can be extracted through a weighted average of the prediction L0 signal and the prediction L1 signal as follows.

[0214] Equation 3

[0215] In the above equation 3, P bi-preddenotes the biprediction signal (biprediction block) extracted by the weighted average, and P0 and P1 denote the L0-prediction samples (L0-prediction block) and L1-prediction samples (L1-prediction block), respectively. In addition, (8-w) and w denote the weights applied to P0 and P1, respectively.

[0216] When generating a bi-predicted signal using a weighted average, five weighting coefficients can be resolved. For example, the weighting coefficient w can be selected from {-2, 3, 4, 5, 10}. For each bi-predicted CU, the weighting coefficient w can be determined in one of two ways. As the first of the two ways, when the current CU does not represent a combining mode (non-combining CU), the weighting index can be transmitted in signaling along with the motion vector difference. For example, the bitstream can include information on the weighting index after information on the motion vector difference. As the second of the two ways, when the current CU represents a combining mode (combining CU), the weighting index can be extracted from neighboring blocks based on the index of a possible combining option (combining index).

[0217] The generation of a bi-prediction signal by means of a weighted average may be limited to being applied only to a CU whose size includes 256 or more samples (component samples of luminance signals). Therefore, bi-prediction by means of a weighted average can only be performed on a CU in which the product of the width and height of the current block is 256 or more. In addition, the weighting coefficient w may be used as one of the five weighting coefficients described above, and one of a different number of weighting coefficients may be used. For example, according to the characteristics of the current image, five weighting coefficients may be used for a low-latency frame, and three weighting coefficients may be used for a frame other than a low-latency frame. In this case, the three weighting coefficients may be {3, 4, 5}.

[0218] Image coding equipment can determine the weight index without significantly increasing complexity by applying a fast search algorithm. In this case, the fast search algorithm can be generalized as follows. In the following, unequal weighting may mean that the weights applied to P0 and P1 are not equal. Alternatively, equal weighting may mean that the weights applied to P0 and P1 may be equal.

[0219] - In case the AMVR mode which adaptively changes the motion vector resolution is applied together, when the current frame is a low latency frame, only the unequal weight coefficient can be conditionally checked for each of the 1-pixel motion vector resolution and the 4-pixel motion vector resolution.

[0220] - In case the affine mode is applied together and the affine mode is selected as the optimal mode of the current block, the image coding equipment can perform affine motion estimation (ME) for each unequal weight coefficient.

[0221] - When two reference frames used for bi-prediction are equal, only the unequal weight coefficient can be conditionally checked.

[0222] - Unequal weighting may not be checked when a predefined condition is met. The predefined frame can be based on the POC distance between the current frame and the reference frame, quantization parameter (QP), temporal level, etc.

[0223] The BCW weight index may be encoded using one context-coded resolution element and one or more subsequent resolution elements after bypass encoding. The encoded resolution element of the first context indicates whether an equal weighting factor is used. When an unequal weighting factor is used, additional resolution elements may be bypass-coded and transmitted in signaling. Additional resolution elements may be transmitted in signaling to indicate which weighting factor is used.

[0224] Weighted prediction (WP) is a tool for efficiently coding images including fading. According to weighted prediction, weight parameters (weight and bias) can be signaled for each reference frame included in each of the L0 and L1 reference frame lists. Then, when motion compensation is performed, the weight(s) and bias(es) can be applied to the corresponding reference frame(s). Weighted prediction and BCW can be used for various image types. To prevent interaction between weighted prediction and BCW, the BCW weight index may not be signaled for a CU using weighted prediction. In this case, the weight can be logically inferred to be 4. Thus, an equal weight can be applied.

[0225] For a CU to which a fusion mode applies, the weight index can be inferred from neighboring blocks based on the index of the possible fusion option. This can be applied to both the general fusion mode and the inherited affine fusion mode.

[0226] In the case of the constructed affine combining mode, the affine motion information can be configured based on the motion information of a maximum of three blocks. The BCW weight index for a CU using the constructed affine combining mode can be set equal to the BCW weight index of the first CP in the combination. Therefore, the BCW may not be applied to a CU encoded in the CIIP mode. For example, the BCW weight index of a CU encoded in the CIIP mode can be set equal to a value indicating an equal weight.

[0227] Bidirectional Optical Flow (BDOF)

[0228] According to the present disclosure, BDOF can be used to refine a biprediction signal. BDOF consists of generating prediction samples by calculating detailed motion information when biprediction is applied to the current block (e.g., CU). Accordingly, the process of calculating detailed motion information by applying BDOF can be included in the above-described motion information extraction step.

[0229] For example, BDOF can be applied at the 4x4 sub-block level. Thus, BDOF can be performed in the current block in units of 4x4 sub-blocks.

[0230] BODF, for example, may be applied to a CU that satisfies at least one or all of the following conditions.

[0231] - CU is encoded in true bi-prediction mode, i.e. one of the two reference frames comes before the current frame in display order, and the other comes after the current frame in display order

[0232] - CU is not in affine mode or ATMVP union mode

[0233] - CU has more than 64 luminance signal samples

[0234] - the height and width of the CU are equal to 8 luminance signal samples or more

[0235] - BCW weight index indicates equal weighting, i.e. applying equal weighting to the L0 forecast sample and the L1 forecast sample

[0236] - Weighted Forecasting (WP) is not applied to the current CU

[0237] - CIIP mode is used for the current CU

[0238] In addition, BDOF may be applied only to the luminance signal component. However, the present disclosure is not limited to this, and BDOF may be applied to the chrominance signal component, or to both the luminance signal component and the chrominance signal component.

[0239] BDOF mode is based on the concept of optical flow. Therefore, it assumes that object motion is smooth. When applying BDOF, motion granularity (vx, vy) can be calculated for each 4x4 sub-block. Motion granularity can be calculated by minimizing the difference between the predicted L0 sample and the predicted L1 sample. Motion granularity can be used to adjust the bi-predicted sample values ​​in a 4x4 sub-block.

[0240] The process of executing BDOF is described in more detail later in this document.

[0241] First, horizontal gradients can be calculated and vertical gradients two predictive signals. In this case, k can be equal to 0 or 1. Gradients can be calculated by directly calculating the difference between two adjacent samples. For example, gradients can be calculated as follows:

[0242] Equation 4

[0243] In the above equation 4, I (k) (i, j) denotes the sample value of coordinates (i, j) of the predictive signal in the list k (k=0, 1). For example, I (0) (i, j) can denote the sample value at position (i, j) in the L0 prediction block, and I (1) (i, j) can denote the sample value at position (i, j) in the L1 prediction block. In Equation 4 above, the first shift shift1 can be determined based on the bit depth of the luminance component. For example, when the bit depth of the luminance component is bitDepth, shift1 can be determined as max(6, bitDepth-6).

[0244] As described above, after the gradients are calculated, the autocorrelation and cross-correlation S1, S2, S3, S5 and S6 between the gradients can be calculated as follows.

[0245] Equation 5 , - Where: - where Ω is a 6×6 window around a 4×4 subblock.

[0246] In the above equation 5, n a and n b can be set equal to min(1, bitDepth-11) and min(4, bitDepth-8), respectively.

[0247] The motion detail (vx, vy) can be extracted as follows using the above-described autocorrelation and cross-correlation between gradients.

[0248] Equation 6 - Where , , , And is a function of the minimum level.

[0249] In the above equation 6, n S2 can be equal to 12. Based on the extracted motion detail and gradients, the following adjustment can be performed with respect to each sample in the 4x4 sub-block.

[0250] Equation 7

[0251] In conclusion, the forecast samples pred BDOF The CU to which BDOF is applied can be calculated by adjusting the bi-predictive CU samples as follows.

[0252] Equation 8

[0253] In the above equations, n a , n b and n S2 can be equal to 3, 6, and 12, respectively. These values ​​can be chosen such that the multiplier does not exceed 15 bits in the BDOF process, and the bit widths of the intermediate parameters are maintained within 32 bits.

[0254] To extract the gradient value, I prediction samples can be formed (k) (i, j) in list k (k=0, 1) existing outside the current CU. Fig. 17 is a view illustrating a CU extended with the ability to perform BDOF.

[0255] As shown in Fig. 17, rows / columns extending around the CU boundary can be used to perform BDOF. In order to control the computational complexity of generating prediction samples outside the boundary, the prediction samples in the extended region (white area in Fig. 17) can be generated using a bilinear filter, and the prediction samples in the CU (gray area in Fig. 17) can be generated using a normal 8-tap interpolation filter based on motion compensation. The sample values ​​at the extended positions can only be used for gradient calculation. When sample values ​​and / or gradient values ​​located outside the CU boundary are required to perform the remaining steps of the BDOF process, the nearest neighboring sample values ​​and / or gradient values ​​can be complemented (repeated) and used.

[0256] When the width and / or height of a CU is greater than 16 luminance samples, the corresponding CU may be split into sub-blocks with a width and / or height equal to 16 luminance samples. The sub-block boundary may be treated identically to the CU boundary described above in the BDOF process. The maximum unit size in which the BDOF process is performed may be limited to 16×16.

[0257] For each sub-block, it can be determined whether or not to perform BDOF. Therefore, the BDOF process for each sub-block can be skipped. For example, when the SAD value between the initial L0 prediction sample and the initial L1 prediction sample is less than a predetermined threshold, the BDOF process may not be applied to the corresponding sub-block. In this case, when the width and height of the corresponding sub-block are W and H, the predetermined threshold value can be set to (8*W*(H>>1). Considering the complexity of additional SAD calculation, the SAD can be repeatedly used between the initial L0 prediction sample and the initial L1 prediction sample calculated in the DMVR process.

[0258] When BCW is available for the current block, for example, when the BCW weight index specifies an unequal weight, BDOF may not be applied. Similarly, when WP is available for the current block, for example, when luma_weight_lx_flag is equal to 1 for at least one of the two reference frames, BDOF may not be applied. In this case, luma_weight_lx_flag may represent information indicating whether WP weights for the Lx-predicted luminance component (x is equal to 0 or 1) are present in the bitstream, or information indicating whether WP is applied to the Lx-predicted luminance component. When the CU is encoded in the symmetric MVD (SMVD) mode or the CIIP mode, BDOF may not be applied.

[0259] Projection Detailing with Optical Flow (PROF)

[0260] This document further describes a method for refining a block predicted with affine motion compensation on a sub-block basis by applying optical flow. The prediction samples generated by performing affine motion compensation on a sub-block basis can be refined based on the difference extracted using the optical flow equation. The refinement of such prediction samples may be referred to as "prediction refinement using optical flow (PROF)" in this disclosure. By using PROF, mutual prediction of the degree of detail at the pixel level can be achieved without increasing the memory access bandwidth.

[0261] The affine motion model parameters can be used to extract the motion vector of each pixel in a CU. However, since pixel-based affine motion compensation prediction results in high complexity and increased memory access bandwidth, sub-block-based affine motion compensation prediction can be implemented. When sub-block-based affine motion compensation prediction is implemented, the CU can be divided into 4x4 sub-blocks, and the motion vector can be determined for each sub-block. In this case, the motion vector of each sub-block can be extracted from the CU's CPMV. Sub-block-based affine motion compensation has a trade-off between coding efficiency and complexity and memory access bandwidth.Since the motion vector is extracted in sub-block units, the complexity and bandwidth of memory access are reduced, but the prediction accuracy is reduced.

[0262] Accordingly, fine-grained motion compensation can be achieved through fine-grained detailing by applying optical flow to sub-block-based affine motion compensation prediction.

[0263] As described above, predicted luminance signal samples can be refined by summing the difference extracted using the optical flow equation after performing sub-block-based affine motion compensation. More specifically, PROF can be performed in the following four steps.

[0264] Step 1) The predicted sub-block I(i, j) is formed by performing affine motion compensation based on the sub-blocks.

[0265] Stage 2) Spatial gradients g x (i, j) and g y(i, j) of the predicted sub-block are calculated at each sample position. In this case, a 3-tap filter can be used, and the filter coefficient can be [-1, 0, 1]. For example, the spatial gradients can be calculated as follows.

[0266] Equation 9

[0267] To calculate gradients, predicted subblocks can be extended by one pixel on each side. In this case, to reduce memory bandwidth and complexity, pixels of the extended boundaries can be copied from the nearest integer pixels in the reference frame. Accordingly, additional interpolation for the padding region can be skipped.

[0268] Step 3) The detail (ΔI(i, j)) of the luminance signal prediction can be calculated using the optical flow equation. For example, the following calculation can be used:

[0269] Equation 10

[0270] In the above equation, Δv(i, j) denotes the difference between the pixel motion vector (pixel MV, v(i, j)) calculated at the sample position (i, j) and the sub-block MV for the sub-block to which the sample (i, j) belongs.

[0271] Fig. 18 is a view illustrating the relationship between Δv(i, j), v(i, j) and the sub-block motion vector.

[0272] In the example shown in Fig. 18, for example, the difference between the motion vector v(i, j) at the upper left sample position of the current sub-block and the motion vector VS B the current subblock can be represented by a thick dashed arrow, and the vector represented by a thick dashed arrow can correspond to Δv(i, j).

[0273] The affine model parameters and pixel positions from the sub-block center are unchanged. Accordingly, Δv(i, j) can only be calculated for the first sub-block and can be reused for other sub-blocks in the same CU. Given that the horizontal offset and vertical offset from the pixel position to the sub-block center are x and y, respectively, Δv(x, y) can be extracted as follows.

[0274] Equation 11 For a 4-parameter affine model: For the 6-parameter affine model:

[0275] Above, (v 0x , v 0y ), (v 1x , v 1y ) and (v 2x , v 2y ) correspond respectively to the upper left CPMV, upper right CPMV and lower left CPMV, and w and h respectively denote the width and height of the CU.

[0276] Step 4) Finally, the final prediction block I'(i, j) can be formed based on the calculated luma signal prediction granularity ΔI(i, j) and the predicted sub-block I(i, j). For example, the final prediction block I' can be formed as follows.

[0277] Equation 12

[0278] Subframe General Representation

[0279] As described above, quantization and dequantization of the luminance component and chrominance component can be performed based on the quantization parameter. In addition, a single frame to be encoded can be divided into units of multiple CTUs, slices, tiles, or bricks, and further, a frame can be divided into units of multiple subframes.

[0280] Within a frame, a subframe can be encoded or decoded regardless of whether the previous subframe is encoded or decoded relative to it. For example, different quantizations or different resolutions can be applied to multiple subframes.

[0281] Additionally, subframes can be processed as individual frames. For example, the frame to be encoded can be a projected frame or a packetized frame in a 360-degree image / video or an omnidirectional image / video.

[0282] In this embodiment, a portion of a frame may be rendered or displayed based on the viewport of a user terminal (e.g., a head-mounted display). Accordingly, to achieve low latency, at least one subframe covering the viewport of the subframes constituting one frame may be encoded or decoded preferentially or independently from the remaining subframes.

[0283] The result of encoding a subframe may be called a "subbitstream," "substream," or simply a "bitstream." Decoding equipment can decode a subframe from a subbitstream, substream, or bitstream. In this case, a high-level syntax (HLS), such as PPS, SPS, VPS, and / or DPS (decoding parameter set), can be used to encode / decode the subframe.

[0284] In the present disclosure, the high-level syntax (HLS) may include at least one of the APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, and slice header syntax. For example, APS (APS syntax) or PPS (PPS syntax) may include information / parameters typically applicable to one or more slices or frames. SPS (SPS syntax) may include information / parameters typically applicable to one or more sequences. VPS (VPS syntax) may include information / parameters typically applicable to multiple layers. DPS (DPS syntax) may include information / parameters typically applicable to the entire video. For example, a DPS may include information / parameters related to the concatenation of a coded video sequence (CVS).

[0285] Subframe Definition

[0286] A subframe can construct a rectangular area of ​​a coded frame. The subframe size can be defined differently within a frame. For all frames in a single sequence, the size and position of a specific subframe can be defined identically. Each subframe sequence can be decoded independently. The tile and slice (and CTB) can be constrained so that they do not overlap the subframe boundary. To this end, the coding equipment can perform coding so that subframes are decoded independently. For this purpose, semantic constraints in the bitstream may be required. In addition, for each frame in a single sequence, the arrangement of tiles, slices, and bricks in a subframe can be constructed differently.

[0287] Purpose of the subframe design solution

[0288] The subframe design aims to abstract or encapsulate a range, either below the frame level or above the slice or tile group level. Therefore, a VCL NAL unit of a constant motion tile set subset (MCTS) can be extracted from one VVC bitstream and rebased in another VVC bitstream without complications such as modification at the VCL level. Here, the MCTS is a coding technique for ensuring spatial and temporal independence between tiles. When an MCTS is applied, information regarding tiles that are not included in the MCTS to which the current tile belongs cannot be referenced. When an image is split into MCTSs and encoded, independent transmission and decoding of the MCTSs are possible.

[0289] This subframe design solution has the advantage of changing the viewing direction in viewport-dependent 360° mixed-resolution streaming schemes.

[0290] Subframe usage option

[0291] The use of a subframe is required in a viewport-dependent 360° layout that provides increased real-world spatial resolution for the viewport. For example, a layout in tiles covering the viewport extracted from a 6K (6144×3072) ERP (equipratical projection) frame or a cubemap projection resolution (CMP) having 4K (HEVC level 5.1) decoding performance equivalent to it is included in sections D.6.3 and D.6.4 of the OMAF and is used in the VR Industry Forum Standards Guide. This resolution is known to be suitable for a head-mounted display using a quadratic HD (2560×1440) display panel.

[0292] Encoding: Content can be encoded in two spatial resolutions, including a resolution with a cube face size of 1656×1536 and a resolution with a cube face size of 768×768. In both bitstreams, a 6×4 tile grid can be used, and MCTS can be encoded at each tile position.

[0293] Streamed MCTS selection: 12 MCTSs can be selected from the high-resolution bitstream, and an additional 12 MCTSs can be obtained from the low-resolution bitstream. Therefore, a hemisphere (180°x180°) of streamed content can be formed from the high-resolution bitstream.

[0294] Decoding using MCTS and bitstream concatenation: MCTSs of one time instance are received, which can be concatenated into an encoded frame with a resolution of 1920×4608, corresponding to HEVC level 5.1. In another embodiment, for the concatenated frame, four columns of tiles have a width value of 768, two columns of tiles have a width value of 384, and three rows of tiles have a height value of 768, thereby constructing a frame consisting of 3840×2304 luminance signal samples. Here, the units of width and height can represent the units of the number of luminance signal samples.

[0295] Transmission in subframe service signals

[0296] Subframe signaling may be performed at the SPS layer, as shown in Fig. 19. Fig. 19 shows the syntax for signaling a syntax element for subframes in the SPS. Hereinafter, the syntax elements of Fig. 19 are described.

[0297] The pic_width_max_in_luma_samples syntax element can specify the maximum width of each decoded frame referencing the SPS in units of luminance samples. The value of pic_width_max_in_luma_samples is greater than 0 and can be an integer multiple of andMinCbSizeY. Here, MinCbSizeY is a variable specifying the minimum coding block size of the luminance component.

[0298] The pic_height_max_in_luma_samples syntax element can specify the maximum height of each decoded frame referencing the SPS in units of luma samples; pic_height_max_in_luma_samples is greater than 0 and can have a value that is an integer multiple of MinCbSizeY.

[0299] The subpic_grid_col_width_minus1 syntax element can be used to specify the width of each subpicture ID grid element. For example, subpic_grid_col_width_minus1 can specify the width of each subpicture ID grid element in units of 4 samples, and the value obtained by adding 1 to subpic_grid_col_width_minus1 can specify the width of an individual subpicture ID grid element in units of 4 samples. The length of the syntax element can be the length in bits in Ceil(Log2(pic_width_max_in_luma_samples / 4)).

[0300] Therefore, the NumSubPicGridCols variable, which indicates the number of columns in the sub-picture grid, can be retrieved as follows.

[0301] NumSubPicGridCols=(pic_width_max_in_luma_samples+subpic_grid_col_width_minus1*4+3) / (subpic_grid_col_width_minus1*4+4)

[0302] The subpic_grid_row_height_minus1 syntax element can be used to specify the height of each subpicture ID grid element. For example, subpic_grid_row_height_minus1 can specify the height of each subpicture ID grid element in units of 4 samples. The value obtained by adding 1 to subpic_grid_row_height_minus1 can specify the height of an individual subpicture ID grid element in units of 4 samples. The length of the syntax element can be the length in bits in Ceil(Log2(pic_height_max_in_luma_samples / 4)).

[0303] Therefore, the NumSubPicGridRows variable, which indicates the number of rows in the sub-picture grid, can be retrieved as follows.

[0304] NumSubPicGridRows=(pic_height_max_in_luma_samples+subpic_grid_row_height_minus1*4+3) / (subpic_grid_row_height_minus1*4+4)

[0305] The syntax element subpic_grid_idx[i][j] can specify the subpicture index of the grid position (i, j). The length of the syntax element can be Ceil(Log2(max_subpics_minus1+1)) bits.

[0306] The variable SubPicTop[subpic_grid_idx[i][j]], SubPicLeft[subpic_grid_idx[i][j]], SubPicWidth[subpic_grid_idx[i][j]], SubPicHeight[subpic_grid_idx[i][j]] and NumSubPics can be extracted as specified in the algorithm of Fig. 20.

[0307] The syntax element subpic_treated_as_pic_flag[i] can specify whether a subframe is treated as identical to a normal frame in a decoding process other than the in-loop filtering process. For example, the first value (e.g., 0) of subpic_treated_as_pic_flag[i] can specify that the i-th subframe of each encoded frame in CVS is not treated as a frame in a decoding process other than the in-loop filtering process. The second value (e.g., 1) of subpic_treated_as_pic_flag[i] can specify that the i-th subframe of each encoded frame in CVS is treated as a frame in a decoding process other than the in-loop filtering process. When the value of subpic_treated_as_pic_flag[i] is not obtained from the bitstream, the value of subpic_treated_as_pic_flag[i] may be extracted as the first value (e.g. 0).

[0308] The syntax element loop_filter_across_subpic_enabled_flag[i] can specify whether or not in-loop filtering is performed across the boundary of the i-th subframe belonging to a single CVS frame being coded. For example, the first value (e.g., 0) of loop_filter_across_subpic_enabled_flag[i] can specify that in-loop filtering is not performed across the boundary of the i-th subframe belonging to a single CVS frame being coded. The second value (e.g., 1) of loop_filter_across_subpic_enabled_flag[i] can specify that in-loop filtering can be performed across the boundary of the i-th subframe belonging to a single CVS frame being coded. When the value of loop_filter_across_subpic_enabled_flag[i] is not obtained from the bitstream, the value of loop_filter_across_subpic_enabled_flag[i] can be extracted as the second value.

[0309] Meanwhile, for bitstream conformity, the following restrictions may apply. For any two subframes subpicA and subpicB, when the index of subpicA is less than the index of subpicB, all coded NAL units of subpicA may have a lower decoding order than all coded NAL units of subpicB. Alternatively, after decoding, the shape of the subframes must have an ideal left boundary and an ideal upper boundary, constituting the frame boundary or the boundary of a previously decoded subframe.

[0310] General Introduction of Subframe-Based Coding and Decoding

[0311] The following disclosure relates to encoding / decoding the above-described frame and / or subframe. The encoding equipment may encode the current frame based on the subframe structure. Alternatively, the encoding equipment may encode at least one subframe constituting the current frame and output a (sub)bitstream including (encoded) information regarding at least one (encoded) subframe.

[0312] The decoding equipment can decode at least a subframe belonging to the current frame based on a (sub)bit stream including (coded) information regarding at least one subframe.

[0313] Fig. 21 is a view illustrating a method for encoding an image using a subframe by an encoding equipment according to an embodiment. The encoding equipment may divide an input frame into a plurality of subframes (S2110). The encoding equipment may encode at least one subframe using information regarding the subframe (S2110). For example, each subframe may be independently divided and encoded using information regarding the subframe. Then, the encoding equipment may output a bitstream by encoding image information including information regarding the subframe (S2130). Here, the bitstream for a subframe may be referred to as a "substream" or a "subbitstream."

[0314] Hereinafter, information regarding a subframe is described in various ways in the present disclosure, and, for example, information regarding whether or not in-loop filtering can be performed across the boundary of a subframe, information regarding a zone of a subframe, information regarding a grid spacing for using a subframe, etc. may be provided.

[0315] Fig. 22 is a view illustrating a method for decoding an image using a subframe by a decoding equipment according to an embodiment. The decoding equipment may obtain information regarding the subframe from a bitstream (S2210). Then, the decoding equipment may extract at least one subframe (S2220) and decode at least one subframe (S2230).

[0316] Thus, the decoding equipment can decode at least one subframe and thereby output at least one decoded subframe or a current frame including at least one subframe. The bitstream may include a substream or a sub-bitstream for a subframe.

[0317] As described above, subframe information can be constructed in the HLS bitstream. Decoding equipment can extract at least one subframe based on the subframe information. The decoding equipment can decode the subframe based on a CABAC method, a prediction method, a residual processing method (transformation, quantization), an in-loop filtering method, etc.

[0318] When a decoded subframe is output, the decoded subframes can be output together in the form of an OPS (output subframe set). For example, when the current frame is related to an omnidirectional (360°) image / video and is partially rendered, only some subframes can be decoded, and some or all of the decoded subframes can be rendered according to the user's viewport.

[0319] When the information indicating whether in-loop filtering across a subframe boundary is enabled, the decoding equipment may perform in-loop filtering (e.g., deblocking filtering) at the subframe boundary located between two subframes. However, when the subframe boundary is equal to the frame boundary, the in-loop filtering process may not be applied to the subframe boundary.

[0320] This disclosure relates to subframe-based encoding / decoding. This document further describes BDOF and PROF, to which an embodiment of this disclosure is applicable.

[0321] As described above, by applying BDOF in the inter-prediction process to refine the reference sample during motion compensation, image compression performance can be improved. BDOF can be performed in normal mode. However, BDOF is not performed in affine mode, GPM mode, or CIIP mode.

[0322] Fig. 23 is a view illustrating the process of extracting a prediction sample of the current block by applying BDOF.

[0323] The BDOF-based inter-prediction procedure of Fig. 23 can be performed by image coding equipment and image decoding equipment.

[0324] First, in step S2310, the motion information of the current block can be extracted. The motion information of the current block can be extracted by various methods described in the present disclosure. For example, the motion information of the current block can be extracted by the regular merging mode, the MMVD mode, or the AMVP mode. The motion information can include biprediction motion information (L0 motion information and L1 motion information). For example, the L0 motion information can include MVL0 (L0 motion vector) and refIdxL0 (L0 reference frame index), as well as L1 motion information MVL1 (L1 motion vector) and refIdxL1 (L1 reference frame index).

[0325] After this, the predicted sample of the current block can be extracted based on the extracted motion information of the current block (S2320). Specifically, the predicted L0 samples for the current block can be extracted based on the L0 motion information. In addition, the predicted L1 samples for the current block can be extracted based on the L1 motion information.

[0326] After this, the BDOF offset can be extracted based on the extracted prediction samples (S2330). The BDOF of step S2330 can be performed according to the method described in this disclosure. For example, the BDOF offset can be extracted based on the gradients (according to the phase) of the L0 prediction samples and the gradients (according to the phase) of the L1 prediction samples.

[0327] After that, based on LX (X=0 or 1) prediction samples and the BDOF offset, the detailed prediction samples of the current block can be extracted (S2340). The detailed prediction samples can be used to form the final prediction block of the current block.

[0328] The image encoding equipment may extract residual samples by comparing with original samples based on the predicted samples of the current block generated according to the method of Fig. 23. Information (residual information) regarding the residual samples may be included and encoded in the image / video information and output in the form of a bitstream, as described above. In addition, the image decoding equipment may generate a reconstructed current block based on the predicted samples of the current block generated according to the method of Fig. 23 and the residual samples obtained based on the residual information in the bitstream, as described above.

[0329] Fig. 24 is a view illustrating the input and output of a BDOF process according to an embodiment of the present disclosure.

[0330] As shown in Fig. 24, the input of the BDOF process may include the width nCbW of the current block, the height CbH), the prediction sub-blocks predSamplesL0 and predSamplesL1 with the boundary zone extended by a predetermined length (for example, 2), the indices predFlagL0 and predFlagL1 of the prediction directions, and the indices refIdxL0 and refIdxL1 of the reference frames. In addition, the input of the BDOF process may further include the BDOF usage flag bdofUtilizationFlag. In this case, the BDOF usage flag may be input in units of sub-blocks in the current block to indicate whether or not the BDOF is applied to the corresponding sub-block.

[0331] In addition, the BDOF process can generate detailed pbSamples prediction block by applying BDOF based on the input information.

[0332] Fig. 25 is a view illustrating variables used for a BDOF process according to an embodiment of the present disclosure. Fig. 25 may represent a process after Fig. 24.

[0333] As shown in Fig. 25, in order to perform the BDOF process, the input bit depth bitDepth of the current block can be set to BitDepth Y. In this case, BitDepth Ycan be extracted based on the bit depth information signaled through the bit stream. In addition, various right shifts can be specified based on the bit depth. For example, the first shift shift1, the second shift shift2, the third shift shift3, and the fourth shift shift4 can be extracted based on the bit depth, as shown in Fig. 24. In addition, the offset offst4 can be specified based on shift4. In addition, the variable mvRefineThres for specifying the motion detail cutoff range can be specified based on the bit depth. The use of the various variables described in Fig. 24 is described below.

[0334] Fig. 26 is a view illustrating a method for generating a predictive sample for each sub-block in the current CU based on whether or not to apply BDOF, according to an embodiment of the present disclosure. Fig. 26 may be a process after Fig. 25.

[0335] The process shown in Fig. 26 may be performed for each sub-block in the current CU, and in this case, the sub-block size may be 4×4. When the BDOF usage flag bdofUtilizationFlag for the current sub-block is the first value (false, "0"), BDOF may not be applied to the current sub-block. In this case, the predicted sample of the current sub-block is retrieved by the weighted sum of the predicted L0 sample and the predicted L1 sample, and in this case, the weight applied to the predicted L0 sample and the weight applied to the predicted L1 sample may be identical. shift4 and offset4 used in equation (1) in Fig. 26 may be the set of values ​​in Fig. 17. When the BDOF-utilization flag bdofUtilizationFlag for the current subblock is the second value (true, "1"), BDOF can be applied to the current subblock.In this case, the predictive sample of the current sub-block may be generated by means of the BDOF process in accordance with the present disclosure.

[0336] Fig. 27 is a view illustrating a method for extracting a gradient, autocorrelation, and cross-correlation of a current sub-block according to an embodiment of the present disclosure. Fig. 27 may be a process after Fig. 26.

[0337] The process shown in Fig. 27 is performed for each sub-block in the current CU, and in this case, the sub-block size may be 4×4.

[0338] Referring to Fig. 27, according to Equation (1) and Equation (2), the position (hx, hy) for each sample position (x, y) in the current sub-block can be extracted. After that, the horizontal gradient and the vertical gradient for each sample position can be extracted according to Equations (3)-(6). After that, the variables (the first intermediate parameter diff and the second intermediate parameters tempH and tempV) for extracting the autocorrelation and cross-correlation can be extracted according to Equations (7)-(9). For example, the first intermediate parameter diff can be extracted using the value obtained by applying a right shift to the prediction samples predSamplesL0 and predSamplesL1 of the current block through the second shift shift2. For example, the second intermediate parameters tempH and tempV can be extracted by applying a right shift to the sum of the gradient in the L0 direction and the gradient in the L1 direction via a third shift shift3, as given in equation (8) and equation (9).After that, the autocorrelation and cross-correlation can be extracted based on the extracted first intermediate parameter and the second intermediate parameters according to equations (10)-(16).

[0339] Fig. 28 is a view illustrating a method for extracting motion detail (vx, vy), extracting BDOF offset, and generating a prediction sample of the current sub-block, according to an embodiment of the present disclosure. Fig. 28 may be a process after Fig. 27.

[0340] The process shown in Fig. 28 is performed for each sub-block in the current CU, and in this case, the sub-block size may be 4×4.

[0341] According to Equation 28, the motion granularity (vx, vy) can be extracted according to Equation (1) and Equation (2). The motion granularity can be clipped within the range specified by mvRefineThres. In addition, based on the motion granularity and gradient, the BDOF offset bdofOffset can be extracted according to Equation (3). The prediction sample pbSamples of the current sub-block can be generated using the extracted BDOF offset according to Equation (4).

[0342] By continuously implementing the methods described with reference to Figs. 24-28, the BDOF process according to the first embodiment of the present disclosure can be realized. In the embodiment according to Figs. 24-28, the first shift shift1 is set to Max(6, bitDepth-6), and mvRefineThres is set to 1< <Max(bitDepth-7). Соответственно, битовые ширины predSample и каждый параметр BDOF по данным BitDepth могут извлекаться так, как показано в следующей таблице.

[0343] Table 1 BitDepth beforeSample Shift1 Gradient in, you bdofOffset 8 16 [-25022, 24958] 6 11 [-779, 779] 6 [-32, 31] 17 [-49856, 48298] 10 16 6 11 6 17 12 16 6 11 6 17 14 18 8 11 8 19 16 20 10 11 10 21

[0344] As described above, by applying BDOF in the inter-prediction process to refine the reference sample during motion compensation, image compression performance can be improved. BDOF can be performed in normal mode. However, BDOF is not performed in affine mode, GPM mode, or CIIP mode.

[0345] PROF can be performed on a block encoded in affine mode as a method similar to BDOF. As described above, by fine-tuning the reference sample in each 4x4 sub-block via PROF, image compression performance can be improved.

[0346] According to the present disclosure, the above-described affine motion information (sub-block-based motion) of the current block can be extracted, and the affine motion information can be refined through the above-described PROF process, or a predictive sample extracted based on the affine motion information can be refined.

[0347] Fig. 29 is a view illustrating the process of extracting a prediction sample of the current block by applying PROF.

[0348] The PROF-based inter-prediction procedure of Fig. 29 can be performed by image coding equipment and image decoding equipment.

[0349] First, in step S2910, motion information of the current block can be extracted. The motion information of the current block can be extracted using various methods described in the present disclosure. For example, the motion information of the current block can be extracted using the method described in the above-described affine mode or the TMVP mode based on sub-blocks. The motion information may include motion information of sub-blocks relative to the current block. The motion information of the sub-blocks may include motion information of bi-prediction sub-blocks (L0 motion information of sub-blocks and L1 motion information of sub-blocks). For example, the L0 motion information of the sub-blocks may include sbMVL0 (sub-block L0 motion vector) and refIdxL0 (reference L0 frame index), and the L1 motion information of the sub-blocks may include sbMVL1 (sub-block L1 motion vector) and refIdxL1 (reference L1 frame index).

[0350] After this, the predicted sample of the current block can be extracted based on the extracted motion information of the current block (S2920). Specifically, the predicted L0 samples for each sub-block of the current block can be extracted based on the L0 motion information of the sub-blocks. In addition, the predicted L1 samples for each sub-block of the current block can be extracted based on the L1 motion information of the sub-blocks.

[0351] After this, the PROF bias can be extracted based on the extracted prediction samples (S2930). The PROF of step S2930 can be performed according to the method described in this disclosure. For example, the motion difference vector diffMv and the gradients LX (X=0 or 1) of the prediction samples can be calculated, and based on them, the PROF bias dI or ΔI can be extracted according to the method described in this disclosure. Various examples of the present disclosure relate to the extraction of motion difference vectors, the extraction of gradients, and / or the extraction of PROF biases.

[0352] After this, based on the LX (X=0 or 1) prediction samples and the PROF bias, the fine-grained prediction samples of the current block can be extracted (S2940). The fine-grained prediction samples can be used to form the final prediction block of the current block. For example, the final prediction block of the current block can be formed by weighted summation of the fine-grained L0 prediction samples and the fine-grained L1 prediction samples.

[0353] The image encoding equipment may extract residual samples by comparing with original samples based on the predicted samples of the current block generated according to the method of Fig. 29. Information (residual information) regarding the residual samples may be included and encoded in the image / video information and output in the form of a bitstream, as described above. In addition, the image decoding equipment may generate a reconstructed current block based on the predicted samples of the current block generated according to the method of Fig. 29 and the residual samples obtained based on the residual information in the bitstream, as described above.

[0354] Fig. 30 is a view illustrating an example of a PROF process according to the present disclosure.

[0355] According to the example of Fig. 30, the PROF process can be executed using the width sbWidth, the height sbHeight of the current sub-block, the prediction sub-block predSamples in which the border area is extended by a predetermined length borderExtention, and the motion difference vector diffMv as input. In this case, the prediction sub-block, for example, can be a prediction sub-block generated by performing affine motion compensation. As a result of executing the PROF process, a detailed prediction sub-block pbSamples can be generated.

[0356] To perform the PROF process, the predefined first shift

shift1

BitDepth

[0357] After that, the horizontal gradient gradientH, g x and vertical gradient gradientV, g y can be calculated for each sample position (x, y) of the input prediction sub-block. The horizontal gradient and the vertical gradient can be calculated according to equation (1) and equation (2) in Fig. 30, respectively.

[0358] After that, based on the horizontal gradient, the vertical gradient, and the motion difference vector diffMv, the PROF offset dI or ΔI can be calculated for each sample position. For example, the PROF offset can be calculated according to equation (3) in Fig. 30. In equation (3), the motion difference vector diffMv used to calculate the PROF offset can mean Δv described with reference to Fig. 18. In this case, diffMv can be cut off by dmvLimit as follows, and dmvLimit can be calculated based on BitDepth Y as follows.

[0359] Equation 13

[0360] After that, the detailed prediction sub-unit pbSamples can be extracted based on the calculated PROF bias and the prediction sub-unit predSamples. For example, the detailed prediction sub-unit can be extracted according to Equation (4) in Fig. 30.

[0361] According to the example in Fig. 30, the first shift shift1 can be set to Max(6, bitDepth-6), and dmvLimit can be set to 1< <Max(5, bitDepth-7). Помимо этого, diffMv может отсекаться в диапазоне [-dmvLimit, dmvLimit-1]. Соответственно, битовые ширины predSample и каждый параметр PROF по данным BitDepth Y can be extracted as shown in the following table.

[0362] Table 2 BitDepthY beforeSample Shift1 Gradient diffMv dI 8 16 [-25022, 24958] 6 11 [-779, 779] 6 [-32, 31] 17 [-49856, 48298] 10 16 6 11 6 17 12 16 6 11 6 17 14 18 8 11 8 19 16 20 10 11 10 21

[0363] The present disclosure can provide various embodiments of the case in which a subframe is treated as a frame (e.g., subpic_treated_as_pic_flag==1) when performing subframe-based encoding / decoding. For example, in the BDOF or PROF process, the process of sampling reference samples can be restricted such that it does not refer to a reference sample included in a subframe different from the subframe to which the current block belongs. In addition, in the process of extracting temporal predictors of motion vectors, the process of bilinear interpolation of a luminance sample, the process of 8-tap interpolation filtering of the process of interpolating luminance samples and / or chrominance samples, a restriction can be added according to the subframe treated as a frame.

[0364] As described above, in the BDOF and / or PROF process, a reference sample extended by a predetermined length around the current block boundary can be used to calculate the gradient. However, when the current subframe to which the current block belongs is treated as a frame, the range of the reference sample for gradient calculation must be limited to the same subframe as the current subframe.

[0365] As described above, the prediction sample can be modified by bdofOffset in the BDOF case and by dI in the PROF case; bdofOffset and dI can be obtained based on the gradient. The gradient can be extracted based on the difference between the reference samples in the reference frame. To extract the gradient, a reference sample sampling process for sampling a reference sample from the reference frame and an 8-tap interpolation filtering process can be performed. The output of the reference sample sampling process can be a sample of luminance signals at an integer pixel position.

[0366] Fig. 31 is a view illustrating a case in which a reference sample to be sampled crosses a subframe boundary.

[0367] In Fig. 31, an area (sampling area) in a reference frame to be sampled may be specified according to a motion vector mv of a current block included in the current subframe [1] in the current frame. In this case, the sampling area may exist across the boundary of the subframe [1] in the reference frame. Thus, the reference sample to be sampled may be included in a subframe different from the current subframe [1].

[0368] Fig. 32 is an enlarged view of the sampling region of Fig. 31.

[0369] As shown in Fig. 32, the reference sample to be sampled may be included in a subframe (subframe [0, subframe [2], subframe [3]) different from the current subframe [1]. Considering the case shown in Fig. 32 where the subframe is treated as a frame, the range of the reference sample to be sampled must be limited in the same subframe as the current subframe.

[0370] Fig. 33 is a view illustrating a process of sampling reference samples according to an embodiment of the present disclosure.

[0371] In the process of sampling the reference samples in Fig. 33, information regarding the motion vector of the current block and the reference frame refPicLX can be input LThe information regarding the current block's motion vector must indicate the sampling region and may be an integer sampling position (xIntL, yIntL) extracted from the current block's motion vector.

[0372] The output of the reference sample sampling process in Fig. 33 may be a prediction block for the current block. In this case, the prediction block may be a prediction block that is to be refined by BDOF or PROF.

[0373] As shown in Fig. 33, it can be determined whether the current subframe is treated as a frame or not. For example, when subpic_treated_as_pic_flag is equal to 1, it can be determined that the current subframe is treated as a frame, and the sampling region can be limited to the same subframe as the current subframe.

[0374] When the current subframe is treated as a frame as shown in equation (1) in Fig. 33, the xInt X coordinate indicating the position of the sample to be sampled may be clipped in the range of [subpicLeftBoundaryPos, SubPicRightBoundaryPos]. SubPicLeftBoundaryPos may indicate the position of the left boundary of the current subframe. In addition, SubPicRightBoundaryPos may indicate the position of the right boundary of the current subframe. According to the above equation (1), since the X coordinate of the reference sample to be sampled exists in the range from SubPicLeftBoundaryPos to SubPicRightBoundaryPos, the reference sample located beyond the left or right boundary of the subframe is not sampled. A method for extracting SubPicLeftBoundaryPos and SubPicRightBoundaryPos is described below.

[0375] Similarly, when the current subframe is treated as a frame, as shown in equation (2) of Fig. 33, the yInt Y coordinate indicating the position of the sample to be sampled may be truncated in the range of [SubPicTopBoundaryPos, SubPicBotBoundaryPos]. SubPicTopBoundaryPos may indicate the position of the upper boundary of the current subframe. In addition, SubPicBotBoundaryPos may indicate the position of the lower boundary of the current subframe. According to the above equation (2), the Y coordinate of the reference sample to be sampled exists in the range from SubPicTopBoundaryPos to SubPicBotBoundaryPos, and the reference sample existing beyond the upper boundary or the lower boundary of the subframe is not sampled. The following describes how to extract SubPicTopBoundaryPos and SubPicBotBoundaryPos.

[0376] When the current subframe is not treated as a frame, for example, when subpic_treated_as_pic_flag is 0, the coordinates of the reference sample to be sampled can be extracted according to equation (3) and equation (4). According to the above equation (3) and equation (4), the coordinates of the reference sample to be sampled are not clipped by the boundary position of the subframe. According to the above equation (3) and equation (4), the coordinates of the reference sample to be sampled can be clipped within the range of the current frame.

[0377] After that, according to the equation (5), sampling of the reference sample from the reference frame can be performed based on the coordinates (xInt, yInt) of the sample to be sampled.

[0378] Fig. 34 is a flow chart of a method illustrating a process for sampling reference samples according to the present disclosure.

[0379] The process of sampling the reference samples in Fig. 33 and 34 may be performed by the image coding equipment and the image decoding equipment to perform BDOF and / or PROF.

[0380] Referring to Fig. 34, first, it can be determined whether the current subpic is treated as a frame or not (S3410). The determination of step S3410 can be performed based on subpic_treated_as_pic_flag.

[0381] When the current subframe is treated as a frame (for example, subpic_treated_as_pic_flag==1), a sampling position may be extracted (S3420), and the extracted sampling position may be trimmed (S3430). Extraction and trimming of the sampling position may be performed according to equation (1) and equation (2) in Fig. 33. Trimming of step S3430 may be a process of changing the corresponding sampling position to a position in the current subframe (for example, a boundary position of the current subframe) when the extracted sampling position is outside the boundary of the current subframe.

[0382] When the current subpic is not treated as a frame (for example, subpic_treated_as_pic_flag==0), the sampling position may be extracted (S3440). Extraction of the sampling position may be performed, for example, according to equation (3) and equation (4) in Fig. 33.

[0383] After this, sampling of reference samples may be performed based on the sampling position extracted in step S3430 or in step S3440 (S3450). Sampling of reference samples may be performed, for example, according to equation (5) in Fig. 33.

[0384] According to the embodiment shown in Fig. 34, when a subframe is treated as a frame, a reference sample outside the boundary of the current subframe may not be sampled in the process of sampling reference samples. Therefore, a reference sample in a reference frame belonging to a subframe different from the current subframe may not be referenced.

[0385] Hereinafter, this document describes a fractional sample interpolation procedure according to another embodiment of the present disclosure.

[0386] As described above, when the motion vector of the current block specifies a fractional sampling unit, the interpolation procedure can be performed, and the prediction samples of the current block can be extracted based on the reference samples of the fractional sampling unit in the reference frame, respectively.

[0387] Fig. 35 is a view illustrating a portion of a fractional sample interpolation procedure according to the present disclosure.

[0388] To perform the fractional sample interpolation procedure, the variables fRefWidth and fRefHeight can be extracted. As shown in Fig. 35, fRefWidth and fRefHeight can be extracted differently according to whether the current subpic is treated as a frame (subpic_treated_as_pic_flag).

[0389] When the current subpic is treated as a frame, such as when subpic_treated_as_pic_flag is 1, fRefWidth and fRefHeight can be extracted as follows.

[0390] fRefWidth=(SubPicWidth[SubPicIdx]*(subpic_grid_col_width_minus1+1)*4)

[0391] fRefHeight=(SubPicHeight[SubPicIdx]*(subpic_grid_row_height_minus1+1)*4)

[0392] Above, SubPicWidth[SubPicIdx] and SubPicHeight[SubPicIdx] can respectively refer to the width and height of the current sub-picture. In this case, the sub-picture width and height can be expressed in terms of the number of grids configuring the sub-picture. For example, a sub-picture width of 4 can mean that the corresponding sub-picture includes four grids in the horizontal direction. In addition, subpic_grid_col_width_minus1 and subpic_grid_row_height_minus1 can respectively refer to the width and height of the grid configuring the sub-picture. In this case, the grid width and height can be expressed in units of 4 pixels. For example, a grid width of 4 can mean that the grid width is 16 (4×4) pixels.

[0393] When the current subpic is not treated as a frame, such as when subpic_treated_as_pic_flag is 0, fRefWidth and fRefHeight can be extracted as the width PicOutputWidthL and height PicOutputHeightL of the reference frame output image, respectively.

[0394] fRefWidth and fRefHeight extracted as described above can be used to extract the hori_scale_fp and vert_scale_fp scaling coefficients according to Equation (1) and Equation (2) in Fig. 35. After that, according to Equations (3) to (6), the position (refxL, refyL) of the reference sample indicated by the motion vector can be extracted. Based on the extracted position of the reference sample, the integer position (xIntL, yIntL) and the fractional position (xFracL, yFracL) can be extracted. The above-described process of sampling the reference samples or the 8-tap interpolation filtering process can be performed based on the extracted integer position and / or fractional position.

[0395] Hereinafter, this document describes a method for sbTMVP extraction according to another embodiment of the present disclosure.

[0396] As described with reference to Fig. 16, a motion offset can be extracted and applied to the current block, thereby specifying a sub-block (co-located sub-block) in a co-located frame corresponding to each sub-block configuring the current block. Then, using the motion information of the corresponding sub-block (co-located sub-block) of the co-located frame, the motion information of each sub-block of the current block can be extracted. Based on the extracted motion information of the sub-block, sbTMVP can be extracted.

[0397] Fig. 36 is a view illustrating a part of the sbTMVP extraction method according to the present disclosure.

[0398] Fig. 36 shows a portion of the process after the motion offset for the current block is retrieved during the sbTMVP retrieval process. Specifically, the process shown in Fig. 36 includes a process of retrieving the position of the corresponding co-located sub-block for each sub-block configuring the current block.

[0399] According to Fig. 36, firstly, according to Equation (1) and Equation (2), the position (xSb, ySb) of the lower right center sample of the current sub-block can be extracted. After that, the position (xColSb, yColSb) of the co-located sub-block can be extracted according to whether or not the current sub-picture is treated as a frame (e.g., subpic_treated_as_pic_flag).

[0400] Specifically, when subpic_treated_as_pic_flag is 1, the Y coordinate yColSb of the co-located sub-block can be extracted according to Equation (3). In this case, yColSb extracted by ySb and the motion offset tempMv can be truncated to the position specified by SubPicTopBoundaryPos and SubPicBotBoundaryPos and the Y coordinate yCtb of the current CTB. According to Equation (3), the Y coordinate of the co-located sub-block exists in the current CTB and the current subframe. In addition, when subpic_treated_as_pic_flag is 0, according to Equation (4), the Y coordinate yColSb of the co-located sub-block can be extracted. In this case, yColSb, extracted by ySb and the motion offset tempMv, can be clipped to the position specified by the yCtb Y coordinate of the current CTB and the height of the current frame. By equation (4), the Y coordinate of the co-located subblock exists in the current CTB and the current frame.

[0401] Similarly, when subpic_treated_as_pic_flag is 1, according to Equation (5), the xColSb X coordinate of the co-located sub-block can be extracted. In this case, xColSb extracted by xSb and the motion shift tempMv can be clipped to the position specified by SubPicLeftBoundaryPos and SubPicRightBoundaryPos and the xCtb X coordinate of the current CTB. According to Equation (5), the x coordinate of the co-located sub-block exists in the current CTB and the current subframe. In addition, when subpic_treated_as_pic_flag is 0, according to Equation (6), the yColSb X coordinate of the co-located sub-block can be extracted. In this case, xColSb, extracted by xSb and the tempMv motion offset, can be truncated to the position specified by the xCtb X coordinate of the current CTB and the width of the current frame. By equation (6), the X coordinate of the co-located subblock exists in the current CTB and the current frame.

[0402] According to the present disclosure, when the current subframe is treated as a frame, the co-located subblock of the current subblock for extracting sbTMVP exists in the same subframe with the current subframe.

[0403] Hereinafter, this document describes a method for extracting boundary positions of subframes according to another embodiment of the present disclosure.

[0404] Fig. 37 is a view illustrating a method for extracting a boundary position of a subframe according to the present disclosure.

[0405] According to the present disclosure, the left boundary position SubPicLeftBoundaryPos of a subframe, the right boundary position SubPicRightBoundaryPos of a subframe, the top boundary position SubPicTopBoundaryPos of a subframe, and the bottom boundary position SubPicBotBoundaryPos of a subframe can be extracted. In the present disclosure, SubPicIdx can represent an index for identifying each of the subframes in the current frame.

[0406] When the current subpicture indicated by SubPicIdx is treated as a frame, SubPicLeftBoundaryPos can be extracted based on the SubPicLeft position information of a predetermined unit indicating the left position of the current subpicture and the subpic_grid_col_width_minus1 width information of the corresponding unit. The predetermined unit can be a grid. However, the present disclosure is not limited to this, and for example, the predetermined unit can be a CTU. When the predetermined unit is a CTU, the left position of the current subpicture can be extracted as the product of the position information of the CTU unit indicating the left position of the current subpicture and the CTU size.

[0407] When the current subframe is treated as a frame, SubPicRightBoundaryPos can be extracted based on the position information of a predetermined unit indicating the left position of the current subframe, the width information of a predetermined unit indicating the width of the current subframe, and the width information of the corresponding unit. For example, the position information of a predetermined unit indicating the right position of the current subframe can be extracted by adding the left position of the current subframe and the width of the current subframe. SubPicRightBoundaryPos can be extracted by performing the "-1" operation on the final calculated value. The predetermined unit can be a grid. However, the present disclosure is not limited to this, and, for example, the predetermined unit can be a CTU.When the predetermined unit is a CTU, the right position of the current subframe can be extracted by performing the "-1" operation to product the CTU unit position information indicating the right position of the current subframe and the CTU size. The CTU unit position information indicating the right position of the current subframe can be extracted as the sum of the CTU unit position information indicating the left position of the current subframe and the CTU unit width information indicating the width of the current subframe.

[0408] Similarly, SubPicTopBoundaryPos can be extracted based on the SubPicTop position information of a predetermined unit indicating the top position of the current subframe and the subpic_grid_low_height_minus1 height information of the corresponding unit. The predetermined unit can be a grid. However, the present disclosure is not limited to this, and for example, the predetermined unit can be a CTU. When the predetermined unit is a CTU, the top position of the current subframe can be extracted as the product of the CTU position information indicating the top position of the current subframe and the CTU size.

[0409] When the current subframe is treated as a frame, SubPicBotBoundaryPos can be extracted based on the position information of a predetermined unit indicating the upper position of the current subframe, the height information of the predetermined unit indicating the height of the current subframe, and the height of the corresponding unit. For example, the position information of a predetermined unit indicating the lower position of the current subframe can be extracted by adding the upper position of the current subframe and the height of the current subframe. SubPicBotBoundaryPos can be extracted by performing the "-1" operation on the final calculated value. The predetermined unit can be a grid. However, the present disclosure is not limited to this, and, for example, the predetermined unit can be a CTU.When the predetermined unit is a CTU, the lower position of the current subframe can be extracted by performing the "-1" operation to product the CTU unit position information indicating the lower position of the current subframe and the CTU size. The CTU unit position information indicating the lower position of the current subframe can be extracted as the sum of the CTU unit position information indicating the upper position of the current subframe and the CTU unit height information indicating the height of the current subframe.

[0410] The various embodiments described in this disclosure may be implemented separately or in combination with other embodiments. Alternatively, part of an embodiment may be added to another embodiment, or part of an embodiment may be replaced by part of another embodiment.

[0411] Although the exemplary methods of the present disclosure described above are presented as a sequence of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and the steps may be performed simultaneously or in a different order, if necessary. To implement the method according to the present disclosure, the described steps may additionally include other steps, may include the remaining steps excluding some steps, or may include other additional steps excluding some steps.

[0412] In the present disclosure, the image encoding equipment or image decoding equipment that performs a predetermined operation (step) may perform an operation (step) of confirming a condition or situation of executing the corresponding operation (step). For example, if it is described that the predetermined operation is executed when the predetermined condition is satisfied, the image encoding equipment or image decoding equipment may perform the predetermined operation after determining whether the predetermined condition is satisfied.

[0413] The various embodiments of the present disclosure do not represent a list of all possible combinations and are intended to describe characteristic aspects of the present disclosure, and the issues described in the various embodiments may be applied independently or in combination of two or more of them.

[0414] Various embodiments of the present disclosure may be implemented in hardware, firmware, software, or a combination thereof. If the present disclosure is implemented in hardware, the present disclosure may be implemented using application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general processors, controllers, microcontrollers, microprocessors, etc.

[0415] In addition, the image decoding equipment and the image encoding equipment to which the embodiments of the present disclosure are applied can be included in a multimedia broadcast transceiver, a mobile communication terminal, a video device of a home theater system, a video device of a digital theater system, a surveillance camera, a video chat device, a real-time communication device such as a video communication, a mobile streaming device, a storage medium, a recording video camera, a video-on-demand (VoD) service device, an OTT (video over network) video device, an Internet streaming service device, a 3D video device, a video device of a video telephony system, a medical video device, and the like, and can be used to process video signals or data signals.For example, OTT video devices may include a gaming console, Blu-ray player, internet-enabled TV, home theater system, smartphone, tablet PC, digital video recorder (DVR), etc.

[0416] Fig. 38 is a view showing a content streaming system to which an embodiment of the present disclosure is applicable.

[0417] Referring to Fig. 38, the content streaming system to which the embodiment(s) of the present document are applied may mainly include an encoding server, a streaming server, a web server, a media storage, a user device, and a media input device.

[0418] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, camcorders, etc. into digital data to form a bitstream and transmits the bitstream to the streaming server. As another example, when multimedia input devices such as smartphones, cameras, camcorders, etc. directly form a bitstream, the encoding server may be omitted.

[0419] The bit stream may be generated by an image coding method or image coding equipment to which an embodiment of the present disclosure is applied, and the streaming server may temporarily store the bit stream in the process of transmitting or receiving the bit stream.

[0420] The streaming server transmits media data to the user device based on the user's request via the web server, and the web server serves as a medium for informing the user about the service. When the user requests the desired service from the web server, the web server can deliver it to the streaming server, and the streaming server can transmit the media data to the user. In this case, the content streaming system may include a separate control server. In this case, the control server serves to manage the command / response between devices in the content streaming system.

[0421] A streaming server may receive content from a media storage and / or encoding server. For example, when receiving content from an encoding server, the content may be received in real time. In this case, to provide a smooth streaming service, the streaming server may store the bitstream for a predetermined period of time.

[0422] Examples of the user device may include a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate tablet PC, tablet PCs, ultrabooks, wearable devices (such as smart watches, smart glasses, head-mounted displays), digital TVs, desktop computers, digital signage systems, and the like.

[0423] Each server in the content streaming system can work as a distributed server, in which case the data received from each server can be distributed.

[0424] The scope of the disclosure of the entity includes software or computer-executable instructions (for example, an operating system, an application, firmware, a program, etc.) for enabling the execution of operations according to the methods of various embodiments in equipment or a computer, a non-transitory computer-readable medium having such software or instructions stored and executed in the equipment or computer. Industrial applicability

[0425] Embodiments of the present disclosure may be used to encode or decode an image.

Claims

1. A method for decoding images, performed using image decoding equipment, wherein the method for decoding images comprises: determining whether bidirectional optical flow (BDOF) or prediction refinement with optical flow (PROF) is applied to the current sub-block; generating a prediction sample of the current sub-block from a reference frame of the current sub-block based on motion information of the current sub-block; and deriving a refined prediction sample for the current sub-block by applying BDOF or PROF to the current sub-block based on the generated prediction sample, wherein the generation of the prediction sample of the current sub-block is performed based on whether the current sub-frame is processed as a frame, wherein the generation of the prediction sample of the current sub-block is performed based on the position within the reference image for generating the prediction sample, in this case, the position within the reference frame is cropped within a predetermined range, wherein, based on the current subframe being processed as a frame, the predetermined range is specified by the position of the boundary of the current subframe, in this case, the position within the reference frame includes the x-coordinate and the y-coordinate, in this case, the x-coordinate is cut off in the range from the left position of the border to the right position of the border of the current subframe, in this case, the y coordinate is cut off in the range from the upper position of the border to the lower position of the border of the current subframe, wherein the left position of the boundary of the current subframe is output as the product of the information on the position of a predetermined unit indicating the left position of the current subframe and the width of the predetermined unit, wherein the right position of the boundary of the current subframe is output by performing the operation "-1" with respect to the product of the position information of the predetermined unit indicating the right position of the current subframe and the width of the predetermined unit, wherein the upper position of the boundary of the current subframe is output as the product of the information on the position of a predetermined unit indicating the upper position of the current subframe and the height of the predetermined unit, and wherein the lower position of the boundary of the current subframe is output by performing the operation "-1" with respect to the product of the position information of the predetermined unit indicating the lower position of the current subframe and the height of the predetermined unit.

2. The method according to claim 1, characterized in that whether the current subframe is processed as a frame is determined on the basis of flag information transmitted via a bit stream.

3. The method according to claim 2, characterized in that the flag information is transmitted via a sequence parameter set (SPS).

4. The method according to claim 1, characterized in that the predetermined unit is a grid or CTU.

5. The method according to claim 1, characterized in that, based on the current subframe, which is not processed as a frame, the predetermined range is the range of the current frame containing the current subblock.

6. A method of encoding images, performed using image encoding equipment, wherein the method of encoding images comprises: determining whether bidirectional optical flow (BDOF) or prediction refinement with optical flow (PROF) is applied to the current sub-block; generating a prediction sample of the current sub-block from a reference frame of the current sub-block based on motion information of the current sub-block; and deriving a refined prediction sample for the current sub-block by applying BDOF or PROF to the current sub-block based on the generated prediction sample, wherein the generation of the prediction sample of the current sub-block is performed based on whether the current sub-frame is processed as a frame, wherein the generation of the prediction sample of the current sub-block is performed based on the position within the reference image for generating the prediction sample, in this case, the position within the reference frame is cropped within a predetermined range, wherein, based on the current subframe being processed as a frame, the predetermined range is specified by the position of the boundary of the current subframe, in this case, the position within the reference frame includes the x-coordinate and the y-coordinate, in this case, the x-coordinate is cut off in the range from the left position of the border to the right position of the border of the current subframe, in this case, the y coordinate is cut off in the range from the upper position of the border to the lower position of the border of the current subframe, wherein the left position of the boundary of the current subframe is output as the product of the information on the position of a predetermined unit indicating the left position of the current subframe and the width of the predetermined unit, wherein the right position of the boundary of the current subframe is output by performing the operation "-1" with respect to the product of the position information of the predetermined unit indicating the right position of the current subframe and the width of the predetermined unit, wherein the upper position of the boundary of the current subframe is output as the product of the information on the position of a predetermined unit indicating the upper position of the current subframe and the height of the predetermined unit, and wherein the lower position of the boundary of the current subframe is output by performing the operation "-1" with respect to the product of the position information of the predetermined unit indicating the lower position of the current subframe and the height of the predetermined unit.

7. A non-volatile computer-readable data carrier storing a bit stream and computer programs, wherein the computer programs, when executed by a computer, cause the computer to perform the image encoding method according to claim 6 with the generation of a bit stream.

Images