Optical flow-based inter-prediction correction method, device, and recording medium

Abstract

An image encoding / decoding method, device, and recording medium according to the present disclosure may comprise the steps of: configuring a reference sample on the basis of inter-prediction mode information about the current block; calculating a pixel gradient value of the reference sample on the basis of the current picture including the current block and a distance value between the current block and a reference picture; calculating a motion vector correction value of the current block based on the pixel gradient value; and generating a prediction block of the current block on the basis of the motion vector correction value and the inter- prediction mode information.

Description

[1] The present disclosure may relate to a method, an apparatus and a recording medium for optical flow-based inter prediction correction. [Background Art] [2] Due to the vast video data volume, various technologies for reducing the data size through compression for the efficiency of storage and transmission are being studied. Various technologies for analyzing statistical characteristics for a redundant signal and using corresponding characteristics are applied to video compression technologies. In addition, effective methods are being invented for the transmission of information necessary to reconstruct a signal removed during an encoding process in a decoder. [Disclosure] [Technical Problem] [3] The present disclosure relates to a method for deriving the correction value of inter prediction information based on an optical flow during inter prediction and correcting inter prediction information or a prediction value through a derived correction value, and may propose an effective inter prediction method. [Technical Solution] [4] An image encoding / decoding method, apparatus and recording medium of the present disclosure may configure a reference sample based on neighboring regions of a current block and an inter prediction reference block, calculate a pixel gradient correction value based on the pixel gradient value, and correct a motion vector of a boundary region of the current block based on the motion vector correction value. [5] An image encoding / decoding method, apparatus and recording medium of the present disclosure may include configuring a reference sample based on inter prediction mode information of a current block; calculating a pixel gradient value of the reference sample based a distance value between a current picture including the current block and a reference picture; calculating a motion vector correction value of the current block based on the pixel gradient value; and generating a prediction block of the current block based on the motion vector correction value and the inter prediction mode information. [6] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, the reference sample includes a first reference block determined by a motion vector for a first reference picture of the current block and a second reference block determined by a motion vector for a second reference picture of the current block, and directions of the first reference picture and the second reference picture may be different from each other. [7] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, the reference sample includes a first reference block determined by a motion vector for a first reference picture of the current block and a second reference block determined by a motion vector for a second reference picture of the current block, and directions of the first reference picture and the second reference picture may be equal to each other. [8] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, the reference sample is composed of a reference block, and a motion vector determining the reference block may be determined based on pixel information in a fractional pixel unit for the reference block. [9] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, the pixel information in the fractional pixel unit for the reference block may be obtained by applying interpolation filtering to the reference block.

[10] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, a size of a reference region used for the interpolation filtering may be determined based on a size of the current block and the number of taps of an interpolation filter.

[11] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, the size of the determined reference region may be limited to a specific size or less.

[12] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, the distance value between the current block and the reference picture may be calculated as a difference value between a display order of the current picture and a display order of the reference picture.

[13] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, the pixel gradient value may include a horizontal gradient value and a vertical gradient value.

[14] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, the pixel gradient value may be calculated by using an edge detection filter.

[15] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, a filter size and a filter coefficient of the edge detection filter may be determined based on a size of the current block and the inter prediction mode information.

[16] In the image encoding / decoding method, apparatus and recording medium of the present disclosure, in response to the edge detection filter being applied to a pixel located at an edge of the reference sample, a pixel at a position outside a reference sample region of the reference sample may be filled through a padding operation. [Technical Effect]

[17] Through the present disclosure, effective prediction between inter predictions of a signal may be performed. [Brief Description of Drawings]

[18] FIG. 1 illustrates a block diagram of a decoder.

[19] FIG. 2 illustrates the process of generating a prediction block through an optical flow-based inter prediction correction method.

[20] FIGS. 3 and 4 illustrate a distance value between a current picture and a reference picture.

[21] FIG. 5 illustrates filter coefficients of a horizontal filter and a vertical filter.

[22] FIG. 6 illustrates an embodiment of subsampling.

[23] FIG. 7 illustrates the process of generating a prediction block by correcting inter prediction information for one of the two reference pictures through an optical flow-based inter prediction correction method.

[24] FIG. 8 illustrates the process of correcting the motion vector of inter prediction information through an optical flow based inter prediction correction method.

[25] FIG. 9 illustrates an embodiment of reference sample group configuration.

[26] FIG. 10 illustrates the neighboring line of a current block.

[27] FIG. 11 illustrates the boundary region of a current block.

[28] FIG. 12 illustrates an example in which the motion vector of the boundary region of a current block is corrected.

[29] FIG. 13 illustrates a method for generating an inter prediction block by applying an optical flow-based inter prediction correction method when a current block is in a mode that performs inter prediction by using a temporal interpolation block and one reference block.

[30] FIG. 14 illustrates a method for generating an inter prediction block by applying an optical flow-based inter prediction correction method when a current block is in a mode that performs inter prediction by using a geometric partitioning mode. [Mode for Invention]

[31] As the present invention may make various changes and have multiple embodiments, specific embodiments are illustrated in a drawing and are described in detail in a detailed description. But, it is not to limit the present invention to a specific embodiment, and should be understood as including all changes, equivalents and substitutes included in the idea and technical scope of the present invention. In describing each drawing, similar reference numerals were used for similar components.

[32] A term such as first, second, etc. may be used to describe a variety of components, but the components should not be limited by the terms. The terms are used only to distinguish one component from other components. For example, without departing from the scope of a right of the present invention, the first component may be referred to as the second component and likewise, the second component may be also referred to as the first element. A term of and / or includes a combination of a plurality of relevant described items or any of a plurality of relevant described items.

[33] When a component is referred to as being "linked" or connected" to other component, it should be understood that it may be directly linked or connected to other component, but other component may exist in the middle. On the other hand, when a component is referred to as being "directly linked" or "directly connected" to other component, it should be understood that other component does not exist in the middle.

[34] As a term used in the present specification is just used to describe a specific embodiment, it is not intended to limit the present invention. Expression of the singular includes expression of the plural unless it clearly has a different meaning contextually. In this application, it should be understood that a term such as "include" or "have", etc. is to designate the existence of features, numbers, steps, motions, components, parts or combinations thereof described on the specification and is not to exclude the possibility of addition or existence of one or more other features, numbers, steps, motions, components, parts or combinations thereof in advance.

[35] Hereinafter, the desirable embodiment of the present invention will be described in more detail with reference to the attached drawings. Hereinafter, the same reference numerals are used for the same components on the drawings, and an overlapping description for the same components is omitted.

[36]

[37] Hereinafter, terms “frame” and “picture” have the same meaning, and may be used interchangeably. Hereinafter, terms “optical flow” and “optical flow” have the same meaning, and may be used interchangeably. In the present invention, the order of each process in each block diagram may be changed or omitted.

[38]

[39] FIG. 1 illustrates a block diagram of a decoder.

[40] FIG. 1 is a block diagram in which the prediction and inverse transform units of a decoder are determined and a prediction and an inverse transform are performed for a determined unit, mode and method to finally perform reconstruction, as an embodiment of the present invention. The prediction unit determination unit is a step for determining the size and shape of a block for performing a prediction, and may include all methods according to direct or indirect information transmitted from an encoder. It may include not only a method for the direct size and shape information of a current decoding block, but also a method for utilizing information that may affect a determination on the size information of a current decoding block such as the number of partitions, a depth, the shape of partitioning, the direction of partitioning, size information for the minimum partitioning block, the partitioning information of a pre-decoded neighboring block, a prediction mode, etc. The prediction technology determination unit is a step for determining a prediction technology for each block. The prediction technology may refer to an intra prediction technology, an inter prediction technology, an intra block copy (IBC) technology, a palette mode technology, a technology in which intra and inter prediction methods are mixed, etc. The prediction mode determination unit is a step for determining a prediction mode according to a technology determined in the prediction technology determination unit, and is a step for determining a method for determining the pixel value of an actual prediction block such as an intra prediction direction, the number of reference pictures in inter prediction, an affine mode, a merge mode, etc. According to an embodiment, the prediction technology determination unit and the prediction mode determination unit may be integrated into one step and operate in the form of the prediction technology and mode determination unit. The prediction performance unit may generate the first prediction block or blocks through the determined prediction technology and prediction mode.

[41] The inverse transform unit determination unit determines an inverse transform unit in which an inverse transform will be performed on a residual signal. The inverse transform unit may refer to a unit that determines whether to perform an inverse transform on a residual signal and transmits information for an inverse transform. The inverse transform kernel determination unit determines a kernel used for an inverse transform. In this case, at least one inverse transform kernel may be determined for one inverse transform unit. The inverse transform performance unit may perform an inverse transform on a residual signal through the determined inverse transform unit and kernel. A residual signal generated in the inverse transform performance unit may be added to a prediction signal generated in the prediction performance unit to produce a reconstructed signal. According to an embodiment, an inverse transform may be performed for the Nth order, and the size of an inverse transform unit and the size of an inverse transform kernel may not be the same. In other words, an inverse transform may be performed only on some of the residual signals or transform signals input to the inverse transform performance unit.

[42] According to an embodiment, the process of FIG. 1 may be performed independently for a luma block and a chroma block, respectively, and when the color format of an input video is a YUV format (YUV420, YUV411, YUV422, YUV444, etc.), it may be performed for a chroma block after being performed for a luma block. According to an embodiment, when the color format of an input video is RGB, encoding may be performed after performing a color conversion to YUV.

[43] Hereinafter, the contents of the present invention are described in detail for FIG. 1.

[44] Prediction Unit Determination Unit

[45] According to an embodiment, in an encoder / a decoder, a prediction unit is determined in the prediction unit determination unit, and according to an embodiment, a prediction unit may be the value of a current block, one of the sub-blocks into which a current block is partitioned, a set of pixels or one pixel.

[46] A prediction unit may include size information and shape information for performing a prediction on a chroma component and a luma component. A prediction unit may be determined dependently or independently for a chroma component and a luma component. Dependent means that the prediction units of luma components or chroma components are not determined independently for each component, but when the prediction unit of one component is determined, the prediction units of another component or other components are determined to have a corresponding size and shape. In this case, one component may correspond to one or more of a luma component and a chroma component. In other words, it may include a case in which a luma component is determined through the information of chroma components. In another embodiment, the prediction unit of a chroma component may have a size corresponding to the prediction unit of a luma component according to the color format of an input video or a converted color format. When determined dependently, information for the prediction unit of another component corresponding to one component, i.e., a component determined dependently, may be omitted. When determined independently, it means that the prediction units of a luma component and a chroma component are determined separately, and when determined independently, information for a prediction unit for each component may be transmitted separately.

[47] Prediction Technology Determination Unit

[48] According to an embodiment, the prediction technology of each prediction unit may be determined in the prediction technology determination unit. A prediction technology may be one of inter prediction, intra prediction, an IBC technology, a palette mode technology, a technology in which intra and inter prediction methods are mixed, etc.

[49] According to an embodiment, when a current block is not an intra prediction technology, a 1-bit flag is signaled / parsed, and when a corresponding flag is a skip, it may be determined as an inter prediction merge mode or an IBC prediction merge mode, and in this case, a prediction signal may be used as a reconstructed signal by omitting an inverse transform process. Here, a skip may refer to a case in which motion information (a motion vector, a reference picture index, a reference picture list, etc.) is not transmitted or a case in which motion information is transmitted through only one syntax information, or in addition thereto, a case in which the difference signal of a current block is not transmitted.

[50] According to an embodiment, when a current block is not a skip, a 1-bit flag for the prediction technology of a current block may be signaled / parsed to determine one of inter prediction, intra prediction, an IBC technology, a palette mode technology, a technology in which intra and inter prediction methods are mixed, etc.

[51] Prediction Mode Determination Unit

[52] According to an embodiment, the determination of a prediction mode is performed according to the determined prediction technology.

[53] According to an embodiment, when the prediction technology of a current prediction unit block is intra prediction, the that generates the prediction signal of a current prediction unit block by using at least one of a directional prediction mode, a planar mode (Horizontal Planar, Vertical Planar or Regular Planar), a DC mode, a smooth mode, a recursive prediction mode or a prediction mode based on an inter-component correlation (e.g., chroma from luma (CfL), a cross component linear model (CCLM), multi-hypothesis cross component prediction (MHCCP), a convolutional cross component model (CCCM), etc.), etc.

[54] According to an embodiment, when the prediction technology of a current prediction unit block is intra prediction and is predicted in a directional mode, a current prediction unit may be predicted by using reference pixel information in a prediction direction which is corrected by using additional delta angle information. The delta angle information may be used by being signaled / parsed from an encoder.

[55] According to an embodiment, when the prediction technology of a current prediction unit block is intra prediction, intra prediction mode information of a prediction block existing around a current prediction block may be used to predict an intra prediction mode in a current prediction unit. When the intra prediction mode of a current prediction block is predicted from a neighboring prediction block and is used, an intra prediction mode candidate list may be constructed by combining the intra prediction mode of a neighboring prediction block, and the intra prediction mode of a current prediction block may be determined in a candidate list constructed by signaling / parsing index information for a candidate list. According to an embodiment, delta angle information may be determined by being derived together with an intra prediction mode in an intra prediction mode candidate list. According to an embodiment, an intra prediction mode candidate list may have one or more sets, and index information of a candidate list set for determining a set may be additionally signaled / parsed /

[56] According to an embodiment, when the prediction technology of a current prediction unit block is intra prediction and is predicted in a directional mode, a prediction block may be generated by weighted summing a reference pixel existing in both directions based on the prediction direction of a current directional mode. In this case, a weight used for the weighted sum may be determined according to the size of a current prediction unit, the prediction mode of a current prediction unit, the position of a predicted pixel, etc. According to an embodiment, when a current prediction unit is predicted in a DC mode, a prediction block may be generated by weighted summing a prediction block generated in a DC mode and a reference pixel existing on the top and left. In this case, a weight used for the weighted sum may be determined according to the size of a current prediction unit, the position of a predicted pixel, etc.

[57] According to an embodiment, when the prediction technology of a current prediction unit block is intra prediction, it may be an intra geometric partitioning-based prediction mode that partitions a current prediction unit block into one or more sub regions through geometric partitioning, generates a prediction signal by using a prediction mode within an intra prediction technology including a different directional prediction mode, a planar mode, a DC mode, etc. in each region and generates the prediction signal of a current prediction unit block through the weighted sum of each prediction signal.

[58] According to an embodiment, when the prediction technology of a current prediction unit block is intra prediction, it may be a matrix-based intra prediction mode that performs a prediction by signaling / parsing the index of a matrix by using a matrix pre-defined by a promise between an encoder and a decoder or by signaling / parsing a matrix.

[59] According to an embodiment, when the prediction technology of a current prediction unit block is intra prediction, it may be an intra template matching prediction mode that defines a pre-reconstructed region around a current prediction unit block as a template, performs template matching in the pre reconstructed neighboring region of a current prediction unit block and generates a prediction signal.

[60] According to an embodiment, when the prediction technology of a current prediction unit block is intra prediction, a pre reconstructed region around a current prediction unit block may be defined as a template and a corresponding template may be used to derive an intra prediction mode. Subsequently, the final prediction signal of a current prediction unit block may be generated by using a derived mode. In this case, a template may also include a region that is not adjacent to a current prediction unit block.

[61] According to an embodiment, when a current prediction unit block is a chroma block and the prediction technology of a current prediction unit block is intra prediction, a prediction may be performed through a direct mode (DM). For example, a DM mode may be a method for performing the prediction of a current chroma block by using the same prediction method as the prediction method of a luma block at the corresponding position of a current chroma block.

[62] According to an embodiment, when a current prediction unit block is a chroma block and the prediction technology of a current prediction unit block is intra prediction, a relationship between the neighboring pre-reconstructed chroma samples of a current chroma component and the neighboring pre reconstructed luma samples of a luma block at the corresponding position of a current chroma prediction unit block may be modeled as a linear or / and nonlinear model to generate the prediction signal of a current chroma prediction unit block.

[63] When the prediction technology of a current prediction unit is inter prediction, the prediction mode of a current prediction unit block may use motion information or several pieces of motion information for a prediction unit to perform motion compensation and determine a predicted signal or signals through motion compensation. And, it may be a mode that generates the prediction signal of a current prediction unit block through the weighted sum of multiple prediction signals including a prediction signal generated through one or more motion compensations. In this case, one or more of the prediction signals may be the signal of the pre-reconstructed region of the same frame as a current block.

[64] According to an embodiment, when the prediction technology of a current prediction unit block is inter prediction, a current prediction block may construct an inter prediction information list, signal / parse an index for a list and perform inter prediction by using inter prediction information corresponding to a corresponding index in a list as a predictor. The inter prediction information list may be composed of inter prediction information of a region which is spatially adjacent to a current block in a current picture or / and inter prediction information of a region that is adjacent to the position of a current block in a reference picture or / and inter prediction information of a pre-decoded region before a current block. The inter prediction information may include information such as a motion vector, a reference picture, etc. Inter prediction may be performed by signaling / parsing a motion vector predictor and a motion vector difference derived through an index for the inter prediction information list of a current prediction block. The motion vector difference may be signaled / parsed as an index for a table for motion vector differences calculated in an encoder or motion vector differences calculated in an encoder and approximated to one of the pre-promised values between an encoder and a decoder or motion vector differences consisting of pre-promised values between an encoder and a decoder. In this case, when a current prediction unit block performs inter prediction by using two or more motion information, only one motion vector difference may be signaled / parsed and the remaining motion vector differences may be derived by being scaled from a parsed motion vector difference.

[65] According to an embodiment, when a current block is not a skip and a prediction technology is determined as an inter prediction or IBC technology, it may be determined whether to perform a prediction on a current block in a merge mode or in an advanced motion vector prediction (AMVP) mode by signaling / parsing a 1-bit flag. The AMVP mode may refer to a mode that transmits each element within motion information, respectively, signals / parses a motion vector difference in addition to motion information, and performs motion compensation by using corrected motion information during a subsequent prediction process.

[66] According to an embodiment, when the prediction technology of a current prediction unit block is inter prediction, a current prediction unit may have a shape and a size determined through geometric partitioning. In this case, a prediction signal may be a geometric partitioning-based prediction mode that generates the prediction signal of a current prediction unit block through the weighted sum of prediction signals for each geometric partitioning unit.

[67] According to an embodiment, when the prediction technology of a current prediction unit uses a technology in which intra and inter prediction methods are mixed, the neighboring pre reconstructed region of a current prediction unit block may be defined as a template and an intra prediction mode may be derived by using the information of some or all pixels of a corresponding template. In this case, a template may include both a region adjacent to a current prediction unit block and a region not adjacent to a current prediction unit block, and a non-adjacent region may be a region within a distance between a certain pixel line and a current prediction unit block. When a non-adjacent region is used as a template, a corresponding distance may be transmitted from an encoder to a decoder. According to an embodiment, information for a distance may be defined by a promise between an encoder and a decoder, and the transmission of corresponding information may be omitted. When defined by a promise, a corresponding value may be fixed as a specific constant value or may be determined variably by the length of the horizontal or vertical pixel of a prediction unit block or the area of a block, etc. Subsequently, the final prediction signal of a current block may be generated by using a derived intra prediction mode.

[68] Prediction Performance Unit

[69] FIG. 2 illustrates the process of generating a prediction block through an optical flow-based inter prediction correction method.

[70] FIG. 2 is an embodiment to which the present invention is applied, and illustrates the process of generating a prediction block by correcting inter prediction information for all of the two reference pictures through an optical flow-based inter prediction correction method when a current block is in a mode that performs inter prediction by using two reference pictures.

[71] Referring to FIG. 2, a reference sample may be configured according to the inter prediction mode information of a current prediction block (S200).

[72] The inter prediction mode information may include an inter prediction mode, a prediction direction flag, a reference picture index, a motion vector, etc.

[73] The reference sample may include the first reference sample group and the second reference sample group, and the reference sample group may be composed of reference blocks determined through motion vector information for each reference picture.

[74] FIGS. 3 and 4 illustrate a distance value between a current picture and a reference picture.

[75] As an example, as shown in FIGS. 3 and 4, when the inter prediction mode of a current prediction block is a mode that performs inter prediction by using two reference pictures (the first reference picture and the second reference picture), the reference sample may include the first reference sample group and the second reference sample group, the first reference sample group may be composed of reference blocks (the first reference blocks) determined through motion vector information for the first reference picture, and the second reference sample group may be composed of reference blocks (the second reference blocks) determined through motion vector information for the second reference picture.

[76] The reference block may include pixel information in an integer pixel unit or less generated through an interpolation filter such as a DCT-based interpolation filter, etc., and the motion vector may be represented with precision in a fractional pixel unit less than or equal to an integer pixel unit to refer to the interpolated pixel of a reference block. In this case, when an interpolation filter is applied to generate pixel information in a fractional pixel unit for a reference block, a reference region for interpolation filtering may be configured in a reference picture based on a motion vector. The reference region may be composed of pixel information in an integer unit, and the size of the reference region may be determined based on the size of a current block and the number of taps of an interpolation filter. As an example, when a current block has a size of 8x8 and the number of taps of an interpolation filter is 15, a reference region for interpolation filtering may have a size of 23x23, which is the sum of the size of a current block and the number of taps of an interpolation filters. In this case, since the number of taps is odd, the number of top and bottom reference sample lines in a reference region and the number of left and right reference sample lines may be different. For example, in the example of a case in which the current block has a size of 8x8 and the number of taps of an interpolation filter is 15, the number of reference sample lines in a left reference region may be 7 and the number of reference sample lines in a right reference region may be 8.

[77] According to an embodiment, the size of a reference region for interpolation filtering may be limited to a specific size or less, and when the size of a reference region is limited to a specific size or less, pixel information located at the edge of a reference region may be padded and used in an interpolation filtering process.

[78] In the example of a case in which the current block has a size of 8x8 and the number of taps of an interpolation filter is 15, when the size of a reference region is limited, the number of reference sample lines in a left reference region may be limited to 3 and the number of reference sample lines in a right reference region may be limited to 4. The number of top and bottom reference sample lines may also be limited in the same manner as described above.

[79]

[80] Referring to FIG. 2, a distance value between a current picture and a reference picture and the pixel gradient value of a configured reference sample may be calculated (S210).

[81] The distance value between a current picture and a reference picture may be calculated as a difference value between the display order of a current picture and the display order of a reference picture.

[82] As an example, as shown in FIG. 3, when the inter prediction mode of a current prediction block refers to both a past frame (the first reference picture) and a future frame (the second reference picture) based on a current frame, a distance value between a current picture and the first reference picture may be calculated by C-(C-51)=51, a distance value between a current picture and the second reference picture may be calculated by C-(C+62)=-62, and the two distance values may have a different sign.

[83] As an example, as shown in FIG. 4(a), when the inter prediction mode of a current prediction block refers to only a past frame based on a current frame, a distance value between a current picture and the first reference picture may be calculated by 0-(0-51)=51, a distance value between a current picture and the second reference picture may be calculated by C- (0-52)=52, and the two distance values may have the same sign.

[84] As an example, as shown in FIG. 4(b), when the inter prediction mode of a current prediction block refers to only a future frame based on a current frame, a distance value between a current picture and the first reference picture may be calculated by C-(C+51)=-51, a distance value between a current picture and the second reference picture may be calculated by C- (C+52)=-52, and the two distance values may have the same sign.

[85] The pixel gradient value of the reference sample may include horizontal and vertical gradient values.

[86] FIG. 5 illustrates filter coefficients of a horizontal filter and a vertical filter.

[87] The horizontal and vertical gradient values of the reference sample may be calculated by using an edge detection filter as shown in the example of FIG. 5 for each pixel of a reference sample. In this case, the filter size and filter coefficient of an edge detection filter for calculating a gradient value may be defined according to a promise between an image encoding apparatus and decoding apparatus or may be determined based on the size of a current prediction block and inter prediction mode information.

[88] For a pixel located at the edge of a reference sample, a padding operation may be used to fill a pixel value for a position outside a reference sample region and a filled pixel value may be used to calculate a gradient. According to an embodiment, a padding operation may be performed by using a method for filling a pixel value with 0, a method for filling a value with the most adjacent pixel value, etc. Here, a reference sample adjacent to a reference region boundary may be a reference sample adjacent to the boundary of a limited reference region when the size of a reference region is limited.

[89] The reference sample may include pixel information in an integer pixel unit or less generated through an interpolation filter such as a DCT-based interpolation filter, a bilinear interpolation filter, a bicubic interpolation filter, etc. and may calculate the gradient value of a reference sample by using pixel information of an integer pixel or less. According to an embodiment, a filter obtained by combining an interpolation filter and an edge detection filter may be used to calculate a gradient value for pixel information in an integer pixel unit or less from the reference sample.

[90]

[91] FIG. 6 illustrates an embodiment of subsampling.

[92] As shown in the example of FIG. 6, the pixel gradient value of the reference sample may be calculated only for subsampled pixels after subsampling the pixels of a reference sample at a preset sampling interval. In this case, a sampling interval and a subsampling position may be defined according to a promise between an image encoding apparatus and decoding apparatus or may be determined based on the size of a current prediction block and inter prediction mode information.

[93] Referring to FIG. 2, a motion vector correction value may be calculated through an equation based on a bidirectional optical flow (S220).

[94] The equation may be derived by applying an optical flow equation between reference sample groups included in a reference sample.

[95] As an example, an equation such as Equation 3 may be derived by applying an optical flow equation (Equation 1 and Equation 2) between the first reference sample group and the second reference sample group.

[96] Equation 1

[97] gxt- (dt ■ v^gr, ■ (d, ■ vy)+(Pl-p2)-0

[98] Equation 2 [99,                     ■ (d2 ■ vy)+(P2-Pl)-0

[100] Equation 3

[101] WMW           -gy^-gy^-v^-p,]

[102] An equation obtained by applying an optical flow from the first reference sample group toward the second reference sample group may be represented as shown in Equation 1. Equation 1 above represents an optical flow equation in each pixel unit included in a reference sample group, and in Equation 1 above, gx1 and gy1 may refer to the horizontal and vertical gradients of the first reference sample group, d1 may refer to a distance value between a current picture and a reference picture including the first reference sample group, p1 and p2 may refer to the pixel values of the first reference sample group and the second reference sample group and vx and vy may refer to a motion vector correction value.

[103] An equation obtained by applying an optical flow from the second reference sample group toward the first reference sample group may be represented as shown in Equation 2. Equation 2 above represents an optical flow equation in each pixel unit included in a reference sample group, and in Equation 2 above, gx2 and gy2 may refer to the horizontal and vertical gradients of the second reference sample group, d2 may refer to a distance value between a current picture and a reference picture including the second reference sample group, p1 and p2 may refer to the pixel values of the first reference sample group and the second reference sample group and vx and vy may refer to a motion vector correction value.

[104] In Equation 3 above, p1 and p2 may refer to the pixel values of the first reference sample group and the second reference sample group, d1 and d2 may refer to a distance value between a current picture and a reference picture including the first reference sample group and a distance value between a current picture and a reference picture including the second reference sample group, gx1, gy1, gx2 and gy2 may refer to the horizontal and vertical gradients of the first reference sample group and the second reference sample group, Pdiff may refer to a difference in pixel values between the first reference sample group and the second reference sample group, and vx and vy may refer to a motion vector correction value.

[105] The motion vector correction value may be calculated by calculating vx and vy that minimize Pdiff in Equation 3 in the specific region unit of a reference sample group. According to an embodiment, one motion vector correction value may be calculated by applying Equation 3 to the entire region of a reference sample group. According to an embodiment, each reference sample group may be partitioned into M sub-regions, and Equation 3 may be applied to each sub-region to calculate M motion vector correction values for each sub-region. In this case, when the Sum of Absolute Difference (SAD) between two reference sample groups of each sub-region is less than a specific threshold, the process of calculating a motion vector correction value may be omitted for a corresponding sub-region.

[106] The vx and vy that minimize Pdiff in Equation 3 may be calculated through an optimization algorithm.

[107] As an example, as shown in Equation 4, the vx and vy that minimize Pdiff in Equation 3 may be calculated through a least square method.

[108] Equation 4

[109]

[110] In Equation 4 above, vx and vy may refer to a motion vector correction value, Pdiff may refer to a difference in pixel values between the first reference sample group and the second reference sample group, Q may refer to the specific region of a reference sample group which will calculate a motion vector correction value, and i and j may refer to an index representing a pixel position inside the specific region of a reference sample group which will calculate a motion vector correction value.

[111] The motion vector correction value may be calculated with precision in a fractional pixel unit less than or equal to an integer pixel unit. According to an embodiment, the precision of the motion vector correction value may be the same as or different from the precision of a motion vector included in the inter prediction mode information of a current prediction block. When the precision of the motion vector correction value is calculated to be different from the precision of the motion vector, it may be used by scaling the precision of the motion vector correction value or / and the precision of the motion vector to ensure that the precision of the motion vector correction value and the precision of the motion vector become the same.

[112] The range of the motion vector correction value may be limited according to the precision of the motion vector.

[113] For example, when a motion vector included in the inter prediction mode information of a current prediction block has 1 / 8 precision and a motion vector correction value is calculated to have 1 / 16 precision, it may be used by scaling the precision of the motion vector to 1 / 16 precision. In this case, a motion vector correction value may be limited to a range such as -1 +1.

[114] The process of calculating a motion vector correction value through an equation based on an optical flow may be performed by using a pixel located at the same position as a pixel subsampled for calculating a pixel gradient value in S210.

[115] Referring to FIG. 2, an inter prediction block may be generated by using the inter prediction mode information of a current block and the correction value of a calculated motion vector (S230).

[116] According to an embodiment, a corrected motion vector may be derived by weighted summing a motion vector included in the inter prediction mode information of a current block and a calculated motion vector correction value, and the inter prediction block of a current block may be generated through a reference block referenced by a corrected motion vector.

[117] As an example, when a motion vector correction value is calculated through an equation based on an optical flow between the first reference sample group and the second reference sample group, a corrected motion vector for the first reference picture and the second reference picture may be represented as shown in Equation 5, and the inter prediction block of a current block generated through a corrected motion vector may be represented as shown in Equation 6.

[118] Equation 5

[119] refine_MV1 = MV1 + (d1•vx, d1•vy) [12 0] refine_MV2 = MV2 + (d2-vx, d2^vy)

[121] Equation 6

[122] PredBlOCkinter = W1•RefBlOCkrefine_MV1 + W2^RefBlOCkrefine_MV2

[123] In Equation 5 above, refine_MV1 and refine MV2 may refer to a corrected motion vector for the first reference picture and the second reference picture, MV1 and MV2 may refer to a motion vector for the first reference picture and the second reference picture included in the inter prediction mode information of a current block, d1 and d2 may refer to a distance value between a current picture and the first reference picture and a distance value between a current picture and the second reference picture, and vx and vy may refer to a calculated motion vector correction value. According to an embodiment, when one motion vector correction value is calculated for the entire region of a reference sample group, one corrected motion vector for the entire region of a current block may be calculated. According to an embodiment, when M motion vector correction values are calculated for M sub-regions partitioned in each reference sample group, a corrected motion vector may be calculated for each of the M sub-blocks partitioned in a current block.

[124] In Equation 6 above, PredBlockinter refers to the inter prediction block of a current block generated through a corrected motion vector, RefBlockrefine_MV1 and RefBlockrefine_MV2 refer to a reference block referenced by a corrected motion vector, and w1 and w2 refer to a weight for each reference block. In this case, a prediction block may be generated through the corrected motion vector of each sub-block in a sub-block unit in which the corrected motion vector is calculated.

[125] A weight for the reference block may be determined to be the same as a weight for a reference block included in the inter prediction mode information of a current block or may be determined based on a distance between a current picture and a reference picture.

[126] According to an embodiment, the inter prediction block of a current block may be generated by correcting a prediction block generated through the inter prediction mode of a current block using an offset value derived through an optical flow equation and a calculated motion vector correction value.

[127] As an example, when a motion vector correction value is calculated through an equation based on an optical flow between the first reference sample group and the second reference sample group, an offset value in a pixel unit may be derived as shown in Equation 7, and the inter prediction block of a current block may be generated as shown in Equation 8.

[128] Equation 7 1 1 offsetixy)-— •    • gxx(x,y)-d2 • gx2(x,yS) • v^+— (dx • gy^x,yyd2 • gy2(x,yy) • vy

[129] 2                                  2

[130] Equation 8

[131] PredBlockinter(x,y) = w1-RefBlockMV1(x,y) + w2^RefBlockMV2(x,y) + w3•offset(x,y)

[132] In Equation 7 above, offset(x,y) may refer to the offset value in each pixel unit of a current prediction block, d1 and d2 may refer to a distance value between a current picture and the first reference picture and a distance value between a current picture and the second reference picture, gx1(x,y), gy1(x,y), gx2(x,y) and gy2(x,y) may refer to the horizontal and vertical gradients of the first reference sample group and the second reference sample group in each pixel unit, and vx and vy may refer to a motion vector correction value. According to an embodiment, when one motion vector correction value is calculated for the entire region of a reference sample group, one motion vector correction value for the entire region of a current block may be used to calculate an offset value in each pixel unit. According to an embodiment, when M motion vector correction values are calculated for M sub-regions partitioned in each reference sample group, an offset value in each pixel unit may be calculated by using each motion vector correction value in M sub-blocks partitioned in a current block.

[133] In Equation 8 above, PredBlockinter(x,y) may refer to each pixel value of the inter prediction block of a current block which is corrected using an offset value in a pixel unit, RefBlockMV1(x,y) and RefBlockMV2(x,y) may refer to a reference block referenced by a motion vector for the first reference picture and the second reference picture included in the inter prediction mode information of a current block, offset(x,y) may refer to the offset value in each pixel unit of a current prediction block, and w1, w2 and w3 may refer to a weight for each reference block and offset value.

[134] FIG. 7 illustrates the process of generating a prediction block by correcting inter prediction information for one of the two reference pictures through an optical flow-based inter prediction correction method.

[135] FIG. 7 is an embodiment to which the present invention is applied, and illustrates the process of generating a prediction block by correcting inter prediction information for one of the two reference pictures through an optical flow-based inter prediction correction method when a current block is in a mode that performs inter prediction by using two reference pictures.

[136] Referring to FIG. 7, a reference sample may be configured according to the inter prediction mode information of a current prediction block (S700).

[137] The inter prediction mode information may include an inter prediction mode, a prediction direction flag, a reference picture index, a motion vector, etc.

[138] The reference sample may include the first reference sample group and the second reference sample group, and the reference sample group may be composed of reference blocks determined through motion vector information for each reference picture.

[139] As an example, as shown in FIGS. 3 and 4, when the inter prediction mode of a current prediction block is a mode that performs inter prediction by using two reference pictures (the first reference picture and the second reference picture), the reference sample may include the first reference sample group and the second reference sample group, the first reference sample group may be composed of reference blocks (the first reference blocks) determined through motion vector information for the first reference picture, and the second reference sample group may be composed of reference blocks (the second reference blocks) determined through motion vector information for the second reference picture.

[140] The reference block may include pixel information in an integer pixel unit or less generated through an interpolation filter such as a DCT-based interpolation filter, etc., and the motion vector may be represented with precision in a fractional pixel unit less than or equal to an integer pixel unit to refer to the interpolated pixel of a reference block. In this case, when an interpolation filter is applied to generate pixel information in a fractional pixel unit for a reference block, a reference region for interpolation filtering may be configured in a reference picture based on a motion vector. The reference region may be composed of pixel information in an integer unit, and the size of the reference region may be determined based on the size of a current block and the number of taps of an interpolation filter. As an example, when a current block has a size of 8x8 and the number of taps of an interpolation filter is 15, a reference region for interpolation filtering may have a size of 23x23, which is the sum of the size of a current block and the number of taps of an interpolation filters. According to an embodiment, the size of a reference region for interpolation filtering may be limited to a specific size or less, and when the size of a reference region is limited to a specific size or less, pixel information located at the edge of a reference region may be padded and used in an interpolation filtering process. In this case, since the number of taps is odd, the number of top and bottom reference sample lines in a reference region and the number of left and right reference sample lines may be different. For example, in the example of a case in which the current block has a size of 8x8 and the number of taps of an interpolation filter is 15, the number of reference sample lines in a left reference region may be 7 and the number of reference sample lines in a right reference region may be 8.

[141]

[142] Referring to FIG. 7, the pixel gradient value of a configured reference sample may be calculated. (S710)

[143] The process of calculating the gradient value of the reference sample may be performed for only one of the first reference sample group and the second reference sample group. According to an embodiment, when correcting the inter prediction information of the first reference picture among the two reference pictures, the process of calculating the gradient value of a reference sample may be performed only for the second reference sample group. According to an embodiment, when correcting the inter prediction information of the second reference picture among the two reference pictures, the process of calculating the gradient value of a reference sample may be performed only for the first reference sample group.

[144] Between the first reference sample group and the second reference sample group, a reference sample group subject to inter prediction information correction may be determined by signaling / parsing a 1-bit flag or may be determined based on the inter prediction information of a current block. According to an embodiment, a reference sample group for a reference picture for which a motion vector difference is not included in the inter prediction information of a current block may be determined to be subject to inter prediction information correction. For example, when the inter prediction information of a current block includes both a motion vector predictor and a motion vector difference for the first reference picture and includes only a motion vector predictor for the second reference picture, the second reference sample group may be determined as a reference sample group subject to inter prediction information correction. According to an embodiment, when the inter prediction information of a current block is derived by signaling / parsing only one motion vector difference and scaling the remaining motion vector differences from a parsed motion vector difference, a reference sample group for a reference picture that uses a signaled / parsed motion vector difference may be determined to be subject to inter prediction information correction. For example, when the inter prediction information of a current block signals / parses a motion vector difference for the first reference picture and a motion vector difference for the second reference picture is derived by scaling a motion vector difference for the first reference picture, the second reference sample group may be determined as a reference sample group subject to inter prediction information correction.

[145] The pixel gradient value of the reference sample may include horizontal and vertical gradient values.

[146] The horizontal and vertical gradient values of the reference sample may be calculated by using an edge detection filter as shown in the example of FIG. 5 for each pixel of a reference sample. In this case, the filter size and filter coefficient of an edge detection filter for calculating a gradient value may be defined according to a promise between an image encoding apparatus and decoding apparatus or may be determined based on the size of a current prediction block and inter prediction mode information.

[147] For a pixel located at the edge of a reference sample, a padding operation may be used to fill a pixel value for a position outside a reference sample region and a filled pixel value may be used to calculate a gradient. According to an embodiment, a padding operation may be performed by using a method for filling a pixel value with 0, a method for filling a value with the most adjacent pixel value, etc. Here, a reference sample adjacent to a reference region boundary may be a reference sample adjacent to the boundary of a limited reference region when the size of a reference region is limited.

[148] The reference sample may include pixel information in an integer pixel unit or less generated through an interpolation filter such as a DCT-based interpolation filter, a bilinear interpolation filter, a bicubic interpolation filter, etc. and may calculate the gradient value of a reference sample by using pixel information of an integer pixel or less. According to an embodiment, a filter obtained by combining an interpolation filter and an edge detection filter may be used to calculate a gradient value for pixel information in an integer pixel unit or less from the reference sample.

[149] As shown in the example of FIG. 6, the pixel gradient value of the reference sample may be calculated for subsampled pixels after subsampling the pixels of a reference sample at a preset sampling interval. In this case, a sampling interval and a subsampling position may be defined according to a promise between an image encoding apparatus and decoding apparatus or may be determined based on the size of a current prediction block and inter prediction mode information.

[150] Referring to FIG. 7, a motion vector correction value may be calculated through a unidirectional optical flow equation. (S720)

[151] When a pixel gradient value for the second reference sample group is calculated in S710, a motion vector correction value for the first reference picture may be calculated through the process of calculating the motion vector correction value, and when a pixel gradient value for the first reference sample group is calculated in S710, a motion vector correction value for the second reference picture may be calculated through the process of calculating the motion vector correction value.

[152] The optical flow equation may be derived by applying an optical flow between reference sample groups included in a reference sample.

[153] For example, when correcting the inter prediction information of the second reference picture among the two reference pictures, an equation obtained by applying an optical flow from the first reference sample group toward the second reference sample group may be represented as shown in Equation 9.

[154] Equation 9

[155]

[156] Equation 9 above represents an optical flow equation in each pixel unit included in a reference sample group, and in Equation 9 above, gx and gy may refer to the horizontal and vertical gradients of the first reference sample group, p1 and p2 may refer to the pixel value of the first reference sample group and the second reference sample group, and vx and vy may refer to a motion vector correction value.

[157] The motion vector correction value may calculate vx and vy by calculating an optical flow equation in Equation 9 in the specific region unit of a reference sample group. According to an embodiment, one motion vector correction value may be calculated by applying Equation 9 to the entire region of a reference sample group. According to an embodiment, each reference sample group may be partitioned into M sub-regions, and Equation 9 may be applied to each sub-region to calculate M motion vector correction values for each sub-region. In this case, when the Sum of Absolute Difference (SAD) between two reference sample groups of each sub-region is less than a specific threshold, the process of calculating a motion vector correction value may be omitted for a corresponding sub-region.

[158] The optical flow equation of Equation 9 above may be calculated through an optimization algorithm.

[159] For example, as shown in Equation 10, the optical flow equation of Equation 9 above may be calculated through a least square method.

[160] Equation 10

[161] / -^-^+^-^1-^

[162]

[163] In Equation 10 above, vx and vy may refer to a motion vector correction value, f may refer to an optical flow equation, Q may refer to the specific region of a reference sample group which will calculate a motion vector correction value, and i and j may refer to an index representing a pixel position inside the specific region of a reference sample group which will calculate a motion vector correction value.

[164] The motion vector correction value may be calculated with precision in a fractional pixel unit less than or equal to an integer pixel unit. According to an embodiment, the precision of the motion vector correction value may be the same as or different from the precision of a motion vector included in the inter prediction mode information of a current prediction block. When the precision of the motion vector correction value is calculated to be different from the precision of the motion vector, it may be used by scaling the precision of the motion vector correction value or / and the precision of the motion vector to ensure that the precision of the motion vector correction value and the precision of the motion vector become the same.

[165] The range of the motion vector correction value may be limited according to the precision of the motion vector.

[166] For example, when a motion vector included in the inter prediction mode information of a current prediction block has 1 / 8 precision and a motion vector correction value is calculated to have 1 / 16 precision, it may be used by scaling the precision of the motion vector to 1 / 16 precision. In this case, a motion vector correction value may be limited to a range such as -1 ~ +1.

[167] The process of calculating a motion vector correction value through the optical flow equation may be performed by using a pixel located at the same position as a pixel subsampled for calculating a pixel gradient value in S710.

[168] Referring to FIG. 7, an inter prediction block may be generated by using the inter prediction mode information of a current block and the correction value of a calculated motion vector. (S730)

[169] According to an embodiment, a corrected motion vector may be derived by weighted summing a motion vector included in the inter prediction mode information of a current block and a calculated motion vector correction value, and the inter prediction block of a current block may be generated through a reference block referenced by a corrected motion vector.

[170] For example, when a motion vector correction value is calculated for the second reference picture among the two reference pictures, a corrected motion vector for the second reference picture may be represented as shown in Equation 11, and the inter prediction block of a current block generated through a corrected motion vector may be represented as shown in Equation 12.

[171] Equation 11

[172] refine_MV2 = MV2 + (vx,vy)

[173] Equation 12

[174] PredBlockinter = w1-RefBlockMV1 + w2^RefBlockrefineMV2

[175] In Equation 11 above, refine_MV2 may refer to a corrected motion vector for the second reference picture, MV2 may refer to a motion vector for the second reference picture included in the inter prediction mode information of a current block, and vx and vy may refer to a calculated motion vector correction value. According to an embodiment, when one motion vector correction value is calculated for the entire region of a reference sample group, one corrected motion vector for the entire region of a current block may be calculated. According to an embodiment, when M motion vector correction values are calculated for M sub regions partitioned in each reference sample group, a corrected motion vector may be calculated for each of the M sub-blocks partitioned in a current block.

[176] In Equation 12 above, PredBlockinter refers to the inter prediction block of a current block generated through a corrected motion vector, RefBlockMV1 refers to a reference block referenced by a motion vector for the first reference picture included in the inter prediction mode information of a current block, RefBlockrefine_MV2 refers to a reference block referenced by a corrected motion vector for the second reference picture, and w1 and w2 refer to a weight for each reference block. In this case, a prediction block may be generated through the corrected motion vector of each sub-block in a sub-block unit in which the corrected motion vector is calculated.

[177]

[178] FIG. 8 illustrates the process of correcting the motion vector of inter prediction information through an optical flowbased inter prediction correction method.

[179] FIG. 8 is an embodiment to which the present invention is applied, and illustrates the process of correcting the motion vector of inter prediction information through an optical flowbased inter prediction correction method.

[180] Referring to FIG. 8, a reference sample may be configured according to the inter prediction mode information of a current prediction block (S800).

[181] The inter prediction mode information may include an inter prediction mode, a prediction direction flag, a reference picture index, a motion vector, etc.

[182] The reference sample may include the first reference sample group and the second reference sample group, and the reference sample group may be composed of a neighboring region adjacent to a current block and a neighboring region adjacent to a reference block determined through motion vector information for a reference picture.

[183] FIG. 9 illustrates an embodiment of reference sample group configuration.

[184] For example, as shown in FIG. 9, the first reference sample group may be composed of neighboring regions adjacent to a current block, and the second reference sample group may be composed of neighboring regions adjacent to the first reference block.

[185] FIG. 10 illustrates the neighboring line of a current block.

[186] The neighboring region may be determined according to the size and aspect ratio of a current block. For example, as shown in FIG. 10, the neighboring region may be composed of X vertical lines adjacent to the left of a current block or / and Y horizontal lines adjacent to the top of a current block or / and an XxY-sized region adjacent to the top-left of a current block.

[187] Referring to FIG. 8, the pixel gradient value of a configured reference sample may be calculated (S810).

[188] The process of calculating the gradient value of the reference sample may be performed only for a reference sample group consisting of neighboring regions of a current block.

[189] The pixel gradient value of the reference sample may include horizontal and vertical gradient values.

[190] The horizontal and vertical gradient values of the reference sample may be calculated by using an edge detection filter as shown in the example of FIG. 5 for each pixel of a reference sample. In this case, the filter size and filter coefficient of an edge detection filter for calculating a gradient value may be defined according to a promise between an image encoding apparatus and decoding apparatus or may be determined based on the size of a current prediction block and inter prediction mode information.

[191] For a pixel located at the edge of a reference sample, a padding operation may be used to fill a pixel value for a position outside a reference sample region and a filled pixel value may be used to calculate a gradient. According to an embodiment, a padding operation may be performed by using a method for filling a pixel value with 0, a method for filling a value with the most adjacent pixel value, etc. Here, a reference sample adjacent to a reference region boundary may be a reference sample adjacent to the boundary of a limited reference region when the size of a reference region is limited.

[192] The reference sample may include pixel information in an integer pixel unit or less generated through an interpolation filter such as a DCT-based interpolation filter, a bilinear interpolation filter, a bicubic interpolation filter, etc. and may calculate the gradient value of a reference sample by using pixel information of an integer pixel or less. According to an embodiment, a filter obtained by combining an interpolation filter and an edge detection filter may be used to calculate a gradient value for pixel information in an integer pixel unit or less from the reference sample.

[193] As shown in the example of FIG. 6, the pixel gradient value of the reference sample may be calculated for subsampled pixels after subsampling the pixels of a reference sample at a preset sampling interval. In this case, a sampling interval and a subsampling position may be defined according to a promise between an image encoding apparatus and decoding apparatus or may be determined based on the size of a current prediction block and inter prediction mode information.

[194] Referring to FIG. 8, a motion vector correction value may be calculated through a unidirectional optical flow equation (S820).

[195] The optical flow equation may be derived by applying an optical flow between a reference sample group composed of neighboring regions of a current block and a reference sample group composed of neighboring regions of a reference block.

[196] For example, an equation obtained by applying an optical flow from the first reference sample group composed of neighboring regions of a current block toward the second reference sample group composed of neighboring regions of a reference block may be represented as shown in Equation 13 above.

[197] Equation 13

[198]

[199] Equation 13 above represents an optical flow equation in each pixel unit included in a reference sample group, and in Equation 13 above, gx and gy may refer to the horizontal and vertical gradients of the first reference sample group composed of neighboring regions of a current block, p1 and p2 may refer to the pixel value of the first reference sample group and the second reference sample group composed of neighboring regions of a reference block, and vx and vy may refer to a motion vector correction value.

[200] The motion vector correction value may calculate vx and vy by calculating an optical flow equation in Equation 13 in the specific region unit of a reference sample group. According to an embodiment, one motion vector correction value may be calculated by applying Equation 13 to the entire region of a reference sample group. According to an embodiment, each reference sample group may be partitioned into M sub-regions, and Equation 13 may be applied to each sub-region to calculate M motion vector correction values for each sub-region. In this case, when the Sum of Absolute Difference (SAD) between two reference sample groups of each sub-region is less than a specific threshold, the process of calculating a motion vector correction value may be omitted for a corresponding sub-region.

[201] The optical flow equation of Equation 13 above may be calculated through an optimization algorithm.

[202] For example, as shown in Equation 14, the optical flow equation of Equation 13 above may be calculated through a least square method.

[203] Equation 14

[204] / -^-^+^-^1-^ (VX,Vy)- mm S Ai,j]

[205]

[206] In Equation 14 above, vx and vy may refer to a motion vector correction value, f may refer to an optical flow equation, Q may refer to the specific region of a reference sample group which will calculate a motion vector correction value, and i and j may refer to an index representing a pixel position inside the specific region of a reference sample group which will calculate a motion vector correction value.

[207] The motion vector correction value may be calculated with precision in a fractional pixel unit less than or equal to an integer pixel unit. According to an embodiment, the precision of the motion vector correction value may be the same as or different from the precision of a motion vector included in the inter prediction mode information of a current block. When the precision of the motion vector correction value is calculated to be different from the precision of the motion vector, it may be used by scaling the precision of a motion vector correction value or / and the precision of a motion vector to ensure that the precision of a motion vector correction value and the precision of a motion vector become the same.

[208] The range of the motion vector correction value may be limited according to the precision of the motion vector.

[209] For example, when a motion vector included in the inter prediction mode information of a current prediction block has 1 / 8 precision and a motion vector correction value is calculated to have 1 / 16 precision, it may be used by scaling the precision of the motion vector to 1 / 16 precision. In this case, a motion vector correction value may be limited to a range such as -1 ~ +1.

[210] The process of calculating a motion vector correction value through the optical flow equation may be performed by using a pixel located at the same position as a pixel subsampled for calculating a pixel gradient value in S810.

[211] Referring to FIG. 8, the motion vector of the boundary region of a current block may be corrected by using a calculated motion vector correction value (S830). For example, the motion vector of the boundary region of a current block may represent the motion vector of a region (or a sub-region) which is adjacent to the boundary of a current block and is located within a current block.

[212] FIG. 11 illustrates the boundary region of a current block.

[213] As shown in FIG. 11, the boundary region may be a region within a current block that is adjacent to a reference sample group in which a motion vector correction value is calculated. For example, as shown in FIG. 11, the boundary region may be composed of I vertical lines at the left edge within a current block or / and J horizontal lines at the top edge within a current block. In this case, the size of the boundary region may be determined according to the size and aspect ratio of a current block.

[214] FIG. 12 illustrates an example in which the motion vector of the boundary region of a current block is corrected.

[215] The motion vector of the boundary region may be corrected by using a motion vector correction value calculated in a reference sample group region adjacent to the left, top and top left. In this case, when a reference sample group is partitioned into M sub-regions and M motion vector correction values are calculated for each sub-region, as shown in the example of FIG. 12, a boundary region may be partitioned into K sub-regions and a motion vector for each of the K partitioned sub-regions may be corrected by using the motion vector correction value of the sub-region of an adjacent reference sample group.

[216] A prediction block may be generated by using an existing motion vector included in the inter prediction mode information of a current prediction block and the motion vector of a boundary region corrected by a motion vector correction value per sub-region. In this case, a filtering operation may be performed on a sub-block boundary in a sub-block unit in a generated prediction block. According to an embodiment, a gradient may be calculated for each sub-block boundary, and when a gradient is greater than or equal to a specific threshold, a filtering operation may be performed by using a smoothing filter. According to an embodiment, a different filtering operation may be performed on a boundary between prediction sub blocks generated through the motion vector of the corrected boundary region and a boundary between a prediction sub-block generated through an existing motion vector and a prediction sub-block generated through the motion vector of the corrected boundary region. For example, a stronger smoothing operation may be performed on a boundary between a prediction sub-block generated through an existing motion vector and a prediction sub-block generated through the motion vector of the corrected boundary region.

[217] When a current block is in a mode that performs inter prediction by using two or more reference blocks, the motion vector of the boundary region of a current block may be corrected by applying the process of FIG. 8 between a current block and each reference block.

[218] For example, when a current block is in a mode that performs inter prediction by using three reference blocks (the first reference block, the second reference block, the third reference block), a motion vector may be corrected by using a value obtained by weighted summing a motion vector correction value generated through the process of FIG. 8 between a current block and the first reference block (the first process), a motion vector correction value generated through the process of FIG. 8 between a current block and the second reference block (the second process) and a motion vector correction value generated through the process of FIG. 8 between a current block and the third reference block (the third process).

[219] FIG. 13 illustrates a method for generating an inter prediction block by applying an optical flow-based inter prediction correction method when a current block is in a mode that performs inter prediction by using a temporal interpolation block and one reference block.

[220] FIG. 13 is an embodiment to which the present invention is applied, and illustrates a method for generating an inter prediction block by applying an optical flow-based inter prediction correction method in FIGS. 2 and 7 when a current block is in a mode that performs inter prediction by using a temporal interpolation block and one reference block.

[221] The temporal interpolation block may be generated by first generating a temporal interpolation motion vector by interpolating or extrapolating, based on a distance between pictures, the motion vector of the first temporal interpolation picture block that refers to the second temporal interpolation picture block among the reference pictures included in the inter prediction information of a current frame, and then performing inter prediction by referring to the first temporal interpolation reference block and the second temporal interpolation reference block through the temporal interpolation motion vector.

[222] The distance between pictures may be calculated based on a difference between the display order of a current picture and the display order of a temporal interpolation reference picture.

[223] The inter prediction process of FIG. 13 may consist of the process of generating a temporal interpolation block through the optical flow-based inter prediction correction method of FIG. 2 (the first process) and the process of generating the inter prediction block of a current block through the optical flow based inter prediction correction method of FIG. 7 (the second process).

[224] In the first process, a temporal interpolation block may be generated by using the optical flow-based inter prediction correction method of FIG. 2 between the first temporal interpolation reference block and the second temporal interpolation reference block. In this case, a distance value between a current picture and a reference picture in S210 may be calculated by a display order difference value (C-T1) between a current picture and the first temporal interpolation reference picture and a display order difference value (C-T2) between a current picture and the second temporal interpolation reference picture.

[225] In the second process, the inter prediction block of a current block may be generated by using the optical flow-based inter prediction correction method of FIG. 7 between a temporal interpolation block generated during the first process and the first reference block. In this case, the pixel gradient calculation process of S710 may be performed only for a temporal interpolation block, and accordingly, only a motion vector correction value for the first reference block may be calculated in S720.

[226] FIG. 14 illustrates a method for generating an inter prediction block by applying an optical flow-based inter prediction correction method when a current block is in a mode that performs inter prediction by using a geometric partitioning mode.

[227] The geometric partitioning mode may partition a current block into five sub-regions according to a geometric partitioning line and generate the inter prediction block of a current block by using an inter prediction block generated by using N reference blocks as a reference block for each sub region.

[228] The geometric partitioning line may be defined according to an angle value formed between a partitioning line and a horizontal or vertical line and a distance value between the center of a current block and a partitioning line.

[229] FIG. 14 is an example in which a current block is partitioned into two sub-regions according to a geometric partitioning line and an inter prediction block generated by using two respective reference blocks (the first prediction block and the second prediction block) are used as the reference picture of each sub-region.

[230] A reference picture (the first reference picture, the second reference picture, the third reference picture, the fourth reference picture) including the reference blocks of the first prediction block and the second prediction block (the first reference block, the second reference block, the third reference block, the fourth reference block) may be the same as or different from each other.

[231] The inter prediction process of FIG. 14 may consist of the process of generating the first prediction block through the optical flow-based inter prediction correction method of FIG. 2 or FIG. 7 (the first process), the process of generating the second prediction block through the optical flow-based inter prediction correction method of FIG. 2 or FIG. 7 (the second process) and the process of generating the inter prediction block of a geometric partitioning mode by using the first prediction block and the second prediction block (the third process).

[232] In the first process, the first prediction block may be generated by using the optical flow-based inter prediction correction method of FIG. 2 or FIG. 7 between the first reference block and the second reference block. When the method of FIG. 2 is used, a distance value between a current picture and a reference picture in S210 may be calculated by a display order difference value (C-R1) between a current picture and the first reference picture and a display order difference value (C- R2) between a current picture and the second reference picture. When the method of FIG. 7 is used, the pixel gradient calculation process of S710 may be performed only for the first reference block or the second reference block, and accordingly, only a motion vector correction value for the second reference block or the first reference block may be calculated in S720.

[233] In the second process, the second prediction block may be generated by using the optical flow-based inter prediction correction method of FIG. 2 or FIG. 7 between the third reference block and the fourth reference block. When the method of FIG. 2 is used, a distance value between a current picture and a reference picture in S210 may be calculated by a display order difference value (C-R3) between a current picture and the third reference picture and a display order difference value (C- R4) between a current picture and the fourth reference picture. When the method of FIG. 7 is used, the pixel gradient calculation process of S710 may be performed only for the third reference block or the fourth reference block, and accordingly, only a motion vector correction value for the fourth reference block or the third reference block may be calculated in S720.

[234] In the third process, inter prediction may be performed by using the first prediction block generated in the first process and the second prediction block generated in the second process as the reference picture of each sub-region partitioned according to a geometric partitioning line, thereby generating the inter prediction block of a current block.

[235] Inverse Transform Unit Determination Unit

[236] According to an embodiment, transform and inverse transform units in an encoder / a decoder may be one transform unit (TU) or each sub-block in which one TU is partitioned into multiple parts in the inverse transform unit determination unit.

[237] Inverse Transform Kernel Determination Unit

[238] According to an embodiment, the inverse transform kernel determination unit may determine separable vertical and horizontal primary inverse transform kernels or / and a non separable secondary inverse transform kernel or may determine a non-separable primary inverse transform kernel.

[239] Inverse Transform Performance Unit

[240] According to an embodiment, the inverse transform performance unit may perform an inverse transform on a dequantized transform coefficient by using an inverse transform unit determined in the inverse transform unit determination unit and an inverse transform kernel determined in the inverse transform kernel determination unit.

[241] According to an embodiment, whether to perform a non separable primary inverse transform and whether to perform a non-separable secondary inverse transform may be determined through signaling / parsing, and may be implicitly determined based on the size of a current TU, etc.

[242] Transform Coefficient Entropy Decoding unit

[243] According to an embodiment, the transform coefficient entropy decoding unit may reconstruct a quantized secondary transform coefficient when a secondary transform is applied through an entropy decoding method (CABAC(Context-Adaptive Binary Arithmetic Coding), CAVLC(Context-Adaptive VariableLength Coding), etc.), and may reconstruct a quantized primary transform coefficient when a secondary transform is not applied.

[244] Transform Coefficient Dequantization Unit

[245] According to an embodiment, the transform coefficient dequantization unit may parse information such as a quantization method, a quantization parameter, etc. and perform dequantization on a reconstructed transform coefficient to obtain a dequantized transform coefficient.

[246] Reconstruction Performance Unit

[247] According to an embodiment, the inverse transform performance unit may generate a final reconstructed signal by combining a final prediction signal generated through the prediction performance unit and a reconstructed residual signal.

[248]

[249] Although the exemplary methods of the present disclosure are expressed as a series of operations for clarity of description, this is not intended to limit the order in which steps are performed, and if necessary, each step may be performed simultaneously or in different order. In order to implement a method according to the present disclosure, another step may be additionally included in an exemplary step or the remaining steps may be included excluding some steps or another additional step may be included excluding some steps.

[250] The various embodiments of the present disclosure do not list all possible combinations, but are intended to describe the representative aspect of the present disclosure, and matters described in various embodiments may be applied independently or in a combination of at least two.

[251] In addition, the various embodiments of the present disclosure may be implemented by hardware, firmware, software or a combination thereof. For implementation by hardware, they may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.

[252] The range of the present disclosure includes software or machine-executable instructions (i.e., an operating system, an application, firmware, a program, etc.) that enable operations according to the methods of various embodiments to be executed on a device or computer, and a non-transitory computer-readable recording medium in which such software or instructions are stored and executable on a device or computer. [industrial Applicability]

[253] The present disclosure may be used in the industrial fields where an optical flow-based inter prediction correction method, apparatus and recording medium are used.

Claims

[claims]

1. An image decoding method, comprising:configuring a reference sample based on inter predictionmode information of a current block;calculating a pixel gradient value of the reference samplebased on a distance value between a current picture includingthe current block and a reference picture;calculating a motion vector correction value of the currentblock based on the pixel gradient value; andgenerating a prediction block of the current block based onthe motion vector correction value and the inter prediction mode information.

2. The method of claim 1,block determined by a motion vector for a first referencepicture of the current block and a second reference block determined by a motion vector for a second reference picture of the current block, andwherein directions of the first reference picture and the second reference picture are different from each other.

3. The method of claim 1,wherein the reference sample includes a first reference block determined by a motion vector for a first reference picture of the current block and a second reference block determined by a motion vector for a second reference picture of the current block, andwherein directions of the first reference picture and the second reference picture are equal to each other.

4. The method of claim 1,andwherein a motion vector determining the reference block is determined based on pixel information in a fractional pixel unit for the reference block.

5. The method of claim 4,wherein the pixel information in the fractional pixel unit for the reference block is obtained by applying interpolation filtering to the reference block.

6. The method of claim 5,wherein a size of a reference region used for the interpolation filtering is determined based on a size of the current block and a number of taps of an interpolation filter.

7. The method of claim 6,wherein the size of the determined reference region is limited to a specific size or less.

8. The method of claim 1,wherein the distance value between the current block and the reference picture is calculated as a difference value between a display order of the current picture and a display order of the reference picture.

9. The method of claim 1,wherein the pixel gradient value includes a horizontal gradient value and a vertical gradient value.

10. The method of claim 9,wherein the pixel gradient value is calculated by using anedge detection filter.

11. The method of claim 10,wherein a filter size and a filter coefficient of the edge detection filter are determined based on a size of the current block and the inter prediction mode information.

12. The method of claim 11,wherein in response to the edge detection filter being applied to a pixel located at an edge of the reference sample, a pixel at a position outside a reference sample region of the reference sample is filled through a padding operation.

13. An image encoding method, comprising:configuring a reference sample based on inter predictionmode information of a current block;calculating a pixel gradient value of the reference samplebased on a distance value between a current picture includingthe current block and a reference picture;calculating a motion vector correction value of the currentblock based on the pixel gradient value; andgenerating a prediction block of the current block based onthe motion vector correction value and the inter prediction modeinformation.

14. A non-transitory computer-readable recording medium forstoring a bitstream generated by an image encoding method,wherein the image encoding method comprises:configuring a reference sample based on inter predictionmode information of a current block;calculating a pixel gradient value of the reference samplebased on a distance value between a current picture includingthe current block and a reference picture;block based on the pixel gradient value; andgenerating a prediction block of the current block based onthe motion vector correction value and the inter prediction modeinformation.

Images