Video decoding method and apparatus, and video encoding method and apparatus

Abstract

A video decoding method for obtaining a prediction block of a current block is provided. The method determines a horizontal or vertical displacement vector per unit time of pixels of a current block, including neighboring pixels inside the boundary of the current block, using values ​​related to reference pixels included in the first and second reference blocks without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks, and performs block-wise motion compensation and pixel group-wise motion compensation for the current block based on the horizontal or vertical gradient values ​​of corresponding reference pixels in the first and second reference blocks corresponding to a current pixel included in a current pixel group in the current block, the pixel values ​​of the corresponding reference pixels, and the horizontal or vertical displacement vector per unit time of the current pixel, thereby obtaining a prediction block for the current block. The pixel group includes at least one pixel.

Description

[Technical Field]

[0001] The present invention relates to a video decoding method and video encoding, and more particularly to a video decoding method and video encoding method that perform inter prediction in a bi-directional motion prediction mode. [Background technology]

[0002] With the development and widespread use of hardware capable of playing and storing high-resolution or high-quality video content, there is an increasing need for video codecs that can effectively encode and decode high-resolution or high-quality video content. In existing video codecs, video is encoded using a limited encoding method based on a tree-structured coding unit.

[0003] Using frequency transform, spatial domain video data is transformed into frequency domain coefficients. To speed up frequency transform calculations, video codecs divide video into blocks of a predetermined size, perform DCT transform on each block, and encode frequency coefficients for each block. Compared to spatial domain video data, frequency domain coefficients are more easily compressed. In particular, since spatial domain video pixel values ​​are expressed as prediction errors through inter-prediction or intra-prediction of video codecs, when frequency transform is performed on the prediction errors, much of the data is converted to zero. Video codecs reduce data volume by replacing continuously occurring data with small-sized data. Summary of the Invention [Problem to be solved by the invention]

[0004] According to various embodiments, in a bidirectional motion prediction mode, a predicted pixel value of a current block is generated not only by using pixel values ​​of a first reference block of a first reference picture and pixel values ​​of a second reference block of a second reference picture but also by using both a first gradient value of the first reference block and a second gradient value of the second reference block. Therefore, a predicted block similar to the original block is generated, thereby improving encoding / decoding efficiency.

[0005] When performing pixel group-based motion compensation, the pixel values ​​of the first reference block, the pixel values ​​of the second reference block, the first gradient value of the first reference block, and the second gradient value of the second reference block are used to determine a horizontal or vertical displacement vector of the current block. In particular, to determine the horizontal or vertical displacement vector of a current pixel in the current block, not only the pixel value and gradient value of the first reference pixel in the first reference block corresponding to the current pixel and the pixel value and gradient value of the second reference pixel in the second reference block are used, but also the pixel values ​​and gradient values ​​of surrounding pixels included in a window of a predetermined size centered on the first and second reference pixels. Therefore, when the current pixel is located on a boundary, the surrounding pixels of the reference pixel corresponding to the current pixel are located outside the reference block, so the pixel values ​​and gradient values ​​of the pixels located outside the reference block must be referenced, which causes a problem of requiring additional memory access.

[0006] According to various embodiments, the number of memory accesses can be minimized by determining the horizontal or vertical displacement vector of the current block by referring only to the pixel values ​​and gradient values ​​of pixels located inside the reference block, without referring to the pixel values ​​and gradient values ​​stored for pixels located outside the reference block.

[0007] According to various embodiments, instead of using pixel values ​​of integer pixels as input and horizontal and vertical gradient filters and interpolation filters to determine the horizontal or vertical gradient value of a reference pixel, the interpolation filter is applied to the pixel values ​​of the integer pixels to determine the pixel value of a pixel located at a fractional pixel unit, and a horizontal or vertical gradient filter with a relatively short filter length is applied to the pixel value of the pixel located at the fractional pixel unit to determine the horizontal or vertical gradient value of the reference pixel, thereby minimizing the need for complex multiplication operations.

[0008] According to various embodiments, by performing motion compensation in units of pixel groups, it is possible to minimize the need for more complex multiplication operations than when performing motion compensation in units of pixels.

[0009] The present invention may also include a computer-readable recording medium having a program recorded thereon for implementing the methods according to various embodiments.

[0010] Here, the technical problems of the various embodiments are not limited to the features mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0011] The technical problems of the present invention are not limited to the features mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

[0012] According to various embodiments, a video decoding method includes the steps of: obtaining motion prediction mode information for a current block in a current picture from a bitstream; If the acquired motion prediction mode information indicates a bi-directional motion prediction mode, acquiring information about a first motion vector indicating a first reference block of a current block in a first reference picture and information about a second motion vector indicating a second reference block of the current block in a second reference picture from the bitstream; determining a displacement vector per unit time in a horizontal or vertical direction of pixels of a current block including adjacent pixels inside a boundary of the current block by using values ​​related to reference pixels included in the first reference block and the second reference block without using values ​​stored for pixels located outside the boundaries of the first reference block and the second reference block; obtaining a prediction block of the current block by performing block-wise motion compensation and pixel group-wise motion compensation of the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel; obtaining information about a residual block of a current block from the bitstream; reconstructing the current block based on the predicted block and the residual block; The pixel group may include at least one pixel.

[0013] In various embodiments of the video decoding method, the step of obtaining a prediction block of the current block comprises: further comprising applying a horizontal or vertical gradient filter to pixel values ​​of sub-position pixels in the first reference block or the second reference block to calculate horizontal or vertical gradient values ​​of the first corresponding reference pixel or the second corresponding reference pixel; the gradient filter is a 5-tap filter; The fractional position pixel is also a pixel in which at least one of the horizontal and vertical components of the coordinates indicating the pixel position has a fractional value.

[0014] In various embodiments of the video decoding method, the pixel values ​​of the sub-position pixels in the first reference block or the second reference block may also be calculated by applying a horizontal or vertical interpolation filter to the pixel values ​​of the integer position pixels.

[0015] In the video decoding method according to various embodiments, the size of the pixel group may also be determined based on the minimum value of the height and width of the current block.

[0016] In various embodiments of the video decoding method, the horizontal or vertical displacement vector per unit time for the current pixel group is also a displacement vector per unit time determined using a value determined based on pixel values ​​and gradient values ​​of a first corresponding reference pixel and its neighboring pixels included in a first corresponding reference pixel group in a first reference picture corresponding to the current pixel group and a second corresponding reference pixel and its neighboring pixels included in a second corresponding reference pixel group in a second reference picture, a first POC (picture order count) difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture.

[0017] In the video decoding method according to various embodiments, determining a displacement vector per unit time in a horizontal or vertical direction of pixels of the current block comprises: When the first corresponding reference pixel or the second corresponding reference pixel is a boundary peripheral pixel adjacent to the inside of the boundary of the first reference block or the second reference block, deriving pixel values ​​and gradient values ​​of pixels located outside the boundary of the first reference block or the second reference block using the pixel values ​​of the boundary peripheral pixels; and determining a displacement vector per unit time in a horizontal or vertical direction of the current pixel based on pixel values ​​and gradient values ​​of the boundary-periphery pixels and pixel values ​​and gradient values ​​of pixels located outside the boundary of the current block derived using the pixel values ​​of the boundary-periphery pixels.

[0018] In the video decoding method according to various embodiments, determining a displacement vector per unit time in a horizontal or vertical direction of pixels of the current block comprises: calculating a value related to the current pixel using a pixel value of a first corresponding reference pixel included in the first reference block, a pixel value of a second corresponding reference pixel included in the second reference block, a gradient value of the first corresponding reference pixel, a gradient value of the second corresponding reference pixel, a first POC difference between the first reference picture and a current picture, and a second POC difference between the second reference picture and a current picture; calculating values ​​related to the surrounding pixels calculated using pixel values ​​of first corresponding surrounding pixels of the first corresponding reference pixel, pixel values ​​of second corresponding surrounding pixels of the second corresponding reference pixel, gradient values ​​of the first corresponding surrounding pixels, gradient values ​​of the second corresponding surrounding pixels, a first POC difference between the first reference picture and a current picture, and a second POC difference between the second reference picture and a current picture; calculating a weighted average value for the current pixel, which is necessary to calculate a displacement vector per unit time in a horizontal or vertical direction, using a value for the current pixel, a value for the neighboring pixels, and a weight; The method may further include determining a displacement vector per unit time in the horizontal or vertical direction of the current pixel by using the calculated weighted average value for the current pixel.

[0019] In various embodiments of the video decoding method, the weighted average value for the current pixel may be calculated by applying an exponential smoothing technique to values ​​related to pixels included in the first reference block and the second reference block in the up, down, left, and right directions.

[0020] A video decoding apparatus according to various embodiments includes: acquiring motion prediction mode information for a current block in a current picture from a bitstream; an acquiring unit that acquires, when the acquired motion prediction mode information indicates a bidirectional motion prediction mode, information about a first motion vector indicating a first reference block of a current block in a first reference picture and information about a second motion vector indicating a second reference block of the current block in a second reference picture from the bitstream, and acquires information about a residual block of the current block from the bitstream; an inter prediction unit that determines a horizontal or vertical displacement vector per unit time of pixels of a current block including neighboring pixels inside a boundary of the current block by using values ​​related to reference pixels included in the first and second reference blocks without using values ​​stored for pixels located outside boundaries of the first and second reference blocks, and performs block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to a current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel, thereby obtaining a prediction block of the current block; a decoding unit that reconstructs the current block based on the predicted block and the residual block, The pixel group may include at least one pixel.

[0021] In various embodiments of the video decoding apparatus, the inter prediction unit applying a horizontal or vertical gradient filter to a pixel value of a fractional position pixel in the first reference block or the second reference block to calculate a horizontal or vertical gradient value of the first corresponding reference pixel or the second corresponding reference pixel; the gradient filter is a 5-tap filter; The fractional position pixel is also a pixel in which at least one of the horizontal and vertical components of the coordinates indicating the pixel position has a fractional value.

[0022] In various embodiments of the video decoding apparatus, the inter prediction unit The horizontal or vertical displacement vector per unit time for the current pixel group is also a displacement vector per unit time determined using a value determined based on pixel values ​​and gradient values ​​of a first corresponding reference pixel and its surrounding pixels included in a first corresponding reference pixel group in a first reference picture corresponding to the current pixel group, and a second corresponding reference pixel and its surrounding pixels included in a second corresponding reference pixel group in a second reference picture, a first POC difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture.

[0023] In various embodiments of the video decoding apparatus, the inter prediction unit calculating a value related to the current pixel using a pixel value of a first corresponding reference pixel included in the first reference block, a pixel value of a second corresponding reference pixel included in the second reference block, a gradient value of the first corresponding reference pixel, a gradient value of the second corresponding reference pixel, a first POC difference between the first reference picture and a current picture, and a second POC difference between the second reference picture and a current picture; calculating values ​​related to the neighboring pixels calculated using pixel values ​​of first corresponding neighboring pixels of the first corresponding reference pixel, pixel values ​​of second corresponding neighboring pixels of the second corresponding reference pixel, gradient values ​​of the first corresponding neighboring pixels, gradient values ​​of the second corresponding neighboring pixels, a first POC difference between the first reference picture and a current picture, and a second POC difference between the second reference picture and a current picture; calculating a weighted average value for the current pixel required to calculate a displacement vector per unit time in a horizontal or vertical direction using a value for the current pixel, a value for the neighboring pixels, and a weight; The calculated weighted average value for the current pixel can be used to determine a displacement vector per unit time in the horizontal or vertical direction of the current pixel.

[0024] According to various embodiments, a video encoding method includes: performing block-based motion compensation and pixel group-based motion compensation on a current block to obtain a prediction block, a first motion vector, and a second motion vector of the current block; generating a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether a motion prediction mode for the current block is a bidirectional motion prediction mode; the pixel group includes at least one pixel; the first motion vector is a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block; the second motion vector is a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block; A prediction block of the current block is obtained by performing block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel; The horizontal or vertical displacement vector per unit time of pixels of the current block, including adjacent pixels inside the boundary of the current block, can be determined using values ​​related to reference pixels included in the first reference block and the second reference block, without using values ​​stored for pixels located outside the boundary of the first reference block and the second reference block.

[0025] A video encoding apparatus according to various embodiments includes an inter prediction unit that performs block-based motion compensation and pixel group-based motion compensation on a current block to obtain a prediction block, a first motion vector, and a second motion vector of the current block; a bitstream generator configured to generate a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether a motion prediction mode for the current block is a bidirectional motion prediction mode, the pixel group includes at least one pixel; the first motion vector is a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block, and the second motion vector is a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block; A prediction block of the current block is obtained by performing block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel; The horizontal or vertical displacement vector per unit time of pixels of the current block, including adjacent pixels inside the boundary of the current block, can be determined using values ​​related to reference pixels included in the first reference block and the second reference block, without using values ​​stored for pixels located outside the boundary of the first reference block and the second reference block. The present invention may also include a computer-readable recording medium having a program recorded thereon for implementing the methods according to various embodiments. [Effects of the Invention]

[0026] According to various embodiments, in a bidirectional motion prediction mode, inter-prediction for a current block is performed using gradient values ​​of a reference block of a reference picture, and a value similar to the value of the original block of the current block is predicted, thereby improving encoding / decoding efficiency. [Brief explanation of the drawings]

[0027] [Figure 1A] 1 is a block diagram of a video decoding device according to various embodiments. [Figure 1B] 1 is a flowchart of a video decoding method according to various embodiments. [Figure 1C] 1 is a block diagram of a video encoding device according to various embodiments. [Figure 1D] 1 is a flowchart of a video encoding method according to various embodiments. [Figure 1E] 3A and 3B are block diagrams of a video decoder according to various embodiments. [Figure 1F] 3A and 3B are block diagrams of a video decoder according to various embodiments. [Figure 2] 1 is a reference diagram for explaining a process of block-based bidirectional motion prediction and compensation thereof according to an embodiment; [Figure 3A] 1 is a reference diagram for explaining a process of performing pixel-based motion compensation according to an embodiment; [Figure 3B] 1 is a reference diagram for explaining a process of performing pixel-based motion compensation according to an embodiment; [Figure 3C] 1 is a reference diagram for explaining a process of performing pixel-based motion compensation according to an embodiment; [Figure 4] 1 is a reference diagram for explaining a process of calculating horizontal and vertical gradient values ​​according to an embodiment. [Figure 5] 10 is a reference diagram for explaining a process of calculating gradient values ​​in the horizontal and vertical directions according to another embodiment; [Figure 6A] 10 is a diagram illustrating a process of determining horizontal and vertical gradient values ​​using a one-dimensional filter according to an embodiment; [Figure 6B] 10 is a diagram illustrating a process of determining horizontal and vertical gradient values ​​using a one-dimensional filter according to an embodiment; [Figure 7A] 1 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values, according to one embodiment. [Figure 7B] 1 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values, according to one embodiment. [Figure 7C] 1 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values, according to one embodiment. [Figure 7D] 1 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values, according to one embodiment. [Figure 7E] 1 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values, according to one embodiment. [Figure 8A] 10 is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel according to an embodiment. [Figure 8B] 10 is a reference diagram illustrating a process of determining horizontal and vertical displacement vectors for a pixel group according to an embodiment; [Figure 8C]10 is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel according to an embodiment; [Figure 8D] 10 is a reference diagram illustrating a process of determining horizontal and vertical displacement vectors for a pixel group according to an embodiment; [Figure 9A] 10 is a diagram illustrating a process of determining a gradient value in the horizontal or vertical direction by adding an offset and performing inverse scaling after filtering, according to an embodiment; [Figure 9B] 10 is a diagram illustrating a process of determining a gradient value in a horizontal or vertical direction by adding an offset and performing inverse scaling after filtering according to another embodiment. [Figure 9C] 10 is a diagram illustrating an input range required to determine a horizontal displacement vector and a vertical displacement vector in a process of performing pixel-wise motion compensation on a current block. [Figure 9D] 10 is a diagram illustrating an input range of a region used in a process of performing pixel-based motion compensation according to various embodiments; [Figure 9E] 10 is a diagram illustrating an input range of a region used in a process of performing pixel-based motion compensation according to various embodiments; [Figure 9F] 10 is a diagram illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block. [Figure 9G] 10 is a diagram illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block according to another embodiment. [Figure 9H] 10 is a diagram illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block according to another embodiment. [Figure 9I]10 is a diagram illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block according to another embodiment. [Figure 9J] This is a diagram illustrating a process of determining horizontal and vertical displacement vectors for each pixel by applying exponential smoothing techniques to the top, bottom, left, and right, without extending the block, in one embodiment, by referring to pixel values ​​and gradient values ​​of a reference block. [Figure 9K] 10 is a diagram illustrating a process of determining pixel values ​​of reference pixels in a reference block and horizontal and vertical gradient values ​​using a filter to perform motion compensation for a current block, according to an embodiment. [Figure 9L] 10 is a diagram illustrating a process of determining pixel values ​​of reference pixels in a reference block and horizontal and vertical gradient values ​​using a filter to perform motion compensation for a current block according to another embodiment. [Figure 10] 10 is a diagram illustrating a process of dividing a current coding unit and determining at least one coding unit according to an embodiment. [Figure 11] 10 is a diagram illustrating a process of dividing a non-square coding unit and determining at least one coding unit according to an embodiment. [Figure 12] 10 is a diagram illustrating a process of dividing a coding unit based on at least one of block shape information and division shape information, according to an embodiment; [Figure 13] 1 is a diagram illustrating a method for determining a predetermined coding unit from among an odd number of coding units, according to an embodiment; [Figure 14] 10 is a diagram illustrating an order in which a plurality of coding units are processed when a current coding unit is divided and a plurality of coding units are determined, according to an embodiment; [Figure 15]10 is a diagram illustrating a process of determining whether a current coding unit is divided into an odd number of coding units when coding units cannot be processed in a predetermined order, according to an embodiment; [Figure 16] 10 is a diagram illustrating a process of dividing a first coding unit and determining at least one coding unit according to an embodiment. [Figure 17] 10 is a diagram illustrating that, according to one embodiment, when a non-square second coding unit determined by dividing a first coding unit satisfies a predetermined condition, the manner in which the second coding unit may be divided is restricted. [Figure 18] 10 is a diagram illustrating a process of dividing a square-shaped coding unit when division form information does not indicate that the coding unit is divided into four square-shaped coding units, according to an embodiment. [Figure 19] 10 is a diagram illustrating that the processing order of a plurality of coding units may vary depending on the division process of the coding units, according to an embodiment. [Figure 20] 10 is a diagram illustrating a process of determining the depth of a coding unit according to changes in the shape and size of the coding unit when the coding unit is recursively divided to determine multiple coding units, according to one embodiment. [Figure 21] 10 is a diagram illustrating a part index (PID) for distinguishing a depth and a coding unit, which is determined depending on the type and size of the coding unit, according to an embodiment. [Figure 22] 1 is a diagram illustrating a plurality of coding units being determined based on a plurality of predetermined data units included in a picture, according to an embodiment; [Figure 23] 10 is a diagram illustrating processing blocks that are used as a reference for determining a determination order of reference coding units included in a picture, according to an embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0028] According to various embodiments, a video decoding method includes the steps of: obtaining motion prediction mode information for a current block in a current picture from a bitstream; If the acquired motion prediction mode information indicates a bi-directional motion prediction mode, acquiring information about a first motion vector indicating a first reference block of a current block in a first reference picture and information about a second motion vector indicating a second reference block of the current block in a second reference picture from the bitstream; determining a displacement vector per unit time in a horizontal or vertical direction of pixels of a current block including adjacent pixels inside a boundary of the current block by using values ​​related to reference pixels included in the first reference block and the second reference block without using values ​​stored for pixels located outside the boundaries of the first reference block and the second reference block; obtaining a prediction block of the current block by performing block-wise motion compensation and pixel group-wise motion compensation of the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel; obtaining information about a residual block of a current block from the bitstream; reconstructing the current block based on the predicted block and the residual block; The pixel group may include at least one pixel.

[0029] A video decoding apparatus according to various embodiments includes: acquiring motion prediction mode information for a current block in a current picture from a bitstream; an acquiring unit that acquires, when the acquired motion prediction mode information indicates a bidirectional motion prediction mode, information about a first motion vector indicating a first reference block of a current block in a first reference picture and information about a second motion vector indicating a second reference block of the current block in a second reference picture from the bitstream, and acquires information about a residual block of the current block from the bitstream; an inter prediction unit that determines a horizontal or vertical displacement vector per unit time of pixels of a current block including neighboring pixels inside a boundary of the current block by using values ​​related to reference pixels included in the first and second reference blocks without using values ​​stored for pixels located outside boundaries of the first and second reference blocks, and performs block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to a current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel, thereby obtaining a prediction block of the current block; a decoding unit that reconstructs the current block based on the predicted block and the residual block, The pixel group may include at least one pixel.

[0030] According to various embodiments, a video encoding method includes: performing block-based motion compensation and pixel group-based motion compensation on a current block to obtain a prediction block, a first motion vector, and a second motion vector of the current block; generating a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether a motion prediction mode for the current block is a bidirectional motion prediction mode; the pixel group includes at least one pixel; the first motion vector is a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block; the second motion vector is a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block; A prediction block of the current block is obtained by performing block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel; The horizontal or vertical displacement vector per unit time of pixels of the current block, including adjacent pixels inside the boundary of the current block, can be determined using values ​​related to reference pixels included in the first reference block and the second reference block, without using values ​​stored for pixels located outside the boundary of the first reference block and the second reference block.

[0031] A video encoding apparatus according to various embodiments includes an inter prediction unit that performs block-based motion compensation and pixel group-based motion compensation on a current block to obtain a prediction block, a first motion vector, and a second motion vector of the current block; a bitstream generator configured to generate a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether a motion prediction mode for the current block is a bidirectional motion prediction mode, the pixel group includes at least one pixel; the first motion vector is a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block, and the second motion vector is a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block; A prediction block of the current block is obtained by performing block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel; The horizontal or vertical displacement vector per unit time of pixels of the current block, including adjacent pixels inside the boundary of the current block, can be determined using values ​​related to reference pixels included in the first reference block and the second reference block, without using values ​​stored for pixels located outside the boundary of the first reference block and the second reference block.

[0032] The present invention may also include a computer-readable recording medium having a program recorded thereon for implementing the methods according to various embodiments.

[0033] Hereinafter, "image" can refer to still or moving video images, i.e., the video itself.

[0034] Hereinafter, a "sample" refers to data assigned to a sampling position in an image and to data to be processed. For example, in a spatial domain image, a pixel is also a sample.

[0035] Hereinafter, the term "current block" refers to a block of an image to be encoded or decoded.

[0036] FIG. 1A illustrates a block diagram of a video decoding device according to various embodiments.

[0037] A video decoding device 100 according to various embodiments includes an acquisition unit 105, an inter prediction unit 110, and a reconstruction unit 125.

[0038] The obtaining unit 105 receives a bitstream including information about the prediction mode of the current block, information indicating the motion prediction mode of the current block, and information about the motion vector.

[0039] The acquiring unit 105 may acquire, from the received bitstream, information about the prediction mode of the current block, information indicating the motion prediction mode of the current block, and information about the motion vector. The acquiring unit 105 may also acquire, from the received bitstream, a reference picture index indicating a reference picture among previously decoded pictures.

[0040] The inter prediction unit 110 performs inter prediction on the current block when the prediction mode of the current block is the inter prediction mode. That is, the inter prediction unit 110 may generate a predicted pixel value of the current block using at least one of pictures decoded before the current picture including the current block. For example, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the inter prediction unit 110 may generate a predicted pixel value of the current block using two pictures decoded before the current picture. That is, when information about the motion prediction mode acquired from the bitstream indicates a bidirectional motion prediction mode, the inter prediction unit 110 may generate a predicted pixel value of the current block using two pictures decoded before the current picture.

[0041] The inter prediction unit 110 may include a block-based motion compensation unit 115 and a pixel group-based motion compensation unit 120 .

[0042] The block-based motion compensation unit 115 can perform block-based motion compensation for the current block.

[0043] The block-based motion compensation unit 115 may determine at least one reference picture from among previously decoded pictures using reference picture indexes acquired from the bitstream. Here, the reference picture indexes refer to reference picture indexes associated with each prediction direction, including the L0 direction and the L1 direction. Here, the reference picture index associated with the L0 direction refers to an index indicating a reference picture from among pictures included in the L0 reference picture list, and the reference picture index associated with the L1 direction refers to an index indicating a reference picture from among pictures included in the L1 reference picture list.

[0044] The block-based motion compensation unit 115 may determine a reference block for a current block located in at least one reference picture using information about a motion vector received from a bitstream. Here, a corresponding block in a reference picture corresponding to a current block in the current picture is also the reference block. That is, the block-based motion compensation unit 115 may determine a reference block for a current block using a motion vector indicating a reference block from the current block. Here, the motion vector refers to a vector indicating a displacement between the reference coordinates of the current block in the current picture and the reference coordinates of the reference block in the reference picture. For example, if the upper left coordinate of the current block is (1,1) and the upper left coordinate of the reference block in the reference picture is (3,3), the motion vector is also (2,2).

[0045] Here, the information about the motion vector may include a motion vector differential value, and the block-based motion compensation unit 115 may reconstruct the motion vector using a motion vector predictor and the motion vector differential value acquired from the bitstream, and determine a reference block for the current block located in at least one reference picture using the reconstructed motion vector. Here, the motion vector differential value refers to a motion vector differential value related to a reference picture associated with each prediction direction including an L0 direction and an L1 direction. Here, the motion vector differential value related to the L0 direction refers to a motion vector differential value indicating a reference block in a reference picture included in an L0 reference picture list, and the motion vector differential value related to the L1 direction refers to a motion vector differential value indicating a reference block in a reference picture included in an L1 reference picture list.

[0046] The block-based motion compensation unit 115 may perform block-based motion compensation for a current block using pixel values ​​of a reference block. The block-based motion compensation unit 115 may perform block-based motion compensation for a current block using reference pixel values ​​in a reference block corresponding to current pixels in the current block. Here, the reference pixels are pixels included in the reference block, and corresponding pixels in the current block corresponding to current pixels are also reference pixels.

[0047] The block-based motion compensation unit 115 may perform block-based motion compensation on the current block using a plurality of reference blocks included in a plurality of reference pictures, respectively. For example, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the block-based motion compensation unit 115 may determine two reference pictures from among previously coded pictures and determine two reference blocks included in the two pictures.

[0048] The block-based motion compensation unit 115 may perform block-based motion compensation for the current block using pixel values ​​of two reference pixels in two reference blocks, and may perform block-based motion compensation for the current block using an average value or a weighted sum of pixel values ​​of the two reference pixels to generate a block-based motion compensation value.

[0049] The reference position of the reference block may be an integer pixel position, but is not limited thereto, and may also be a fractional pixel position. Here, an integer pixel is a pixel whose position component is an integer and is located at an integer pixel position. A fractional pixel is a pixel whose position component is a decimal and is located at a fractional pixel position.

[0050] For example, if the upper left coordinate of the current block is (1,1) and the motion vector is (2.5,2.5), the upper left coordinate of the reference block in the reference picture is also (3.5,3.5). In this case, the position of the fractional pixel can be determined in 1 / 4 pel (pixel element) units or 1 / 16 pel units. The position of the fractional pixel can also be determined in various fractional pel units.

[0051] When the reference position of the reference block is a decimal pixel position, the block-based motion compensation unit 115 can apply an interpolation filter to a first surrounding area including a first pixel among the pixels of the first reference block indicated by the first motion vector, and a second surrounding area including a second pixel among the pixels of the second reference block indicated by the second motion vector, to generate a pixel value of the first pixel and a pixel value of the second pixel.

[0052] That is, the reference pixel value in the reference block is determined using the pixel values ​​of surrounding pixels whose components in a predetermined direction are integers, where the predetermined direction can be either the horizontal direction or the vertical direction.

[0053] For example, the block-based motion compensation unit 115 may use an interpolation filter to filter pixel values ​​of pixels whose directional components are integers, determine reference pixel values ​​using the resulting values, and then determine a block-based motion compensation value for the current block using the reference pixel values. The block-based motion compensation value may also be determined using an average or weighted sum of reference pixels. The interpolation filter may be a DCT (discrete cosine transformation)-based M-tap interpolation filter. The coefficients of the DCT-based M-tap interpolation filter are also derived from the DCT and the IDCT (inverse discrete cosine transform). The coefficients of the interpolation filter may be scaled to integer coefficients to reduce real-number operations during filtering. The interpolation filter may also be a one-dimensional interpolation filter in the horizontal or vertical direction. For example, when expressing pixel positions using x and y Cartesian coordinate components, the horizontal direction refers to the direction parallel to the x-axis, and the vertical direction refers to the direction parallel to the y-axis.

[0054] The block-based motion compensation unit 115 first uses a one-dimensional vertical interpolation filter to filter pixel values ​​at integer positions, and then uses a one-dimensional horizontal interpolation filter to filter the values ​​generated by the filtering to determine reference pixel values ​​at fractional pixel positions.

[0055] On the other hand, when scaled filter coefficients are used, the values ​​generated by filtering are larger than those generated by non-scaled filter coefficients. Therefore, the block-based motion compensation unit 115 can perform de-scaling on the values ​​generated by filtering.

[0056] The block-based motion compensation unit 115 may perform inverse scaling after filtering pixel values ​​at integer positions using a vertical one-dimensional interpolation filter. In this case, the inverse scaling may include bit shifting to the right by the inverse scaling bit count. The inverse scaling bit count may also be determined based on the bit depth of the input image samples. For example, the inverse scaling bit count may be a value obtained by subtracting 8 from the bit depth of the samples.

[0057] In addition, the block-based motion compensation unit 115 may filter pixel values ​​at integer positions using a vertical one-dimensional interpolation filter, filter the filtered values ​​using a horizontal one-dimensional interpolation filter, and then perform inverse scaling. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits is also determined based on the number of scaling bits of the vertical one-dimensional interpolation filter, the number of scaling bits of the horizontal one-dimensional interpolation filter, and the bit depth of the sample. For example, if the number of scaling bits p of the vertical one-dimensional interpolation filter is 6, the number of scaling bits q of the horizontal one-dimensional interpolation filter is 6, and the bit depth of the sample is b, the number of inverse scaling bits is p+q+8-b, or 20-b.

[0058] If the block-based motion compensation unit 115 performs filtering on pixels whose components in a predetermined direction are integers using a one-dimensional interpolation filter and then performs only bit shifting to the right by the number of inverse scaling bits, a rounding error may occur. Therefore, after performing filtering on pixels whose components in a predetermined direction are integers using a one-dimensional interpolation filter, an offset may be added and then inverse scaling may be performed. In this case, the offset may be 2^(number of inverse scaling bits - 1).

[0059] The pixel group-based motion compensation unit 120 may perform motion compensation for a pixel group of a current block and generate a motion compensation value for a pixel group. When the motion prediction mode of the current block is a bidirectional motion prediction mode, the pixel group-based motion compensation unit 120 may perform motion compensation for a pixel group of the current block and generate a motion compensation value for a pixel group.

[0060] The pixel group-based motion compensation unit 120 may perform pixel group-based motion compensation for the current block based on the optical flow of pixel groups of the first and second reference pictures, and generate pixel group-based motion compensation values. The optical flow will be described later in connection with FIG. 3A.

[0061] The pixel group-based motion compensation unit 120 may perform pixel group-based motion compensation on pixel groups included in a reference block of a current block to generate pixel group-based motion compensation values. The pixel group may include at least one pixel. For example, the pixel group may be a single pixel. Alternatively, the pixel group may be a plurality of pixels including two or more pixels. The pixel group may be a plurality of pixels included in a block of size KxK (K is an integer).

[0062] The pixel group unit motion compensation unit 120 may determine a pixel group and perform pixel group unit motion compensation for the current block based on the determined pixel group.

[0063] The pixel group-based motion compensation unit 120 may determine the size of a pixel group based on the size of the current block. For example, the pixel group-based motion compensation unit 120 may determine the maximum value of 2 and a value obtained by dividing the minimum value of the height and width of the current block by 8 as the height and width of the pixel group.

[0064] The pixel group-based motion compensation unit 120 performs motion compensation in units of pixel groups each including a plurality of pixels, thereby enabling a reduction in encoding / decoding complexity compared to when motion compensation is performed in units of pixels from a high video resolution. Also, the pixel group-based motion compensation unit 120 performs motion compensation in units of pixel groups each including a plurality of pixels, thereby enabling a reduction in encoding / decoding complexity compared to when motion compensation is performed in units of pixels from a high frame rate.

[0065] The acquiring unit 105 can acquire information about the size of a pixel group included in a bitstream. The information about the size of a pixel group is also information indicating a height or width K when the size of the pixel group is KxK. The information about the size of the pixel group is also included in a high level syntax carrier.

[0066] The pixel group-based motion compensation unit 120 may determine at least one pixel group partition including pixels having similar pixel values ​​among a plurality of pixels included in a pixel group, and perform motion compensation on the pixel group partition. In this case, since pixel group partitions including pixels having similar pixel values ​​are likely to represent the same object and have similar motions, the pixel group-based motion compensation unit 120 may perform more detailed pixel group-based motion compensation.

[0067] On the other hand, motion compensation in units of pixel groups is performed when the motion prediction mode information indicates a bidirectional motion prediction mode, but even in this case, it is not always performed but is selectively performed.

[0068] The pixel group-based motion compensation unit 120 may determine a reference pixel group in a reference block corresponding to a current pixel group of a current block and determine a gradient value of the reference pixel group. For example, the pixel group-based motion compensation unit 120 may determine the gradient value of the reference pixel group using the gradient value of at least one pixel value included in the reference pixel group.

[0069] The pixel group unit motion compensation unit 120 may perform pixel group unit motion compensation for the current block using the gradient values ​​of the reference pixel group, thereby generating pixel group unit motion compensation values.

[0070] The pixel group-based motion compensation unit 120 can apply a filter to a first peripheral area of ​​a first pixel group including a first pixel group among pixel groups of a first reference block indicated by a first motion vector, and a second peripheral area of ​​a second pixel group including a second pixel group among pixel groups of a second reference block indicated by a second motion vector, to generate gradient values ​​of the first pixel group and gradient values ​​of the second pixel group.

[0071] The pixel group-based motion compensation unit 120 can determine pixel values ​​and pixel gradient values ​​of pixels within a first window of a predetermined size including a first pixel group in a first reference picture, centered on the first pixel group, and can determine pixel values ​​and pixel gradient values ​​of pixels within a second window of a predetermined size including the second reference pixels, centered on a second reference pixel group in a second reference picture.

[0072] The pixel group-based motion compensation unit 120 may determine a displacement vector per unit time for the current pixel group using pixel values ​​and gradient values ​​of pixels in the first window and pixel values ​​and gradient values ​​of pixels in the second window. The displacement vector per unit time for the current pixel group may be adjusted according to a regularization parameter. The regularization parameter is a parameter introduced to prevent errors from occurring when an ill-posed displacement vector per unit time for the current pixel group is determined for motion compensation of the pixel group. The pixel group-based motion compensation unit 120 may perform motion compensation for the current block based on an adjustment parameter for the displacement vector per unit time in the horizontal or vertical direction. The regularization parameter will be described later in the description of FIG. 8A.

[0073] The pixel group unit motion compensation unit 120 may perform pixel group unit motion compensation for the current block using a displacement vector per unit time for the current pixel group and a gradient value of a reference pixel.

[0074] The reference position of the reference block may be, but is not limited to, an integer pixel position, and may also be a fractional pixel position.

[0075] When the reference position of the reference block is a sub-pixel position, the gradient value of the reference pixel in the reference block is also determined using pixel values ​​of surrounding pixels whose predetermined directional components are integers.

[0076] For example, the pixel group-based motion compensation unit 120 may use a gradient filter to filter pixel values ​​of surrounding pixels whose predetermined directional components are integers, and determine the gradient value of the reference pixel using the resulting value. The filter coefficients of the gradient filter may be determined using predetermined coefficients for a DCT-based interpolation filter. The coefficients of the gradient filter may also be filter coefficients scaled to integer coefficients to reduce real-number operations during filtering.

[0077] In this case, the gradient filter is also a one-dimensional gradient filter in the horizontal or vertical direction.

[0078] The pixel group unit motion compensation unit 120 can use a horizontal or vertical one-dimensional gradient filter to determine a horizontal or vertical gradient value for a reference pixel, and perform filtering on surrounding pixels whose corresponding directional components are integers.

[0079] For example, the pixel group unit motion compensation unit 120 can use a horizontal one-dimensional gradient filter to filter pixels located horizontally from pixels located around the reference pixel whose horizontal component is an integer, and determine a horizontal gradient value related to the reference pixel.

[0080] If the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are decimals), the pixel group unit motion compensation unit 120 performs filtering using a vertical one-dimensional interpolation filter on the pixel at the (x, y) position and pixels located vertically from the pixel at the (x, y) position whose vertical components are integers, and can determine the pixel value of (x, y+β) using the resulting value.

[0081] The pixel group unit motion compensation unit 120 filters the pixel value at the (x, y+β) position and pixels located horizontally from the (x, y+β) position whose horizontal components are integers using a horizontal gradient filter, and can determine the horizontal gradient value at the (x+α, y+β) position using the resulting value.

[0082] The order in which the one-dimensional gradient filter and the one-dimensional interpolation filter are used is not limited. In the above description, a vertical interpolation filter is first used to perform filtering on pixels at integer positions, a vertical interpolation filtered value is generated, and a one-dimensional horizontal gradient filter is used to perform filtering on the vertical interpolation filtered value. However, it is also possible to first perform filtering on pixels at integer positions using a one-dimensional horizontal gradient filter, a horizontal interpolation filtered value is generated, and a one-dimensional vertical interpolation filter is used to perform filtering on the horizontal interpolation filtered value.

[0083] The above has been a detailed description of how the pixel group-based motion compensation unit 120 determines the horizontal gradient value at the (x+α, y+β) position. The pixel group-based motion compensation unit 120 also determines the vertical gradient value at the (x+α, y+β) position in a similar manner to the horizontal gradient value, so a detailed description will be omitted.

[0084] The pixel group-based motion compensation unit 230 has been described above in detail for using a one-dimensional gradient filter and a one-dimensional interpolation filter to determine gradient values ​​at fractional pixel positions. However, the present invention is not limited to these. A gradient filter and an interpolation filter may also be used to determine gradient values ​​at integer pixel positions. For integer pixels, pixel values ​​are determined without using an interpolation filter. However, for consistent processing with fractional pixel processing, an interpolation filter may be used to filter integer pixels and neighboring pixels whose predetermined directional components are integers to determine the integer pixel values. For example, the interpolation filter coefficients for integer pixels may be {0,0,64,0,0}. Because the interpolation filter coefficients for neighboring integer pixels are 0, filtering is performed using only the pixel value of the current integer pixel. As a result, the interpolation filter may be used to filter the current integer pixel and neighboring integer pixels to determine the pixel value of the current integer pixel.

[0085] The pixel group unit motion compensation unit 120 may perform inverse scaling after filtering pixels at integer positions using a vertical one-dimensional interpolation filter. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits may be determined based on the bit depth of the sample. The number of inverse scaling bits may also be determined based on specific input data within the block.

[0086] For example, the bit shifting number is the bit depth of the sample minus 8.

[0087] The pixel group-based motion compensation unit 120 may perform inverse scaling after filtering the value generated by the inverse scaling using a horizontal gradient filter. Similarly, in this case, the inverse scaling may include bit shifting to the right by the inverse scaling bit count. The inverse scaling bit count is also determined based on the scaled bit count of the vertical one-dimensional interpolation filter, the scaled bit count of the horizontal one-dimensional gradient filter, and the bit depth of the sample. For example, if the scaling bit count p of the one-dimensional interpolation filter is 6, the scaling bit count q of the one-dimensional gradient filter is 4, and the bit depth of the sample is b, the inverse scaling bit count is p+q+8-b, which is also 18-b.

[0088] If the pixel group unit motion compensation unit 120 performs filtering and then performs only bit shifting to the right by the number of inverse scaling bits on the value generated by the filtering, a rounding error may occur. Therefore, an offset may be added to the value generated by the filtering and then inverse scaling may be performed. In this case, the offset may be 2^(number of inverse scaling bits - 1).

[0089] The inter prediction unit 110 may generate a predicted pixel value of the current block using a block-based motion compensation value related to the current block and a pixel group-based motion compensation value. For example, the inter prediction unit 110 may generate a predicted pixel value of the current block by combining a block-based motion compensation value related to the current block and a pixel group-based motion compensation value. Here, the block-based motion compensation value refers to a value generated by performing block-based motion compensation, and the pixel group-based motion compensation value refers to a value generated by performing pixel group-based motion compensation. The block-based motion compensation value may be an average value or a weighted sum of reference pixels, and the pixel group-based motion compensation value may be a value determined based on a displacement vector per unit time for the current pixel and a gradient value of the reference pixels.

[0090] The pixel group-based motion compensation unit 120 may acquire a shift value for descaling after the interpolation or gradient operation based on at least one of the bit depth of the sample, the input range of the filter used for the interpolation or gradient operation, and the coefficients of the filter. The pixel group-based motion compensation unit 120 may perform descaling after the interpolation or gradient operation on pixels included in the first reference block and the second reference block using the shift value for descaling.

[0091] When performing block-based motion compensation, the inter prediction unit 110 may use motion vectors and store the motion vectors. In this case, the motion vector unit may be a 4x4 block. Meanwhile, when storing motion vectors after block-based motion compensation, the motion vector storage unit may be a block of various sizes other than 4x4 (e.g., an RxR block, where R is an integer). In this case, the motion vector storage unit may be a block larger than 4x4, such as a 16x16 block.

[0092] Meanwhile, when performing motion compensation in units of pixel groups, the size of the target block for performing motion compensation in units of pixel groups is also expanded based on the size of the current block as well as the window size and the interpolation filter length. The reason why the size of the target block is expanded from the size of the current block based on the window size is because, in the case of a pixel located at an edge of the current block, a window is used to perform motion compensation in units of pixel groups for the current block based on the pixel located at the current edge and surrounding pixels.

[0093] Therefore, in order to reduce the number of memory accesses and the number of multiplication operations, the pixel group-based motion compensation unit 120 adjusts the positions of pixels within the window that are outside the current block to the positions of adjacent pixels within the current block in the process of performing pixel group-based motion compensation using a window, thereby reducing the number of memory accesses and the number of multiplication operations.

[0094] The pixel group-based motion compensation unit 120 does not use pixel values ​​of integer-position pixels to determine gradient values ​​of reference pixels, which are values ​​required for pixel group-based motion compensation. That is, the pixel group-based compensation unit 120 may apply a horizontal or vertical gradient filter to pixel values ​​of fractional-position pixels to calculate horizontal or vertical gradient values ​​of a first corresponding reference pixel in a first reference block or a second corresponding reference pixel in a second reference block. In this case, the gradient filter length may be 5. In this case, the filter coefficients may be symmetrical about a center coefficient of the filter. A fractional-position pixel is also a pixel in which at least one of the horizontal and vertical components indicating the pixel's position has a fractional value.

[0095] The pixel values ​​of the sub-position pixels in the first reference block or the second reference block are also calculated by applying a horizontal or vertical interpolation filter to the pixel values ​​of the integer position pixels.

[0096] The horizontal or vertical displacement vector per unit time for the current pixel group is also a displacement vector per unit time determined using a value determined based on the pixel values ​​and gradient values ​​of the first corresponding reference pixel included in the first corresponding reference pixel group in the first reference picture corresponding to the current pixel group, the second corresponding reference pixel included in the second corresponding reference pixel group in the second reference picture, and their surrounding pixels, a first POC (picture order count) difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture.

[0097] When the first corresponding reference pixel or the second corresponding reference pixel is a boundary-neighboring pixel adjacent to the inside of the boundary of the first reference block or the second reference block, the pixel group unit compensation unit 120 can derive the pixel value of the pixel located outside the boundary of the first reference block or the second reference block using the pixel value of the boundary-neighboring pixel.

[0098] The pixel group unit compensation unit 120 may determine a displacement vector per unit time in the horizontal or vertical direction of the current block based on pixel values ​​of the boundary-neighboring pixels and pixel values ​​of pixels located outside the boundary of the current block derived using the pixel values ​​of the boundary-neighboring pixels. That is, there are pixels located outside the boundary among pixels included in a window centered on the boundary-neighboring pixels, and in this case, the pixel values ​​and gradient values ​​of the pixels located outside the boundary are also pixel values ​​and gradient values ​​of pixels derived from the boundary-neighboring pixels that are not values ​​stored in memory.

[0099] The pixel group unit compensation unit 120 may calculate a value for the current pixel using a pixel value of a first corresponding reference pixel included in a first reference block, a pixel value of a second corresponding reference pixel included in a second reference block, a gradient value of the first corresponding reference pixel, a gradient value of the second corresponding reference pixel, a first POC difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture. That is, the value for the current pixel is also a result of a function based on the pixel values ​​and gradient values ​​of corresponding reference pixels of each reference picture and the POC differences between each reference picture and the current picture.

[0100] The pixel group unit compensation unit 120 may calculate values ​​related to the surrounding pixels using the pixel values ​​of the first corresponding surrounding pixels of the first corresponding reference pixel, the gradient values ​​of the first corresponding surrounding pixels, the pixel values ​​of the second corresponding surrounding pixels of the second corresponding reference pixel, the gradient values ​​of the second corresponding surrounding pixels, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture. That is, the values ​​related to the surrounding pixels are also the result of a function based on the pixel values ​​and gradient values ​​of the corresponding reference pixels of each reference picture and the POC difference between each reference picture and the current picture. That is, the values ​​related to the surrounding pixels are also the result of a function based on the pixel values ​​and gradient values ​​of the corresponding surrounding pixels of each reference picture and the POC difference between each reference picture and the current picture.

[0101] The pixel group unit compensation unit 120 may calculate a weighted average value for the current pixel, which is required to calculate a horizontal displacement vector per unit time, using the value associated with the current pixel, the value associated with the surrounding pixels, and a weight value. In this case, the weight value may be determined based on the distance between the current pixel and the surrounding pixels, the distance between the pixel and the block boundary, the number of pixels located outside the boundary, or whether the pixel is located inside or outside the boundary.

[0102] The weighted average value for the current pixel is calculated by applying exponential smoothing to the values ​​of pixels included in the first and second reference blocks in the up, down, left, and right directions. By applying exponential smoothing to the values ​​of pixels in the up, down, left, and right directions, the value calculated for the current pixel has the largest weight for the value of the current pixel, and the weights for the values ​​of the surrounding pixels decrease exponentially according to the distance from the current pixel.

[0103] The pixel group unit compensation unit 120 may determine a displacement vector per unit time in the horizontal or vertical direction of the current pixel using the weighted average value for the current pixel.

[0104] The reconstruction unit 125 may obtain a residual block of the current block from the bitstream and reconstruct the current block using the residual block and a predicted block of the current block. For example, the reconstruction unit 125 may combine pixel values ​​of the residual block of the current block and pixel values ​​of the predicted block of the current block from the bitstream to generate pixel values ​​of a reconstructed block.

[0105] The video decoding device 100 may include a video decoding unit (not shown), which may include an obtaining unit 105, an inter predicting unit 110, and a reconstruction unit 125. The video decoding unit will be described with reference to FIG. 1E.

[0106] FIG. 1B illustrates a flowchart of a video decoding method according to various embodiments.

[0107] In operation S105, the video decoding apparatus 100 may acquire motion prediction mode information for a current block in a current picture from a bitstream. The video decoding apparatus 100 may receive a bitstream including motion prediction mode information for a current block in a current picture and acquire the motion prediction mode information for the current block from the received bitstream. The video decoding apparatus 100 may acquire information about the prediction mode of the current block from the bitstream and determine the prediction mode of the current block based on the information about the prediction mode of the current block. In this case, if the prediction mode of the current block is an inter prediction mode, the video decoding apparatus 100 may acquire motion prediction mode information for the current block.

[0108] For example, the video decoding apparatus 100 may determine the prediction mode of the current block to be the inter prediction mode based on information about the prediction mode of the current block. If the prediction mode of the current block is the inter prediction mode, the video decoding apparatus 100 may obtain motion prediction mode information related to the current block from the bitstream.

[0109] In step S110, if the motion prediction mode information indicates a bidirectional motion prediction mode, the video decoding device 100 can obtain from the bitstream a first motion vector indicating a first reference block of the current block in the first reference picture and a second motion vector indicating a second reference block of the current block in the second reference picture.

[0110] That is, the video decoding device 100 may receive a bitstream including information about the first and second motion vectors, and may obtain the first and second motion vectors from the received bitstream. The video decoding device 100 may obtain reference picture indexes from the bitstream, and may determine the first and second reference pictures from among previously decoded pictures based on the reference picture indexes.

[0111] In operation S115, the video decoding apparatus 100 may determine a horizontal or vertical displacement vector of a pixel of the current block, including an adjacent pixel within the boundary of the current block, by using values ​​associated with reference pixels included in the first and second reference blocks, without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks. In this case, the values ​​stored for pixels located outside the boundaries of the first and second reference blocks and the values ​​associated with reference pixels included in the first and second reference blocks may be pixel values ​​of associated pixels, or horizontal gradient values ​​or vertical gradient values ​​of associated pixels. Alternatively, the values ​​stored for pixels located outside the boundaries of the first and second reference blocks and the values ​​associated with reference pixels included in the first and second reference blocks may be values ​​determined using pixel values ​​or gradient values ​​of associated pixels.

[0112] In step S120, the video decoding device 100 may perform block-based motion compensation and pixel group-based motion compensation for the current block using the horizontal or vertical gradient value of a first corresponding reference pixel in a first reference block corresponding to a current pixel included in a current pixel group in the current block, the horizontal or vertical gradient value of a second corresponding reference pixel in a second reference block, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector of the current pixel, to obtain a prediction block for the current block.

[0113] That is, the video decoding apparatus 100 may perform block-based motion compensation and pixel group-based motion compensation on the current block based on the first and second motion vectors, and generate a prediction block for the current block. The video decoding apparatus 100 may perform block-based motion compensation on the current block using pixel values ​​of a first reference block indicated by the first motion vector and pixel values ​​of a second reference block indicated by the second motion vector. The video decoding apparatus 100 may also perform pixel group-based motion compensation on the current pixel group using a horizontal or vertical gradient value of at least one first corresponding reference pixel in the first reference block corresponding to at least one pixel included in the current pixel group, a horizontal or vertical gradient value of at least one second corresponding reference pixel in the second reference block, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector of the current pixel.

[0114] The video decoding device 100 can obtain a prediction block for the current block by using a block-based motion compensation value generated by performing block-based motion compensation on the current block and a pixel group-based motion compensation value generated by performing pixel group-based motion compensation on the current pixel group.

[0115] In operation S125, the video decoding apparatus 100 may obtain information about the residual block of the current block from the bitstream.

[0116] In operation S130, the video decoding apparatus 100 may reconstruct the current block based on the predicted block and the residual block. That is, the video decoding apparatus 100 may generate pixel values ​​of a reconstructed block of the current block by combining pixel values ​​of the residual block indicated by the residual block related to the current block with predicted pixel values ​​of the predicted block.

[0117] FIG. 1C illustrates a block diagram of a video encoding device according to various embodiments.

[0118] A video encoding device 150 according to various embodiments includes an inter-prediction unit 155 and a bitstream generation unit 170 .

[0119] The inter prediction unit 155 performs inter prediction on the current block by referring to various blocks based on a rate and a distortion cost. That is, the inter prediction unit 155 may generate a predicted pixel value of the current block by using at least one of pictures coded before the current picture including the current block.

[0120] The inter prediction unit 155 may include a block-based motion compensation unit 160 and a pixel group-based motion compensation unit 165 .

[0121] The block-based motion compensation unit 160 may perform block-based motion compensation on the current block to generate a block-based motion compensation value.

[0122] The block-based motion compensation unit 160 may determine at least one reference picture from among previously decoded pictures, and determine a reference block of the current block located within the at least one reference picture.

[0123] The block-based motion compensation unit 160 may perform block-based motion compensation for a current block using pixel values ​​of a reference block to generate block-based motion compensation values. The block-based motion compensation unit 160 may perform block-based motion compensation for a current block using reference pixel values ​​of a reference block corresponding to a current pixel of the current block to generate block-based motion compensation values.

[0124] The block-based motion compensation unit 160 may perform block-based motion compensation for a current block using a plurality of reference blocks included in a plurality of reference pictures, respectively, to generate block-based motion compensation values. For example, when the motion prediction mode of the current block is a bidirectional prediction mode, the block-based motion compensation unit 160 may determine two reference pictures from previously coded pictures and determine two reference blocks included in the two pictures. Here, the bidirectional prediction is not limited to performing inter-prediction using a picture that is earlier in display order than the current picture and a picture that is later in display order than the current picture, but may also mean performing inter-prediction using two pictures coded earlier than the current picture, regardless of display order.

[0125] The block-based motion compensation unit 160 may perform block-based motion compensation for the current block using two reference pixel values ​​in two reference blocks to generate a block-based motion compensation value. The block-based motion compensation unit 160 may perform block-based motion compensation for the current block using an average pixel value or a weighted sum of two reference pixels to generate a block-based motion compensation value.

[0126] The block-based motion compensation unit 160 can output a reference picture index indicating a reference picture for motion compensation of the current block among previously coded pictures.

[0127] The block-based motion compensation unit 160 may determine a motion vector that starts from a current block and ends at a reference block of the current block, and output the motion vector. The motion vector indicates a displacement between the reference coordinates of the current block in the current picture and the reference coordinates of the reference block in the reference picture. For example, if the coordinates of the upper left corner of the current block are (1,1) and the coordinates of the upper left corner of the reference block in the reference picture are (3,3), the motion vector is also (2,2).

[0128] The reference position of the reference block may be, but is not limited to, an integer pixel position, or a fractional pixel position. In this case, the fractional pixel position may be determined in 1 / 4-pel units or 1 / 16-pel units. However, the reference position of the reference block may be determined in various fractional pel units.

[0129] For example, if the reference position of the reference block is (1.5, 1.5) and the coordinate of the upper left corner of the current block is (1, 1), the motion vector is also (0.5, 0.5). If the motion vector is determined to be in 1 / 4-pel or 1 / 16-pel units to indicate the reference position of the reference block, which is a fractional pixel position, the motion vector can be scaled to determine an integer motion vector, and the upscaled motion vector can be used to determine the reference position of the reference block. If the reference position of the reference block is a fractional pixel position, the position of the reference pixel of the reference block is also a fractional pixel position. Therefore, in the reference block, the pixel value of the fractional pixel position can also be determined using the pixel values ​​of surrounding pixels whose predetermined directional components are integers.

[0130] For example, the block-based motion compensation unit 160 may use an interpolation filter to perform filtering on pixel values ​​of neighboring pixels whose predetermined directional components are integers, determine a reference pixel value at a sub-pixel position using the resulting value, and determine a block-based motion compensation value for the current block using the pixel value of the reference pixel. The interpolation filter may be a DCT-based M-tap interpolation filter. The coefficients of the DCT-based M-tap interpolation filter may be derived from the DCT and IDCT. The coefficients of the interpolation filter may also be scaled to integer coefficients to reduce real-number operations during filtering.

[0131] In this case, the interpolation filter is also a one-dimensional interpolation filter in the horizontal or vertical direction.

[0132] The block-based motion compensation unit 160 may first filter surrounding integer pixels using a vertical one-dimensional interpolation filter, and then filter the filtered values ​​using a horizontal one-dimensional interpolation filter to determine reference pixel values ​​at sub-pixel positions. When using scaled filter coefficients, the block-based motion compensation unit 160 may filter pixels at integer positions using a vertical one-dimensional interpolation filter, and then inversely scale the filtered values. In this case, the inverse scaling may include bit-shifting to the right by the inverse scaling bit count. The inverse scaling bit count may also be determined based on the bit depth of the sample. For example, the bit-shifting count may be a value obtained by subtracting 8 from the bit depth of the sample.

[0133] The block-based motion compensation unit 160 may also perform filtering on pixels whose horizontal components are integers using a horizontal one-dimensional interpolation filter, followed by bit-shifting to the right by the number of inverse scaling bits, which is determined based on the number of scaled bits for the vertical one-dimensional interpolation filter coefficients, the number of scaled bits for the horizontal one-dimensional interpolation filter coefficients, and the bit depth of the samples.

[0134] If the block-based motion compensation unit 160 only performs bit shifting to the right by the number of inverse scaling bits, a rounding error may occur. Therefore, it may use a one-dimensional interpolation filter in a predetermined direction to filter pixels whose components in the predetermined direction are integers, add an offset to the filtered value, and then perform inverse scaling on the value to which the offset has been added. In this case, the offset is 2^(number of inverse scaling bits - 1).

[0135] Although it has been described above that the number of inverse scaling bits after filtering using a vertical one-dimensional interpolation filter is determined based on the bit depth of the samples, the number of inverse scaling bits may be determined by taking into consideration not only the bit depth of the samples but also the number of scaled bits for the interpolation filter coefficients. That is, the number of inverse scaling bits may be determined based on the bit depth of the samples and the number of scaled bits for the interpolation filter coefficients, within a range where no overflow occurs, taking into consideration the size of the registers used when filtering is performed and the size of the buffer that stores the values ​​generated during the filtering process.

[0136] The pixel group-based motion compensation unit 165 may perform motion compensation on a pixel group basis for the current block and generate a motion compensation value on a pixel group basis. For example, when the motion prediction mode is a bidirectional motion prediction mode, the pixel group-based motion compensation unit 165 may perform motion compensation on a pixel group basis for the current block and generate a motion compensation value on a pixel group basis.

[0137] The pixel group unit motion compensation unit 165 can perform pixel group unit motion compensation for the current block using gradient values ​​of pixels included in the reference block of the current block, and generate pixel group unit motion compensation values.

[0138] The pixel group unit motion compensation unit 165 can apply a filter to a first peripheral area of ​​a first pixel among pixels of a first reference block in a first reference picture and a second peripheral area of ​​a second pixel among pixels of a second reference block in a second reference picture, thereby generating a gradient value of the first pixel and a gradient value of the second pixel.

[0139] The pixel group-based motion compensation unit 165 may determine pixel values ​​and pixel gradient values ​​of pixels in a first window of a predetermined size including a first reference pixel centered on a first reference pixel in a first reference picture, and may determine pixel values ​​and pixel gradient values ​​of pixels in a second window of a predetermined size including a second reference pixel centered on a second reference pixel in a second reference picture. The pixel group-based motion compensation unit 165 may determine a displacement vector per unit time for a current pixel using the pixel values ​​and pixel gradient values ​​of pixels in the first window and the pixel values ​​and pixel gradient values ​​of pixels in the second window.

[0140] The pixel group unit motion compensation unit 165 can perform pixel group unit motion compensation for the current block using a displacement vector per unit time and a gradient value of a reference pixel, and generate a pixel group unit motion compensation value.

[0141] The reference pixel location may be, but is not limited to, an integer pixel location, and may also be a fractional pixel location.

[0142] When the reference position of the reference block is a sub-pixel position, the gradient value of the reference pixel in the reference block is also determined using pixel values ​​of surrounding pixels whose predetermined directional components are integers.

[0143] For example, the pixel group unit motion compensation unit 165 may filter pixel values ​​of neighboring pixels whose predetermined directional components are integers using a gradient filter, and determine the gradient value of the reference pixel using the resulting value. In this case, the filter coefficients of the gradient filter may be determined using coefficients determined in advance for a DCT-based interpolation filter.

[0144] The coefficients of the gradient filter are also filter coefficients scaled to integer coefficients to reduce real-number operations during filtering. In this case, the gradient filter used may be a one-dimensional gradient filter in the horizontal or vertical direction.

[0145] The pixel group unit motion compensation unit 165 can use a horizontal or vertical one-dimensional gradient filter to determine a horizontal or vertical gradient value for a reference pixel, and perform filtering on surrounding pixels whose corresponding directional components are integers.

[0146] For example, the pixel group unit motion compensation unit 165 can use a vertical one-dimensional interpolation filter to filter pixels whose vertical components are integers among the pixels in the vertical direction from the integer pixels adjacent to the reference pixel, and determine the pixel values ​​of pixels whose vertical components are decimals.

[0147] The pixel group unit motion compensation unit 165 may also determine pixel values ​​at sub-pixel positions in other columns adjacent to the integer pixel adjacent to the reference pixel by filtering the vertically surrounding integer pixels using a vertical one-dimensional interpolation filter, where the positions of the pixels in the other columns are sub-pixel positions in the vertical direction and integer pixel positions in the horizontal direction.

[0148] That is, when the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are decimals), the pixel group unit motion compensation unit 165 can determine the pixel value at the (x, y+β) position by filtering the vertically surrounding integer pixels from the (x, y) position using a vertical interpolation filter.

[0149] The pixel group unit motion compensation unit 165 can determine the horizontal gradient value at the (x+α, y+β) position by filtering the pixel values ​​of pixels located horizontally whose horizontal components are integers using a horizontal gradient filter from the pixel values ​​at the (x, y+β) position and the pixel values ​​at the (x, y+β) position.

[0150] The order in which the one-dimensional gradient filter and the one-dimensional interpolation filter are used is not limited. As described above, a vertical interpolation filter may be used first to perform filtering on pixels at integer positions to generate vertical interpolation filtered values, and a one-dimensional horizontal gradient filter may be used to perform filtering on the vertical interpolation filtered values. However, the order is not limited thereto. For example, a one-dimensional horizontal gradient filter may be used first to perform filtering on pixels at integer positions to generate horizontal gradient filtered values, and a one-dimensional vertical interpolation filter may be used to perform filtering on the horizontal gradient filtered values.

[0151] The above has described in detail how the pixel group unit motion compensation unit 165 determines the horizontal gradient value at the (x+α, y+β) position.

[0152] The pixel group unit motion compensation unit 165 may determine the vertical gradient value at the (x+α, y+β) position in a manner similar to that in which the horizontal gradient value is determined.

[0153] The pixel group-based motion compensation unit 165 may determine a vertical gradient value for the reference pixel by filtering integer pixels surrounding the reference pixel in the vertical direction from the integer pixels surrounding the reference pixel using a vertical one-dimensional gradient filter. The pixel group-based motion compensation unit 165 may also determine a vertical gradient value for pixels located in other columns adjacent to the reference pixel by filtering integer pixels in the vertical direction using a vertical one-dimensional gradient filter. Here, the pixel positions are fractional pixel positions in the vertical direction and integer pixel positions in the horizontal direction.

[0154] That is, when the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are decimals), the pixel group unit motion compensation unit 165 can determine the vertical gradient value at the (x, y+β) position by filtering the surrounding integer pixels in the vertical direction from the (x, y) position using a vertical gradient filter.

[0155] The pixel group unit motion compensation unit 165 can determine the vertical gradient value at the (x+α, y+β) position by filtering the gradient value at the (x, y+β) position and the gradient values ​​of the surrounding integer pixels located horizontally from the (x, y+β) position using a horizontal interpolation filter.

[0156] The order in which the one-dimensional gradient filter and the one-dimensional interpolation filter are used is not limited. As described above, a vertical gradient filter may be used first to perform filtering on pixels at integer positions to generate vertical gradient filtered values, and a one-dimensional horizontal interpolation filter may be used to perform filtering on the vertical gradient filtered values. However, the order is not limited thereto. A one-dimensional horizontal interpolation filter may be used first to perform filtering on pixels at integer positions to generate horizontal interpolation filtered values, and a one-dimensional vertical gradient filter may be used to perform filtering on the horizontal interpolation filtered values.

[0157] The above has described in detail how the pixel group unit motion compensation unit 165 uses gradient filters and interpolation filters to determine gradient values ​​at fractional pixel positions. However, the present invention is not limited to this, and gradient filters and interpolation filters can also be used to determine gradient values ​​at integer pixel positions.

[0158] In the case of integer pixels, their pixel values ​​can be determined without using an interpolation filter, but for processing consistent with fractional pixel processing, filtering can be performed on the integer pixels and surrounding integer pixels using an interpolation filter. For example, the interpolation filter coefficients for integer pixels are {0,0,64,0,0}, and since the interpolation filter coefficients multiplied by the surrounding integer pixels are 0, filtering is performed using only the pixel value of the current integer pixel. As a result, the value generated by filtering the current integer pixel and surrounding integer pixels using the interpolation filter can be determined to be the same as the pixel value of the current integer pixel.

[0159] On the other hand, when using scaled filter coefficients, the pixel group-based motion compensation unit 165 may perform filtering on pixels at integer positions using a horizontal one-dimensional gradient filter and then perform inverse scaling on the filtered values. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits may also be determined based on the bit depth of the sample. For example, the number of inverse scaling bits may be a value obtained by subtracting about 8 from the bit depth of the sample.

[0160] The pixel group unit motion compensation unit 165 may perform inverse scaling after filtering pixels whose vertical components are integers using a vertical interpolation filter. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits is also determined based on the number of scaled bits of the one-dimensional interpolation filter in the vertical direction, the number of scaled bits of the one-dimensional gradient filter in the horizontal direction, and the bit depth of the sample.

[0161] If the pixel group-based motion compensation unit 165 only performs bit shifting to the right by the number of inverse scaling bits, a rounding error may occur. Therefore, after filtering is performed using a one-dimensional interpolation filter, an offset is added to the filtered value, and inverse scaling is performed on the value to which the offset is added. In this case, the offset is 2^(number of bit shifting - 1).

[0162] When performing block-based motion compensation, the inter prediction unit 110 may use motion vectors and store the motion vectors. In this case, the motion vector unit may be a 4x4 block. Meanwhile, when storing motion vectors after block-based motion compensation, the motion vector storage unit may be a block of various sizes other than 4x4 (e.g., an RxR block, where R is an integer). In this case, the motion vector storage unit may be a block larger than 4x4, such as a 16x16 block.

[0163] Meanwhile, when performing motion compensation in units of pixel groups, the size of the target block for performing motion compensation in units of pixel groups may be expanded based on the size of the current block as well as the window size and the interpolation filter length. The reason why the size of the target block is expanded from the size of the current block based on the window size is that, in the case of a pixel located at an edge of the current block, a window is used to perform motion compensation in units of pixel groups related to the current block based on the pixel located at the current edge and surrounding pixels.

[0164] Therefore, in order to reduce the number of memory accesses and the number of multiplication operations, the pixel group-based motion compensation unit 120 adjusts the positions of pixels within the window that are outside the current block to the positions of adjacent pixels within the current block in the process of performing pixel group-based motion compensation using a window, thereby reducing the number of memory accesses and the number of multiplication operations.

[0165] The pixel group-based motion compensation unit 120 does not use pixel values ​​of integer-position pixels to determine gradient values ​​of reference pixels, which are values ​​required for pixel group-based motion compensation. That is, the pixel group-based compensation unit 120 may apply a horizontal or vertical gradient filter to pixel values ​​of fractional-position pixels to calculate horizontal or vertical gradient values ​​of a first corresponding reference pixel in a first reference block or a second corresponding reference pixel in a second reference block. In this case, the gradient filter length may be 5. In this case, the filter coefficients may be symmetrical about a center coefficient of the filter. A fractional-position pixel is also a pixel in which at least one of the horizontal and vertical components indicating the pixel's position has a fractional value.

[0166] The pixel values ​​of the sub-position pixels in the first reference block or the second reference block are also calculated by applying a horizontal or vertical interpolation filter to the pixel values ​​of the integer position pixels.

[0167] The horizontal or vertical displacement vector per unit time for the current pixel group is also a displacement vector per unit time determined using values ​​determined based on the pixel values ​​and gradient values ​​of the first corresponding reference pixel included in the first corresponding reference pixel group in the first reference picture corresponding to the current pixel group, the second corresponding reference pixel included in the second corresponding reference pixel group in the second reference picture, and their surrounding pixels, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture.

[0168] When the first corresponding reference pixel or the second corresponding reference pixel is a boundary-neighboring pixel adjacent to the inside of the boundary of the first reference block or the second reference block, the pixel group unit compensation unit 120 can derive the pixel value of the pixel located outside the boundary of the first reference block or the second reference block using the pixel value of the boundary-neighboring pixel.

[0169] The pixel group unit compensation unit 120 may determine a displacement vector per unit time in the horizontal or vertical direction of the current block based on pixel values ​​of the boundary-neighboring pixels and pixel values ​​of pixels located outside the boundary of the current block derived using the pixel values ​​of the boundary-neighboring pixels. That is, there are pixels located outside the boundary among pixels included in a window centered on the boundary-neighboring pixels, and in this case, the pixel values ​​and gradient values ​​of the pixels located outside the boundary are also pixel values ​​and gradient values ​​of pixels derived from the boundary-neighboring pixels that are not values ​​stored in memory.

[0170] The pixel group unit compensation unit 120 may calculate a value for the current pixel using a pixel value of a first corresponding reference pixel included in a first reference block, a pixel value of a second corresponding reference pixel included in a second reference block, a gradient value of the first corresponding reference pixel, a gradient value of the second corresponding reference pixel, a first POC difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture. That is, the value for the current pixel is also a result of a function based on the pixel values ​​and gradient values ​​of corresponding reference pixels of each reference picture and the POC differences between each reference picture and the current picture.

[0171] The pixel group unit compensation unit 120 may calculate values ​​related to corresponding surrounding pixels calculated using pixel values ​​of first corresponding surrounding pixels of a first corresponding reference pixel, gradient values ​​of the first corresponding surrounding pixels, pixel values ​​of second corresponding surrounding pixels of a second corresponding reference pixel, gradient values ​​of the second corresponding surrounding pixels, a first POC difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture. That is, the values ​​related to surrounding pixels are also the result of a function based on pixel values ​​and gradient values ​​of corresponding reference pixels of each reference picture and the POC difference between each reference picture and the current picture. That is, the values ​​related to corresponding surrounding pixels are also the result of a function based on pixel values ​​and gradient values ​​of corresponding surrounding pixels of each reference picture and the POC difference between each reference picture and the current picture.

[0172] The pixel group unit compensation unit 120 may calculate a weighted average value for the current pixel, which is required to calculate a horizontal displacement vector per unit time, using a value related to the current pixel, a value related to the corresponding surrounding pixel, and a weight value. In this case, the weight value may be determined based on the distance between the current pixel and the surrounding pixel, the distance between the pixel and the block boundary, the number of pixels located outside the boundary, or whether the pixel is located inside or outside the boundary.

[0173] The weighted average value for the current pixel is also a value calculated by applying exponential smoothing to the values ​​of pixels included in the first and second reference blocks in the up, down, left, and right directions. By applying exponential smoothing to the values ​​of pixels in the up, down, left, and right directions, the value calculated for the current pixel is such that the weighted value for the current pixel is the largest, and the weighted values ​​for the values ​​of the surrounding pixels are exponentially decreased according to the distance from the current pixel.

[0174] The pixel group unit compensation unit 120 may determine a displacement vector per unit time in the horizontal or vertical direction of the current pixel using the weighted average value for the current pixel.

[0175] The inter prediction unit 155 may generate a predicted pixel value of the current block using a block-based motion compensation value related to the current block and a pixel group-based motion compensation value related to the current block. For example, the inter prediction unit 155 may generate a predicted pixel value of the current block by combining a block-based motion compensation value related to the current block and a pixel group-based motion compensation value related to the current block. In particular, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the inter prediction unit 155 may generate a predicted pixel value of the current block using a block-based motion compensation value related to the current block and a pixel group-based motion compensation value related to the current block.

[0176] When the motion prediction mode of the current block is a unidirectional motion prediction mode, the inter prediction unit 155 may generate a predicted pixel value of the current block by using a block-based motion compensation value related to the current block. Here, unidirectional means using one reference picture among previously coded pictures. The one reference picture is not limited to being a picture that precedes the current picture in display order, but may also be a picture that follows the current picture.

[0177] The inter predictor 155 may determine a motion prediction mode of a current block and output information indicating the motion prediction mode of the current block. For example, the inter predictor 155 may determine a bidirectional motion prediction mode as the motion prediction mode of the current block and output information indicating the bidirectional motion prediction mode. Here, the bidirectional motion prediction mode refers to a mode in which motion prediction is performed using reference blocks in two decoded reference pictures.

[0178] The bitstream generation unit 170 may generate a bitstream including a motion vector indicating a reference block. The bitstream generation unit 170 may encode the motion vector indicating the reference block and generate a bitstream including the encoded motion vector. The bitstream generation unit 170 may encode a vector difference value of the motion vector indicating the reference block and generate a bitstream including the encoded motion vector difference value. Here, the motion vector difference value refers to a difference value between a motion vector and a motion vector predictor. Here, the motion vector difference value refers to a motion vector difference value related to a reference picture associated with each prediction direction including the L0 direction and the L1 direction. Here, the motion vector difference value related to the L0 direction refers to a motion vector difference value indicating a reference block in a reference picture included in the L0 reference picture list, and the motion vector difference value related to the L1 direction refers to a motion vector difference value indicating a reference block in a reference picture included in the L1 reference picture list.

[0179] In addition, the bitstream generator 170 may generate a bitstream further including information indicating a motion prediction mode of the current block. The bitstream generator 170 may encode reference picture indexes indicating reference pictures of the current block among previously encoded pictures, and generate a bitstream including the encoded reference picture indexes. In this case, the reference picture indexes refer to reference picture indexes associated with each prediction direction, including the L0 direction and the L1 direction. Here, the reference picture index associated with the L0 direction refers to an index indicating a reference picture among pictures included in the L0 reference picture list, and the reference picture index associated with the L1 direction refers to an index indicating a reference picture among pictures included in the L1 reference picture list.

[0180] The video encoding device 150 may include a video encoding unit (not shown), which may include an inter prediction unit 155 and a bitstream generation unit 170. The video encoding unit will be described with reference to FIG. 1F.

[0181] FIG. 1D illustrates a flowchart of a video encoding method according to various embodiments.

[0182] Referring to FIG. 1D, in step S150, the video encoding device 150 performs motion compensation for the current block and motion compensation for each pixel group, and can obtain a predicted block for the current block, a first motion vector, a second motion vector, and parameters related to motion compensation for each pixel group.

[0183] In operation S155, the video encoding device 150 may generate a bitstream including information about the first and second motion vectors and motion prediction mode information indicating whether the motion prediction mode for the current block is a bidirectional motion prediction mode. Here, the first motion vector is also a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block, and the second motion vector is also a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block.

[0184] The video encoding device 150 may encode a residual block of the current block indicating a difference between pixels of a predicted block of the current block and an original block of the current block, and generate a bitstream further including the encoded residual signal. The video encoding device 150 may encode information about a prediction mode of the current block and a reference picture index, and generate a bitstream further including information about the prediction mode of the encoded current block and the reference picture index. For example, the video encoding device 150 may encode information indicating whether the prediction mode of the current block is an inter prediction mode and a reference picture index indicating at least one picture among previously decoded pictures, and generate a bitstream further including information about the prediction mode of the encoded current block and the reference picture index.

[0185] The video encoding device 150 may perform block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to a current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel. The video encoding device 150 may perform block-based motion compensation and pixel group-based motion compensation for the current block to obtain a prediction block for the current block.

[0186] In this case, the horizontal or vertical displacement vector per unit time of the pixels of the current block including the adjacent pixels inside the boundary of the current block can be determined using values ​​related to the reference pixels included in the first and second reference blocks without using values ​​stored for pixels located outside the boundary of the first and second reference blocks. The values ​​related to the reference pixels included in the first and second reference blocks can be pixel values ​​or gradient values ​​of the reference pixels.

[0187] FIG. 1E illustrates a block diagram of a video decoder 600 according to various embodiments.

[0188] The video decoder 600 according to various embodiments performs the same operations as those performed by the video decoder (not shown) of the video decoding device 100 to encode video data.

[0189] 1E, the entropy decoder 615 parses coded video data to be decoded and coding information required for decoding from the bitstream 605. The coded video data is quantized transform coefficients, and the inverse quantizer 620 and the inverse transformer 625 restore residue data from the quantized transform coefficients.

[0190] The intra prediction unit 640 performs intra prediction for each block. The inter prediction unit 635 performs inter prediction for each block using a reference image obtained from the reconstructed picture buffer 630. The inter prediction unit 635 of FIG. 1E corresponds to the inter prediction unit 110 of FIG. 1A.

[0191] By adding the prediction data and residue data for each block generated by the intra prediction unit 640 or the inter prediction unit 635, spatial domain data for the block of the current image 605 is restored, and the deblocking unit 645 and the SAO performing unit 650 perform loop filtering on the restored spatial domain data to output a filtered restored image 660. In addition, the restored image stored in the restored picture buffer 630 may also be output as a reference image.

[0192] In a decoder (not shown) of the video decoding device 100, the step-by-step operations of the video decoder 600 according to various embodiments may be performed block by block to decode the video data.

[0193] FIG. 1F illustrates a block diagram of a video encoder according to various embodiments.

[0194] The video encoder 700 according to various embodiments performs the same operations as those performed by the video encoder (not shown) of the video encoder 150 to encode video data.

[0195] That is, the intra predictor 720 performs intra prediction for each block of the current image 705, and the inter predictor 715 performs inter prediction for each block using the current image 705 and a reference image acquired from the reconstructed picture buffer 710. Here, the inter predictor 715 of FIG. 1E corresponds to the inter predictor 160 of FIG. 1C.

[0196] Residue data is generated by subtracting prediction data for each block output from the intra prediction unit 720 or the inter prediction unit 715 from data for the block to be encoded in the current image 705. The transform unit 725 and the quantization unit 730 transform and quantize the residue data to output quantized transform coefficients for each block. The inverse quantization unit 745 and the inverse transform unit 750 perform inverse quantization and inverse transform on the quantized transform coefficients to restore spatial domain residue data. The restored spatial domain residue data is added to the prediction data for each block output from the intra prediction unit 720 or the inter prediction unit 715 to restore spatial domain data for the block of the current image 705. The deblocking unit 755 and the SAO performing unit 760 perform in-loop filtering on the restored spatial domain data to generate a filtered restored image. The generated restored image is stored in the restored picture buffer 710. The reconstructed image stored in the reconstructed picture buffer 710 is also used as a reference image for inter-prediction of other images. The entropy coding unit 735 performs entropy coding on the quantized transform coefficients, and the entropy-coded coefficients are also output as a bitstream 740.

[0197] Since the image encoder 700 according to various embodiments is applied to the video encoder 150, the operations of the image encoder 700 according to various embodiments may be performed block by block.

[0198] 2 is a reference diagram illustrating a process of block-based bidirectional motion prediction and compensation thereof according to an embodiment. Referring to FIG. 2, a video encoding device 150 performs bidirectional motion prediction to search for a region in a first reference picture 210 and a second reference picture 220 that is most similar to a current block 201 to be coded in a current picture 200. Here, it is assumed that the first reference picture 210 is a picture before the current picture 200, and the second reference picture 220 is a picture after the current picture 200. As a result of the bidirectional motion prediction, the video encoding device 150 determines a first corresponding region 212 in the first reference picture 210 that is most similar to the current block 201, and a second corresponding region 222 in the second reference picture 220 that is most similar to the current block 201. Here, the first corresponding region and the second corresponding region also serve as reference regions for the current block.

[0199] Then, the video encoding device 150 determines a first motion vector MV1 based on the positional difference between a block 211 at the same position as the current block 201 in the first reference picture 210 and the first corresponding area 212, and determines a second motion vector MV2 based on the positional difference between a block 221 at the same position as the current block 201 in the second reference picture 220 and the second corresponding area 222.

[0200] The video encoding device 150 performs block-based bidirectional motion compensation on the current block 201 using the first motion vector MV1 and the second motion vector MV2.

[0201] For example, if the pixel value located at (i,j) (i,j are integers) in the first reference picture 210 is P0(i,j), the pixel value located at (i,j) in the second reference picture 220 is P1(i,j), MV1 = (MVx1,MVy1), and MV2 = (MVx2,MVy2), the block-wise bidirectional motion compensation value P_BiPredBlock(i,j) of the pixel located at (i,j) in the current block 201 can be calculated as follows: P_BiPredBlock(i,j) = {P0(i + MVx1,j + MVy1) + P1(i + MVx2,j + MVy2)} / 2. In this way, the video encoding device 150 can perform block-based motion compensation on the current block 201 using the average value or weighted sum of pixels in the first corresponding area 212 and the second corresponding area 222 indicated by the first motion vector MV1 and the second motion vector MV2, and generate a block-based motion compensation value.

[0202] 3A to 3C are reference diagrams for explaining a process of performing pixel group unit motion compensation according to an embodiment.

[0203] In FIG. 3A, the first corresponding area 310 and the second corresponding area 320 correspond to the first corresponding area 212 and the second corresponding area 222 in FIG. 2, respectively, and are assumed to have been shifted using bidirectional motion vectors (MV1, MV2) so as to overlap the current block 300.

[0204] In addition, the pixel at the bidirectionally predicted position (i,j) (i, j are integers) of the current block 300 is defined as P(i,j), the first reference pixel value of the first reference picture corresponding to the bidirectionally predicted pixel P(i,j) of the current block 300 is defined as P0(i,j), and the second reference pixel value of the second reference picture corresponding to the bidirectionally predicted pixel P(i,j) of the current block 300 is defined as P1(i,j).

[0205] In other words, the first reference pixel value P0(i,j) is the pixel corresponding to pixel P(i,j) of the current block 300 determined by bidirectional motion vector MV1 indicating the first reference picture, and the pixel value P1(i,j) of the second reference pixel is the pixel corresponding to pixel P(i,j) of the current block 300 determined by bidirectional motion vector MV2 indicating the second reference picture.

[0206] Also, the horizontal gradient value of the first reference pixel is

[0207]

number

[0208]

number

[0209]

number

[0210]

number

[0211] Assuming that there is a small motion determined in the video sequence, the pixel in the first corresponding region 310 of the first reference picture that is most similar to the current pixel P(i,j) bidirectionally motion compensated in pixel group units is not the first reference pixel P0(i,j), but is assumed to be the first displaced reference pixel PA obtained by moving the first reference pixel P0(i,j) by a predetermined displacement vector. Assuming that there is a small motion determined in the video sequence as described above, the pixel in the second corresponding region 320 of the second reference picture that is most similar to the current pixel P(i,j) can be estimated to be the second displaced reference pixel PB obtained by moving the second reference pixel P1(i,j) by a predetermined displacement vector.

[0212] The displacement vector is also composed of the x-axis displacement vector Vx and the y-axis displacement vector Vy described above. Therefore, the pixel group-based motion compensation unit 165 calculates the x-axis displacement vector Vx and the y-axis displacement vector Vy that make up such a displacement vector, and performs motion compensation on a pixel group basis using the calculated vectors.

[0213] Optical flow refers to the pattern of apparent motion of an object or surface induced by the relative motion between an observer (eye or a video capture device such as a camera) and a scene. In a video sequence, optical flow can also be expressed by calculating the motion between frames captured at arbitrary times t and t+Δt. The pixel value located at (x, y) in a frame at time t is also defined as I(x, y, t). In other words, I(x, y, t) is also a value that varies spatiotemporally. When I(x, y, t) is differentiated with respect to time t, it is expressed as the following equation (1).

[0214]

number

[0215]

number

[0216] The pixel group-based motion compensation unit 165 calculates a displacement vector Vx in the x-axis direction and a displacement vector Vy in the y-axis direction according to Equation (2), and performs motion compensation on a pixel group basis using the displacement vectors Vx and Vy. In Equation (2), the pixel value I(x, y, t) is the value of the original signal. Therefore, using the original signal value as is would cause a lot of overhead during encoding. Therefore, the pixel group-based motion compensation unit 165 can calculate the displacement vectors Vx and Vy according to Equation (2) using pixels of the first and second reference pictures determined as a result of block-based bidirectional motion prediction. That is, the pixel group-based motion compensation unit 165 determines the x-axis displacement vector Vx and the y-axis displacement vector Vy that minimize Δ within a predetermined size window (Ωij) centered on the current pixel P(i, j) to be bidirectionally motion compensated and including surrounding pixels. It is most desirable that Δ is 0. However, since there is no x-axis displacement vector Vx or y-axis displacement vector Vy that satisfies the case where Δ is 0 for all pixels in the window (Ωij), the x-axis displacement vector Vx and y-axis displacement vector Vy that minimize Δ are determined. The process of determining the displacement vectors Vx and Vy will be described in detail with reference to FIG. 8A.

[0217] To determine the predicted pixel value of the current pixel, a function P(t) related to t may be determined as shown in Equation (3) below.

[0218]

number

[0219] If the temporal distance from a first reference picture (assuming the first reference picture is located earlier than the current picture) to the current picture is τ0 and the temporal distance from a second reference picture (assuming the second reference picture is located later than the current picture) to the current picture is τ1, the reference pixel value in the first reference picture is equal to P(-τ0) and the reference pixel value in the second reference picture is equal to P(τ1). Hereinafter, for convenience of calculation, it is assumed that τ0 and τ1 are both equal to τ.

[0220] The coefficients of each degree of P(t) are also determined by the following equation (4): where P0(i,j) means the pixel value at the (i,j) position of the first reference picture, and P1(i,j) means the pixel value at the (i,j) position of the second reference picture.

[0221]

number

[0222]

number

[0223]

number

[0224] For convenience, the process of determining a predicted pixel value of a current pixel has been described above when the temporal distance from a first reference picture to the current picture is τ and the temporal distance from a second reference picture to the current picture is also the same as τ. However, if the temporal distance from the first reference picture to the current picture is τ0 and the temporal distance from the second reference picture to the current picture is τ1, the predicted pixel value P(0) of the current pixel can also be determined as shown in Equation 7 below.

[0225]

number

[0226]

number

[0227] For example, as shown in FIG. 3B, a first reference picture including a first corresponding area and a second reference picture including a second corresponding area may both be located temporally before the current picture including the current block in the display order.

[0228] In this case, the predicted pixel value P(0) of the current pixel can be determined by equation (9), in which τ1, which indicates the temporal distance difference between the second reference picture and the current picture in equation (8) derived with reference to Figure 3A, is replaced with -τ1.

[0229]

number

[0230] In this case, the predicted pixel value of the current pixel can be determined by Equation (10), in which τ0, which indicates the temporal distance difference between the first reference picture and the current picture in Equation (8) derived with reference to Figure 3A, is replaced with -τ0.

[0231]

number

[0232] 4 is a reference diagram illustrating a process of calculating horizontal and vertical gradient values ​​according to an embodiment. Referring to FIG. 4, the horizontal gradient value of a first reference pixel P0(i,j) 410 of a first reference picture is

[0233]

number

[0234]

number

[0235]

number

[0236]

number

[0237]

number

[0238] According to one embodiment, instead of calculating the change in pixel value at a sub-pixel position and calculating the gradient value as in Equation (11), a predetermined filter can be used to calculate the gradient value at each reference pixel. The filter coefficients of the predetermined filter can also be determined from the coefficients of the interpolation filter used to calculate the pixel value at the sub-pixel position, taking into account the linearity of the filter.

[0239] FIG. 5 is a reference diagram for explaining the process of calculating gradient values ​​in the horizontal and vertical directions according to another embodiment.

[0240] According to another embodiment, the gradient value may be determined by applying a predetermined filter to pixels of the reference picture. Referring to FIG. 5, the video decoding apparatus 100 calculates a horizontal gradient value for a reference pixel P0 500 by applying a predetermined filter to the left of the reference pixel P0 500. Max 520 pixels, and |M on the right Min A predetermined filter is applied to the pixels 510 to calculate the horizontal gradient value of P0 500. The filter coefficients used are the M values ​​used to determine the window size, as shown in FIGS. 7A to 7D. Max Integer pixels and M Min It is also determined by the α value, which indicates the interpolation position with integer pixels (fractional pixel position). For example, referring to FIG. 7A, M Min is -2 and M Max is 3, and when the pixel is about 1 / 4 away from the reference pixel P0 500, that is, when α=1 / 4, the filter coefficients {4, -17, -36, 60, -15, 4} in the second row of FIG. 7A are applied to the surrounding pixels P-2, P-1, P0, P1, P2, and P3. In this case, the horizontal gradient value of the reference pixel 500 is

[0241]

number

[0242]

number

[0243] 6A and 6B are diagrams illustrating a process of determining horizontal and vertical gradient values ​​using a one-dimensional filter according to an embodiment.

[0244] 6A, in order to determine horizontal gradient values ​​of reference pixels in a reference picture, filtering is performed using a plurality of one-dimensional filters for integer pixels. Pixel group-based motion compensation is additional motion compensation performed after block-based motion compensation. Therefore, in the block-based motion compensation process, the reference position of the reference block of the current block indicated by the motion vector is also a fractional pixel position, and pixel group-based motion compensation is performed on reference pixels in the reference block at fractional pixel positions. Therefore, filtering is performed taking into account determining gradient values ​​of pixels at fractional pixel positions.

[0245] 6A, the video decoding apparatus 100 may first perform filtering on pixels located horizontally or vertically from neighboring integer pixels of a reference pixel in a reference picture using a first one-dimensional filter. Similarly, the video decoding apparatus 100 may perform filtering on neighboring integer pixels located in a different row or column from the reference pixel using the first one-dimensional filter. The video decoding apparatus 100 may generate a horizontal gradient value of the reference pixel by filtering the value generated by the filtering using a second one-dimensional filter.

[0246] For example, if the position of the reference pixel is a fractional pixel position of (x+α, y+β) (x and y are integers, and α and β are fractional), then the horizontal integer pixels (x, y) and (x-1, y), (x+1, y), ..., (x+M Min ,y),(x+M Max ,y)(M Min , M Max are integers), a one-dimensional vertical interpolation filter is used to perform filtering as shown in the following equation (12).

[0247]

number

[0248] That is, the first one-dimensional filter is also an interpolation filter for determining the fractional pixel value in the vertical direction. offset1 is an offset to prevent rounding errors, and shift1 means the number of inverse scaling bits. Temp[i,j+β] means the pixel value at fractional pixel position (i,j+β). Temp[i',j+β] (i' is the pixel value at i+M excluding i) min From i+M max (which is an integer up to ) is also determined by replacing i with i' using equation (12).

[0249] Next, the video decoding device 100 can perform filtering using a second one-dimensional filter on the pixel value at the fractional pixel position (i, j+β) and the pixel value at the fractional pixel position (i', j+β).

[0250]

number

[0251] That is, according to Equation (13), the video decoding device 100 performs filtering using a gradient filter (gradFilterα) on the pixel value (Temp[i,j+β]) at (i,j+β) and the pixel value (Temp[i′,j+β]) located vertically from the pixel position (i,j+β), thereby obtaining a horizontal gradient value at (i+α,j+β).

[0252]

number

[0253] Although the description above has been given of first applying an interpolation filter and then applying a gradient filter to determine gradient values ​​in the horizontal direction, the present invention is not limited thereto, and the gradient filter may be applied first and then an interpolation filter may be applied to determine gradient values ​​in the horizontal direction. In the following, an embodiment will be described in which a gradient filter is applied and then an interpolation filter is applied to determine gradient values ​​in the horizontal direction.

[0254] For example, if the position of the reference pixel is a fractional pixel position of (x+α, y+β) (x and y are integers, and α and β are fractional), then the horizontal integer pixels (x, y) and (x-1, y), (x+1, y), ..., (x+M Min ,y),(x+M Max,y)(M Min , M Mmax and are all integers), filtering is performed using a first one-dimensional filter as shown in the following equation (14).

[0255]

number

[0256] That is, the first one-dimensional filter is also an interpolation filter for determining the horizontal gradient value of a pixel at a position where the horizontal component of the pixel position is a decimal. offset3 is an offset to prevent rounding errors, and shift3 means the number of inverse scaling bits. Temp[i+α,j] means the horizontal gradient value at pixel position (i+α,j). Temp[i+α,j'] (j' is j+M excluding i) min From j+M max (which is an integer up to ) is also determined by replacing j with j' using equation (14).

[0257] Next, the video decoding device 100 can perform filtering on the horizontal gradient value at pixel position (i+α,j) and the horizontal gradient value at pixel position (i+α,j') using a second one-dimensional filter as shown in the following equation (15).

[0258]

number

[0259] That is, according to Equation (15), the video decoding device 100 performs filtering using a gradient filter (fracFilterβ) on the horizontal gradient value (Temp[i+α,j]) at (i+α,j) and the horizontal gradient value (Temp[i+α,j′]) of the pixel located vertically from the pixel position (i+α,j), thereby obtaining the horizontal gradient value (Temp[i+α,j′]) at (i+α,j+β).

[0260]

number

[0261] 6B, in order to determine vertical gradient values ​​of reference pixels in a reference picture, filtering is performed using a plurality of one-dimensional filters for integer pixels. Pixel group-based motion compensation is additional motion compensation performed after block-based motion compensation. Therefore, in the block-based motion compensation process, the reference position of the reference block of the current block indicated by the motion vector may be a sub-pixel position, and pixel group-based motion compensation may be performed on reference pixels in the reference block at sub-pixel positions. Therefore, filtering may be performed taking into account determining gradient values ​​of pixels at sub-pixel positions.

[0262] 6B, first, the video decoding apparatus 100 may perform filtering using a first one-dimensional filter on pixels located horizontally or vertically from neighboring integer pixels of a reference pixel in a reference picture. Similarly, the video decoding apparatus 100 may perform filtering using the first one-dimensional filter on neighboring pixels located in a different column or row from the reference pixel. The video decoding apparatus 100 may generate a vertical gradient value of the reference pixel by filtering the filtered value using a second one-dimensional filter.

[0263] For example, if the position of the reference pixel is a fractional pixel position of (x+α, y+β) (x and y are integers, and α and β are fractional), then the integer pixels (x, y) and (x-1, y-1), (x+1, y+1), ..., (x+M Min ,y+M Min ),(x+M Max ,y+M max ) (MMin and MMmax are both integers) is filtered using a first one-dimensional filter as shown in the following equation (16).

[0264]

number

[0265] That is, the first one-dimensional filter is also an interpolation filter for determining the pixel value at the horizontal fractional pixel position α. ​​offset5 is an offset to prevent rounding errors, and shift5 is the number of inverse scaling bits.

[0266] Temp[i+α,j] means the pixel value at the sub-pixel position (i+α,j). Temp[i+α,j'] (j' is j+M excluding j) min From j+M max (which is an integer up to ) is also determined by replacing j with j' using equation (16).

[0267] Next, the video decoding device 100 can perform filtering on the pixel value at pixel position (i+α,j) and the pixel value at pixel position (i+α,j') using a second one-dimensional filter as shown in the following equation (17).

[0268]

number

[0269] That is, according to Equation (17), the video decoding device 100 applies a gradient filter (gradFilter β ) to filter the vertical gradient value at (i+α,j+β)

[0270]

number

[0271] Although the description above has been given of first applying an interpolation filter and then applying a gradient filter to determine a gradient value in the vertical direction, the present invention is not limited thereto, and the gradient filter may be applied first and then an interpolation filter may be applied to determine a gradient value in the horizontal direction. Hereinafter, an embodiment will be described in which a gradient filter is applied and then an interpolation filter is applied to determine a gradient value in the vertical direction.

[0272] For example, if the position of the reference pixel is a fractional pixel position of (x+α, y+β) (x and y are integers, and α and β are fractional), then the integer pixels (x, y) and (x, y-1), (x, y+1), ..., (x, y+M Min ),(x,y+M max )(M Min , M max and are all integers), filtering is performed using a first one-dimensional filter as shown in the following equation (18).

[0273]

number

[0274] That is, the first one-dimensional filter is also an interpolation filter for determining the vertical gradient value of pixels at positions where the vertical component of the pixel position is a fraction. offset7 is an offset to prevent rounding errors, and shift7 is the number of inverse scaling bits.

[0275] Temp[i,j+β] means the vertical gradient value at pixel position (i,j+β). Temp[i',j+β] (i' is the value of i+M excluding i) min From i+M max (which is an integer up to ) is also determined by replacing i with i' using equation (18).

[0276] Next, the video decoding device 100 can use a second one-dimensional filter to filter the vertical gradient value at pixel position (i, j+β) and the vertical gradient value at pixel position (i', j+β) as shown in the following equation (19).

[0277]

number

[0278] That is, according to Equation (19), the video decoding device 100 calculates an interpolation filter (fracFilter α ) to filter the vertical gradient value at (i+α,j+β)

[0279]

number

[0280] According to one embodiment, the video decoding device 100 determines the horizontal and vertical gradient values ​​at (i+α, j+β) by a combination of the various filters described above. For example, to determine the horizontal gradient value, the first one-dimensional filter may use an interpolation filter for determining vertical pixel values, and the second one-dimensional filter may use a gradient filter for horizontal gradient values. To determine the vertical gradient value, the first one-dimensional filter may use a gradient filter for determining vertical gradient values, and the second one-dimensional filter may use an interpolation filter for determining horizontal pixel values.

[0281] 7A-7E are tables illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values, according to one embodiment.

[0282] 7A and 7B are tables showing filter coefficients for filters for determining horizontal or vertical gradient values ​​at fractional pixel positions in 1 / 4 pel units.

[0283] As described above, a one-dimensional gradient filter and a one-dimensional interpolation filter may be used to determine the gradient values ​​in the horizontal or vertical direction. Referring to FIG. 7A, the filter coefficients of the one-dimensional gradient filter are shown. In this case, a 6-tap filter may be used as the gradient filter. The gradient filter coefficients may also be coefficients scaled by 2^4. M min means the difference between the position of the farthest integer pixel in the negative direction that is applied to the filter and the position of the central integer pixel, and M maxmeans the difference between the position of the farthest integer pixel in the positive direction to which the filter is applied, relative to the central integer pixel, and the position of the central integer pixel. For example, the gradient filter coefficients for determining the horizontal gradient value of a pixel at fractional pixel position α=1 / 4 in the horizontal direction are {4, -17, -36, 60, -15, -4}. The gradient filter coefficients for determining the horizontal gradient value of a pixel at fractional pixel position α=0, 1 / 2, 3 / 4 in the horizontal direction may also be determined with reference to FIG. 7A.

[0284] Referring to FIG. 7B, the filter coefficients of a one-dimensional interpolation filter are shown. In this case, a 6-tap filter may be used as the interpolation filter. The filter coefficients of the interpolation filter may be scaled by 2^6. M min means the difference between the position of the farthest integer pixel in the negative direction that is applied to the filter and the position of the central integer pixel, and M max means the difference between the position of the farthest pixel among integer pixels in the positive direction to which the filter is applied, based on the central integer pixel, and the position of the central integer pixel.

[0285] FIG. 7C is a table showing the filter coefficients of a one-dimensional interpolation filter used to determine pixel values ​​at fractional pixel locations in 1 / 4 pel units.

[0286] As previously mentioned, two identical one-dimensional interpolation filters are also used horizontally and vertically to determine pixel values ​​at fractional pixel locations.

[0287] Referring to FIG. 7C, the filter coefficients of the one-dimensional interpolation filter are shown. At this time, the one-dimensional interpolation filter is also a 6-tap filter. The gradient filter coefficients are also coefficients scaled by 2^6. M min means the difference between the position of the farthest integer pixel in the negative direction that is applied to the filter and the position of the central integer pixel, and Mmax means the difference between the position of the farthest pixel among integer pixels in the positive direction to which the filter is applied, based on the central integer pixel, and the position of the central integer pixel.

[0288] FIG. 7D is a table showing filter coefficients for filters utilized to determine horizontal or vertical gradient values ​​at fractional pixel positions in 1 / 16 pel units.

[0289] As described above, a one-dimensional gradient filter and a one-dimensional interpolation filter may be used to determine gradient values ​​in the horizontal or vertical direction. Referring to FIG. 7D , filter coefficients of a one-dimensional gradient filter are shown. A 6-tap gradient filter may be used. The gradient filter coefficients may be scaled by 2^4. For example, the gradient filter coefficients for determining the horizontal gradient value of a pixel whose sub-pixel position α is 1 / 16 in the horizontal direction may be {8, -32, -13, 50, -18, 5}. The gradient filter coefficients for determining the horizontal gradient values ​​of pixels whose sub-pixel positions α are 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, and 1 / 2 in the horizontal direction may also be determined using FIG. 7D . Meanwhile, the gradient filter coefficients for calculating horizontal gradient values ​​for pixels with sub-pixel positions α of 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, and 15 / 16 can also be determined using the symmetry of the filter coefficients based on α=½. That is, the filter coefficients for the left sub-pixel positions are determined based on α=½ as shown in FIG. 7D, and the filter coefficients for the right sub-pixel positions are determined based on the remaining α=½. For example, the filter coefficients for α=15 / 16 can be determined using the filter coefficients {8, -32, -13, 50, -18, 5} for α=½, which is the symmetrical position based on α=½. That is, the filter coefficients at α=15 / 16 may be determined as {5, −18, 50, −13, −32, 8} by arranging the filter coefficients {8, −32, −13, 50, −18, 5} in reverse order.

[0290] Referring to FIG. 7E, the filter coefficients of a one-dimensional interpolation filter are shown. In this case, a 6-tap filter may be used as the interpolation filter. The filter coefficients of the interpolation filter may also be coefficients scaled by 2^6. For example, the one-dimensional interpolation filter coefficients for calculating the horizontal pixel value of a pixel whose sub-pixel position α is 1 / 16 in the horizontal direction may be {1, -3, 64, 4, -2, 0}. The interpolation filter coefficients for calculating the horizontal pixel values ​​of pixels whose sub-pixel positions α are 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, and 1 / 2 in the horizontal direction may also be determined using FIG. 7E. Meanwhile, the interpolation filter coefficients for calculating the horizontal pixel values ​​of pixels whose sub-pixel positions α are 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, and 15 / 16 in the horizontal direction may also be determined using the symmetry of the filter coefficients based on α=1 / 2. That is, the filter coefficients at the left sub-pixel positions may be determined based on α=½ as shown in FIG. 7E, and the filter coefficients at the right sub-pixel positions may be determined based on the remaining α=½. For example, the filter coefficients at α=15 / 16 may be determined based on the filter coefficients {1, -3, 64, 4, -2, 0} at α=1 / 16, which is the symmetrical position based on α=½. That is, the filter coefficients at α=15 / 16 may be determined as {0, -2, 4, 64, -3, 1} by arranging the filter coefficients {1, -3, 64, 4, -2, 0} in reverse order.

[0291] FIG. 8A is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel according to an embodiment.

[0292] Referring to FIG. 8A, a window (Ωij) 800 of a predetermined size has a size of (2M+1)*(2N+1) (M and N are integers) centered on the bidirectionally predicted pixel P(i,j) in the current block.

[0293] The bidirectionally predicted pixel of the current block in the window is denoted by P(i',j') (where iM≦i'≦i+M and jN≦j'≦j+N, (i',j')∈Ωij), the pixel value of the first reference pixel in the first reference picture 810 corresponding to the bidirectionally predicted pixel P(i',j') of the current block is denoted by P0(i',j'), the pixel value of the second reference pixel in the second reference picture 820 corresponding to the bidirectionally predicted pixel P(i',j') of the current block is denoted by P1(i',j'), and the horizontal gradient value of the first reference pixel is denoted by

[0294]

number

[0295]

number

[0296]

number

[0297]

number

[0298]

number

[0299] The difference value Δi′j′ between the first displacement corresponding pixel PA′ and the second displacement corresponding pixel PB′ can also be determined as shown in the following equation (21).

[0300]

number

[0301]

number

[0302]

number

[0303]

number

[0304]

number

[0305]

number

[0306]

number

[0307] A simplified solution to the equation can be determined by first minimizing the horizontal direction and then minimizing the vertical direction. For example, if we assume that only the horizontal displacement vector is changed, we can assume Vy = 0 in the first equation of Equation (24), and therefore the equation: τVx = s3 / s1 can be determined.

[0308] Furthermore, by rearranging the second equation of Equation (24) using the equation: τVx=s3 / s1, it can be determined as the equation: τVy=(s6-τVx*S2) / s5.

[0309] where the gradient value

[0310]

number

[0311] In the process of calculating Vx(i,j) and Vy(i,j), adjustment parameters r and m are also introduced to prevent multiplication operations from being performed with 0 or very small values.

[0312] For convenience, it is assumed that Vx(i,j) and Vy(i,j) are in the opposite direction to that shown in Fig. 3A. For example, Vx(i,j) and Vy(i,j) derived according to Equation (24) based on the direction of Vx(i,j) and Vy(i,j) shown in Fig. 3A have the same magnitude as Vx(i,j) and Vy(i,j) determined in the opposite direction to that shown in Fig. 3A, with only a difference in sign between their values.

[0313] The first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can also be determined as shown in the following Equation (26): In this case, PA' and PB' can also be determined using the first-order linear term of the local Taylor expansion.

[0314]

number

[0315]

number

[0316] [Number]

[0317] [Number] Φ(Vx, Vy) is a function with Vx and Vy as intermediate variables, and the maximum or minimum value is also determined by calculating the values that become 0 by partially differentiating Φ(Vx, Vy) with respect to Vx and Vy, as shown in the following mathematical formula (30).

[0318] [Number] That is, Vx and Vy are also determined as Vx and Vy that minimize the Φ(Vx, Vy) value. To solve the optimization problem, first minimization is performed in the vertical direction, and then minimization is performed in the horizontal direction. By the minimization, Vx is also determined as shown in the following mathematical formula (31).

[0319] [[ID=3�]][Number] > Here, the clip3(x, y, z) function is a function that outputs x if z < x, y if z > y, and z if x < z < y. According to mathematical formula (31), when s1 + r > m, Vx is clip3(-thBIO, thBIO, -s3 / (s1 + r)), and when s1 + r > m is not satisfied, Vx is also 0.

[0320] By this minimization, Vy is also determined as in the following mathematical formula (32).

[0321]

Number

[0322] At this time, s1, s2, s3, and s5 are also determined as in the following mathematical formula (33). s4 can have the same value as s2.

[0323]

Number

[0324]

Number

[0325] However, without being limited thereto, the values ​​of the adjustment parameters r, m, and thBIO may be determined based on information about the adjustment parameters obtained from the bitstream, which may be included in a slice header, a picture parameter set, a sequence parameter set, or various forms of high-level syntax carriers.

[0326] The adjustment parameters are also determined depending on whether or not temporally different bidirectional prediction is available. For example, when temporally different bidirectional prediction is available, the thBIO diff If bidirectional predictions that are temporally identical to each other are available, then thBIO same Bigger, thBIO diff The size of thBIO same It's also twice the size.

[0327] FIG. 8B is a reference diagram for explaining a process of determining horizontal and vertical displacement vectors for a pixel group according to an embodiment.

[0328] Referring to FIG. 8B, a window (Ωij) 810 of a predetermined size is centered on a KxK pixel group 820 of multiple pixels that are not bidirectionally predicted in the current block, and has a size of (2M+K+1)*(2N+K+1) (M and N are integers).

[0329] At this time, the difference from FIG. 8A is that the window size is larger, but apart from that, the horizontal and vertical displacement vectors for the pixel group can be determined in the same manner.

[0330] FIG. 8C is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel according to an embodiment.

[0331] 8C, the video decoding apparatus 100 can determine a horizontal displacement vector and a vertical displacement vector for each pixel. Accordingly, a displacement vector 835 per unit time for each pixel can be determined. At this time, the horizontal displacement vector Vx[i,j] and the vertical displacement vector Vy[i,j] of the displacement vector 835 for each pixel can also be determined by the following Equation (35). Here, i and j represent the x-coordinate component and the y-coordinate component of the pixel. Here, σ1[i,j], σ2[i,j], σ3[i,j], σ5[i,j], and σ6 [i,j] are also s1, s2, s3, s5 and s6 in Equation (33), respectively.

[0332]

number

[0333] Referring to FIG. 8D, the video decoding device 100 can determine σ1[i,j], σ2[i,j], σ3[i,j], σ5[i,j], and σ6[i,j] for each pixel included in each pixel group 840 as shown in equation (35).

[0334] The video decoding device 100 may determine the horizontal displacement vector Vx[i,j] for the pixel group 840 using σ1[i,j] and σ3[i,j] of the pixel as shown in Equation 36 below, where i,j represent the x and y components of the top left coordinates of the pixel group.

[0335]

number

[0336] The video decoding device 100 may determine the vertical displacement vector Vy[i,j] for the pixel group 840 using the pixels σ2[i,j], σ5[i,j], σ6[i,j], and Vx[i,j] as shown in Equation (37), where the horizontal displacement vector Vx[i,j] is also the value determined by Equation (36).

[0337]

number

[0338] Meanwhile, while the detailed description has been given above with reference to Figure 8D assuming that the pixel group has a size of 2x2, this is not limited thereto, and the pixel group may have a size of LxL (where L is an integer).

[0339] In this case, the size L of the pixel group can also be determined as in Equation 38 below: W and H are the width and height of the current block, respectively.

[0340]

number

[0341] [Table 1] That is, when the video decoding device 100 performs motion compensation in units of pixel groups, the number of division operations can be reduced by a value obtained by dividing by L^2, which is a value corresponding to the size of the pixel group, and then the division operations can be performed.

[0342] FIG. 9A is a diagram illustrating a process of determining a gradient value in the horizontal or vertical direction by adding an offset and performing inverse scaling after filtering, according to an embodiment.

[0343] Referring to FIG. 9A, the video decoding apparatus 100 may determine a horizontal or vertical gradient value by filtering a pixel whose directional component is an integer position using a first one-dimensional filter and a second one-dimensional filter. However, for a pixel whose directional component is an integer position, a value filtered using the first one-dimensional filter or the second one-dimensional filter may fall outside a predetermined range. This phenomenon is called an overflow phenomenon. The coefficients of the one-dimensional filter are also determined as integers to perform integer arithmetic instead of inaccurate and complex decimal arithmetic. Since the coefficients of the one-dimensional filter are determined as integers, they may be scaled. While filtering using the scaled one-dimensional filter coefficients enables integer arithmetic, the magnitude of the filtered value may be larger than when filtering is performed using an unscaled one-dimensional filter, which may result in an overflow phenomenon. Therefore, to prevent the overflow phenomenon, inverse scaling may be performed after filtering using the one-dimensional filter. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits is determined while maximizing calculation accuracy and taking into consideration the maximum number of bits of a register for filtering calculation and the maximum number of bits of a temporal buffer for storing filtering calculation results. In particular, the number of inverse scaling bits is determined based on the internal bit depth, the number of scaling bits for an interpolation filter, and the number of scaling bits for a gradient filter.

[0344] In the following, in order to determine a horizontal gradient value, a vertical interpolation filter is first used to perform filtering on pixels at integer positions to generate a vertical interpolation filtering value, and then a horizontal gradient filter is used to perform inverse scaling on the vertical interpolation filtering value during the filtering process.

[0345] According to Equation (12), the video decoding apparatus 100 may first perform filtering on pixels at integer positions using a vertical interpolation filter to determine horizontal gradient values. In this case, shift1 is also determined as b-8, where b is also the internal bit depth of the input image. Hereinafter, with reference to Table 2, the register bit depth (Reg Bitdepth) and the temporary buffer bit depth (Temp Bitdepth) will be described when actual inverse scaling is performed based on shift1.

[0346] [Table 2] At this time, the values ​​of the variables in Table 2 are also determined by the following equation (39).

[0347]

number

[0348] For example, assuming the quarter-pel gradient filter fracFilter disclosed in FIG. 7C is utilized, FilterSumPos is also 88 and FilterSumNeg is also -24.

[0349] The Ceiling(x) function returns the smallest integer greater than or equal to the error x. offset1 is an offset value added to the filtered value to prevent rounding errors that may occur during inverse scaling using shift1. offset1 is also determined as 2^(shift1-1).

[0350] Referring to Table 2, when the internal bit depth b is 8, the register bit depth (RegBitdpeth) is 16, when the internal bit depth b is 9, the register bit depth is 17, and when the internal bit depth b is 10, 11, 12, or 16, the register bit depth is 18, 19, or 24. If the register used for filtering is a 32-bit register, the bit depth of the register in Table 2 will not exceed 32, and therefore no overflow will occur.

[0351] Similarly, if the internal bit depth b is 8, 9, 10, 11, 12, or 16, the bit depth of the temporary buffer (TempBitDepth) is 16. If the temporary buffer used to store the filtered and inversely scaled values ​​is a 16-bit buffer, the bit depth of the temporary buffer in Table 2 is 16, and since none of the values ​​exceed 16, no overflow will occur.

[0352] According to Equation (12), to determine a horizontal gradient value, the video decoding apparatus 100 first uses a vertical interpolation filter to filter pixels at integer positions to generate a vertical interpolation filtered value, and then uses a horizontal gradient filter to filter the vertical interpolation filtered value according to Equation (13). In this case, shift2 may be determined as p+q-shift1. Here, p represents the number of scaled bits for the interpolation filter including the filter coefficients shown in FIG. 7C, and q represents the number of scaled bits for the gradient filter including the filter coefficients shown in FIG. 7A. For example, p is 6 and q is 4, so shift2=18-b.

[0353] The reason why shift2 is determined as described above is that the final filtering result value is the same when the filter coefficients are upscaled and when the filter coefficients are not upscaled, so the total number of bits to be inversely scaled, shift1+shift2, must be the same as the sum of the number of bits upscaled for the filter (p+q).

[0354] Hereinafter, with reference to Table 3, when actual inverse scaling is performed based on shift2, the register bit depth (Reg Bitdepth) and the temporary buffer bit depth (Temp Bitdepth) will be described.

[0355] [Table 3] At this time, the values ​​of the variables in Table 3 are also determined by the following equation (40).

[0356]

number

[0357] offset2 is an offset value added to the filtered value to prevent rounding errors that may occur during inverse scaling using shift2, and offset2 is also determined as 2^(shift2-1).

[0358] Shift1 and shift2 may be determined as described above, but are not limited thereto. Shift1 and shift2 may be determined in various ways so that the sum of shift1 and shift2 is equal to the sum of the scaling bits for the filter. In this case, the shift1 and shift2 values ​​may be determined on the assumption that no overflow occurs. Shift1 and shift2 may also be determined based on the internal bit depth of the input image and the scaling bits for the filter.

[0359] However, it is not necessary to determine shift1 and shift2 so that the sum of shift1 and shift2 is the same as the sum of the number of scaling bits for the filter. For example, shift1 may be determined as d-8, but shift2 may be determined as a fixed number.

[0360] If shift1 is the same as before and shift2 is a fixed integer of 7, OutMax, OutMin, and Temp Bitdepth may be different in Table 3. The bit depth of the temporary buffer (Temp Bitdepth) will be explained below with reference to Table 4.

[0361] [Table 4] Unlike Table 3, in Table 4, the bit depth of the temporary buffer is the same as 16 for all b, and if a 16-bit temporary buffer is used to store the result data, no overflow occurs for the internal bit depth of any input image because the bit depth of the temporary buffer is less than 16. On the other hand, referring to Table 3, if the internal bit depth of the input image is 12 or 16, and a 16-bit temporary buffer is used to store the result data, an overflow occurs because the bit depth of the temporary buffer is greater than 16.

[0362] If shift2 is a fixed number, the result of filtering without using scaled filter coefficients may differ from the result of filtering using scaled filter coefficients and then performing inverse scaling. In this case, it would be easy to understand for a person skilled in the art that additional inverse scaling is required.

[0363] In the above description, in order to determine a horizontal gradient value, a vertical interpolation filter is first used to perform filtering on pixels at integer positions to generate a vertical interpolation filtered value, and then a horizontal gradient filter is used to perform inverse scaling on the vertical interpolation filtered value during the filtering process. However, the present invention is not limited to this, and it will be readily apparent to those skilled in the art that inverse scaling can be similarly performed when filtering is performed on pixels whose predetermined directional components are integers in order to determine horizontal and vertical gradient values ​​using various one-dimensional filter combinations.

[0364] FIG. 9B is a diagram illustrating a process of determining a gradient value in the horizontal or vertical direction by adding an offset and performing inverse scaling after filtering according to another embodiment.

[0365] 9B, the video decoding apparatus 100 may perform filtering using fractional pixels and integer pixels of a reference picture as input. Here, it is assumed that the fractional pixels of the reference picture are values ​​determined by applying one-dimensional filters in the horizontal and vertical directions to the integer pixels of the reference picture.

[0366] The video decoding apparatus 100 may determine horizontal or vertical gradient values ​​for pixels whose position components in a predetermined direction are decimal and integer pixels by filtering using a one-dimensional filter in the horizontal or vertical direction. However, for pixels whose position components in a predetermined direction are decimal and integer pixels, values ​​filtered using a first one-dimensional filter may fall outside a predetermined range. This phenomenon is called an overflow phenomenon. The coefficients of the one-dimensional filter may be determined as integers to perform integer arithmetic instead of inaccurate and complex decimal arithmetic. The coefficients of the one-dimensional filter may also be scaled to be determined as integers. While filtering using the scaled one-dimensional filter coefficients enables integer arithmetic, the magnitude of the filtered values ​​may be larger than when filtering is performed using an unscaled one-dimensional filter, which may result in an overflow phenomenon. Therefore, to prevent the overflow phenomenon, inverse scaling may be performed after filtering using the one-dimensional filter. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits (shift 1) to the right. The number of inverse scaling bits is determined while maximizing calculation accuracy and taking into consideration the maximum number of bits of a register for filtering calculation and the maximum number of bits of a temporary buffer for storing filtering calculation results. In particular, the number of inverse scaling bits is determined based on the internal bit depth and the number of scaling bits for a gradient filter.

[0367] FIG. 9C is a diagram illustrating ranges required for determining horizontal and vertical displacement vectors in the process of performing pixel-by-pixel motion compensation on a current block.

[0368] 9C , in a process of performing pixel-wise motion compensation on a reference block 910 corresponding to a current block, the video decoding apparatus 100 may determine a horizontal displacement vector per unit time and a vertical displacement vector per unit time at the pixel 915 by using a window 920 around a pixel 915 located at the upper left corner of the reference block 910. In this case, the horizontal or vertical displacement vector per unit time may be determined by using pixel values ​​and gradient values ​​of pixels located outside the reference block 910. In a similar manner, in a process of determining horizontal displacement vectors and vertical displacement vectors for pixels located at the boundary of the reference block 910, the video decoding apparatus 100 may determine pixel values ​​and gradient values ​​of pixels located outside the reference block 910. Therefore, the video decoding apparatus 100 may determine a horizontal displacement vector and a vertical displacement vector per unit time by using a block 925 that is larger than the reference block 910. For example, if the size of the current block is AxB and the pixel-wise window size is (2M+1)x(2N+1), the size of the range for determining the horizontal displacement vector and the vertical displacement vector is also (A+2M)x(B+2N).

[0369] 9D and 9E are diagrams illustrating the range of an area used in the process of performing pixel-by-pixel motion compensation according to various embodiments.

[0370] 9D, in the process of performing pixel-based motion compensation, the video decoding apparatus 100 may determine horizontal displacement vectors and vertical displacement vectors per unit time for each pixel included in the reference block 930 based on a block 935 of a range expanded according to the size of a window of pixels located at the boundary of the reference block 930. However, in the process of determining the horizontal and vertical displacement vectors per unit time, the video decoding apparatus 100 needs pixel values ​​and gradient values ​​of pixels located in the block 935, and may use an interpolation filter or a gradient filter to obtain the pixel values ​​and gradient values. In the process of using the interpolation filter or gradient filter for boundary pixels of the block 935, pixel values ​​of surrounding pixels may be used, and therefore pixels located outside the block boundary may be used. Therefore, pixel-based motion compensation may also be performed using a block 940 of a range expanded by a value obtained by subtracting one from the number of taps of the interpolation filter or gradient filter. Thus, if the block size is NxN, the window size per pixel is (2M+1)x(2M+1), and the interpolation filter or gradient filter length is T, then it is also (N+2M+T-1)x(N+2M+T-1).

[0371] 9E, in the process of performing pixel-wise motion compensation, the video decoding apparatus 100 may determine a horizontal displacement vector and a vertical displacement vector per unit time for each pixel using pixel values ​​and gradient values ​​of pixels located within the reference block 945 without extending the reference block according to the window size of pixels located at the boundary of the reference block 945. Specifically, a process for the video decoding apparatus 100 to determine a horizontal displacement vector and a vertical displacement vector per unit time for each pixel without extending the reference block will be described with reference to FIG. 9E. However, to obtain pixel values ​​and gradient values ​​of pixels, an interpolation filter or gradient filter of the reference block 945 may be used, and pixel-wise motion compensation may be performed using the extended block 950. Therefore, if the block size is NxN, the pixel-wise window size is (2M+1)x(2M+1), and the interpolation filter length or gradient filter length is T, then the pixel-wise motion compensation may also be (N+T-1)x(N+T-1).

[0372] FIG. 9F is a diagram illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block.

[0373] 9F, for a pixel located outside the boundary of the reference block 955, the video decoding device 100 adjusts the position of the pixel to the position of the pixel located within the boundary of the reference block 955 that is closest to the pixel available, and the pixel value and gradient value of the pixel located outside the boundary are also determined as the pixel value and gradient value of the pixel located within the boundary of the reference block 955. In this case, the video decoding device 100 determines the position of the pixel located outside the boundary of the reference block 955 using the mathematical formula:

[0374]

number

[0375]

number

[0376] Here, i' is the x-coordinate value of the pixel, j' is the y-coordinate value of the pixel, and H and W are the height and width of the reference block. In this case, the position of the upper left corner of the reference block is assumed to be (0,0). If the position of the upper left corner of the reference block is (xP, yP), the position of the final pixel is also (i'+xP, j'+yP).

[0377] Referring also to FIG. 9D, in a block 935 expanded according to the size of the pixel-by-pixel window, the positions of pixels located outside the boundary of the reference block 930 are adjusted to the positions of adjacent pixels within the boundary of the reference block 930, and as shown in FIG. 9E, the video decoding device 100 can determine horizontal displacement vectors and vertical displacement vectors per unit time for each pixel within the reference block 945 using the pixel values ​​and gradient values ​​of the reference block 945.

[0378] Therefore, the video decoding device 100 does not expand the reference block 945 according to the pixel-by-pixel window size, but instead performs pixel-by-pixel motion compensation, thereby reducing the number of memory accesses for pixel value references, the number of multiplication operations, and the complexity of the operations.

[0379] When the video decoding device 100 performs block-based motion compensation (as when operating according to the HEVC standard), when it performs pixel-based motion compensation with block extension according to the window size, or when it performs pixel-based motion compensation without block extension, the memory access operations and multiplication operations corresponding to the respective numbers of memory accesses and multiplication operations are performed in the video decoding device 100 as shown in Table 5. In this case, it is assumed that the signal (interpolation) filter length is 8, the gradient filter length is 6, the block size is NxN, and the window size per pixel (2M+1) is 5.

[0380] [Table 5] In block-based motion compensation according to the HEVC standard, an 8-tap interpolation filter is used for each sample, requiring 8 neighboring samples. Therefore, if the size of the reference block is NxN, 8-tap interpolation requires (N+7)x(N+7) reference samples. Since bidirectional motion prediction and compensation are used, two reference blocks are used. Therefore, block-based motion compensation according to the HEVC standard requires 2*(N+7)x(N+7) memory accesses, as shown in Table 4. When pixel-based motion compensation is performed with block extension, if M=2, and an 8-tap interpolation filter or gradient filter is used to perform pixel-based motion compensation on an extended block of size (N+4)x(N+4), (N+4+7)x(N+4+7) reference samples are required. Since bidirectional motion prediction and compensation are used, two reference blocks are used. Finally, pixel-based motion compensation performed with block extension requires 2*(N+4+7)x(N+4+7) memory accesses, as shown in Table 5.

[0381] When pixel-based motion compensation is performed without block extension, (N+7) x (N+7) reference samples are required, as in block-based motion compensation according to the HEVC standard, because there is no block extension. Since bidirectional motion prediction and compensation is used, two reference blocks are used. Therefore, pixel-based motion compensation without block extension requires 2*(N+7) x (N+7) memory accesses, as shown in Table 5.

[0382] Meanwhile, in block-based motion compensation according to the HEVC standard, an 8-tap interpolation filter is used for each sample, so the number of samples required for the first horizontal interpolation is (N+7)xN samples. The number of samples required for the second vertical interpolation is NxN samples. Meanwhile, the number of multiplication operations required per 8-tap interpolation filter is 8, and since bidirectional motion prediction and compensation is used, two reference blocks are used. Therefore, block-based motion compensation according to the HEVC standard requires 2*8*((N+7)*N+N*N) multiplication operations as shown in Table 5.

[0383] When pixel-based motion compensation is performed together with block extension, the block size is extended to perform pixel-based motion compensation. Therefore, an 8-tap interpolation filter is used for the extended (N+4)x(N+4) size block, and a total of 2*8*{(N+4+7)x(N+4)+(N+4)x(N+4)} multiplication operations are required to determine the pixel value at a fractional pixel position, as shown in Table 5.

[0384] On the other hand, when pixel-wise motion compensation is performed together with block extension, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine horizontal or vertical gradient values. As the block size is extended, a total of 2*6*{(N+4+5)x(N+4)+(N+4)x(N+4)}*2 multiplication operations are required to determine gradient values ​​using a 6-tap interpolation filter and gradient filter for an extended (N+4)x(N+4) size block, as shown in Table 5.

[0385] When pixel-based motion compensation is performed without block extension, (N+7)x(N+7) reference samples are required, as in block-based motion compensation according to the HEVC standard, because there is no block extension. Furthermore, two reference blocks are used because of bidirectional motion prediction and compensation. Therefore, pixel-based motion compensation without block extension requires an 8-tap interpolation filter for an NxN block, and 2*8*{(N+7)xN+NxN} multiplication operations are required to determine the pixel value at a fractional pixel position, as shown in Table 5.

[0386] On the other hand, when pixel-by-pixel motion compensation is performed without block extension, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine the horizontal or vertical gradient values. For an NxN block, a total of 2*6*{(N+5)xN+NxN}*2 multiplication operations are required to determine the gradient values ​​using a 6-tap interpolation filter and gradient filter, as shown in Table 5.

[0387] 9G to 9I are diagrams illustrating a process for determining horizontal and vertical displacement vectors without extending a reference block according to another embodiment.

[0388] 9D, in the process of performing pixel-by-pixel motion compensation, the video decoding apparatus 100 may determine horizontal displacement vectors and vertical displacement vectors per unit time for each pixel included in the reference block 930 based on a block 935 of a range expanded according to the size of a window of pixels located at the boundary of the reference block 930. For example, when the window size is (2M+1)×(2M+1), if the window is applied to pixels located at the boundary of the reference block 930, the video decoding apparatus 100 may determine horizontal displacement vectors and vertical displacement vectors for each pixel by referring to pixel values ​​and gradient values ​​of pixels located M away from the reference block 930.

[0389] Hereinafter, a method will be described in which the video decoding apparatus 100 determines values ​​(s1 to s6 in Equation (33)) for determining horizontal and vertical displacement vectors for each pixel by using only pixel values ​​and gradient values ​​of a reference block corresponding to a current block without referring to pixel values ​​and gradient values ​​of pixels outside the reference block, according to another embodiment. Here, it is assumed that the window size is 5x5. For convenience, pixels positioned horizontally around the current pixel will be described. Those skilled in the art will easily understand that weights can also be determined for pixels positioned vertically around the current pixel using the same method.

[0390] The video decoding device 100 calculates pixel values ​​P0(i',j' and P1(i',j'), horizontal or vertical gradient values ​​P0(i',j'), P1(i',j'), and P2(i',j') for each pixel included in a window centered on a current pixel for which a horizontal or vertical displacement vector is to be determined.

[0391]

number

[0392] In this case, the result value of the operation performed for each pixel is multiplied by the weight value so that each pixel has the same weight value, and the resulting values ​​are added together to determine values ​​s1 to s6 for determining the horizontal and vertical displacement vectors for each pixel.

[0393] 9G, in the process of determining horizontal and vertical displacement vectors for a current pixel 961 in a current block 960, the video decoding device 100 may determine weights for the pixels in the window so that the weights have the same value of 1. The video decoding device 100 may multiply the resultant values ​​calculated for each pixel by the weights determined for each pixel, add the resultant values, and determine values ​​s1 to s6 for determining the horizontal and vertical displacement vectors for the current pixel.

[0394] 9H, in a current block 970, if a current pixel 971 is immediately adjacent to the boundary of the current block 970, the video decoding device 100 may determine the weight value of the pixel adjacent to the boundary of the block 970 that is located outside the boundary of the current block 970 to be 3. The video decoding device 100 may determine the weight value of the other pixel 973 to be 1.

[0395] 9H, if a current pixel 981 is located near the boundary of the current block 980 (the current pixel is about one pixel away from the boundary), the video decoding device 100 may determine a weight value of 0 for a pixel 982 located outside the boundary of the current block 980, and a weight value of 2 for a pixel 983 adjacent to the boundary of the current block 980. The video decoding device 100 may determine a weight value of 1 for the other pixel 984.

[0396] As described with reference to Figures 9G to 9I, the video decoding device 100 assigns different weights to each pixel within the window depending on the position of the current pixel, thereby being able to determine values ​​s1 to s6 for determining horizontal and vertical displacement vectors for each pixel using pixel values ​​and gradient values ​​of pixels located inside the reference block, rather than using pixel values ​​and gradient values ​​of pixels located outside the reference block corresponding to the current block.

[0397] Figure 9J is a diagram illustrating a process of determining horizontal and vertical displacement vectors for each pixel by applying exponential smoothing techniques to the top, bottom, left, and right without extending the block, in accordance with one embodiment.

[0398] 9J, the video decoding apparatus 100 may obtain, for each pixel included in the current block 990, pixel values ​​P0(i',j') and P1(i',j') of corresponding reference pixels included in the corresponding reference block, horizontal or vertical gradient values ​​of corresponding reference pixels included in the corresponding reference block, and

[0399]

number

[0400] The video decoding device 100 combines the function operation result value of the current pixel with the function operation result value of the surrounding pixels to obtain values ​​s1 to s6 (σ) for determining horizontal and vertical displacement vectors of the current pixel. k(k=1,2,3,4,5,6) ) is also determined. That is, the values ​​s1 to s6 for determining the horizontal and vertical displacement vectors of the current pixel are also expressed as a weighted average of the calculation results for the current pixel and the surrounding pixels as shown in Equation (41) below. At this time, the position coordinates of the pixel included in the window Ω are (i',j'). Also, W[i',j'] means the weight value for the pixel included in the window Ω. Here, the size of the window Ω is also (2M+1)x(2M+1) (M is an integer). Also, the function A k [i',j'] is the pixel value P0(i',j') and P1(i',j') (I[i',j'](0,1)) of the corresponding reference pixel for the pixel at position (i',j') included in the window Ω, and the horizontal or vertical gradient value of the corresponding reference pixel included in the pixel of the corresponding reference block.

[0401]

number

[0402]

number

[0403] [Table 6] Here I x (0) [i',j'], I x (1) [i',j'] are respectively

[0404]

number

[0405]

number

[0406] The video decoding device 100 is k To determine the weighted average over [i',j'], A k Exponential smoothing techniques can be applied to [i',j'] in the up, down, left and right directions to perform averaging.

[0407] Referring to FIG. 9J, first, the video decoding device 100 performs a process of dividing the image data into two parts, A, B, C, and D, in the left and right directions. k An exponential smoothing technique can be applied to [i', j']. In the following, the video decoding device 100 performs the exponential smoothing k Apply exponential smoothing to [i',j'] and A k The details of determining the weighted average value related to [i', j'] will be explained below.

[0408] First, the video decoding device 100 applies exponential smoothing to perform averaging to the right. The video decoding device 100 performs averaging to the right using the following pseudo code 1: k Averaging can be performed to the right for [i',j']. Here, H is the height of the current block, W is the width of the current block, and Stride is the distance between one line and the next line in a one-dimensional array. In other words, a two-dimensional array A[i,j] can also be expressed by a one-dimensional array A[i+j*Stride]. [Pseudocode 1]

[0409]

number

[0410] The video decoding device 100 applies exponential smoothing to the left and averages the data in the left direction. k Averaging can be performed to the left for [i',j']. Here, H is the height of the current block, W is the width of the current block, and Stride is the distance between one line and the next line in a one-dimensional array. In other words, a two-dimensional array A[i,j] can also be expressed by a one-dimensional array A[i+j*Stride]. [Pseudocode 2]

[0411]

number

[0412] The video decoding device 100 applies exponential smoothing to average downwards. The video decoding device 100 calculates A by the following pseudocode 3: k Averaging can be performed downward for [i',j']. Here, H is the height of the current block, W is the width of the current block, and Stride is the distance between one line and the next line in a one-dimensional array. In other words, a two-dimensional array A[i,j] can also be expressed by a one-dimensional array A[i+j*Stride]. [Pseudocode 3]

[0413]

number

[0414]

number

[0415] Therefore, the video decoding device 100 uses an exponential smoothing technique on the current block 990 to average in the up, down, left, and right directions, thereby obtaining a weighted average σ necessary for determining a horizontal or vertical displacement vector for each pixel. kThat is, the video decoding apparatus 100 may determine a horizontal or vertical displacement vector for each pixel by referring only to the pixel values ​​and gradient values ​​of the reference block corresponding to the current block 990, without referring to the pixel values ​​and gradient values ​​of the reference block corresponding to the block 996 obtained by extending the current block 990.

[0416] Figure 9K is a diagram illustrating a process of determining pixel values ​​of reference pixels in a reference block and horizontal and vertical gradient values ​​using a filter to perform motion compensation for a current block according to one embodiment.

[0417] 9K, the video decoding apparatus 100 may perform pixel-based and block-based motion compensation for the current block using pixel values ​​and gradient values ​​of reference pixels in a reference block corresponding to the current block. Therefore, to perform pixel-based and block-based motion compensation for the current block, pixel values ​​and gradient values ​​of reference pixels in a reference block corresponding to the current block must be determined. Here, the pixel values ​​and gradient values ​​of the reference pixels in the reference block may be in fractional pixel units. For example, the pixel values ​​and gradient values ​​of the reference pixels in the reference block may be in 1 / 16 pixel (1 / 16 pel) units.

[0418] The video decoding device 100 may perform filtering on pixel values ​​of integer pixels of the reference block to determine pixel values ​​and gradient values ​​of reference pixels in the fractional pixel unit reference block.

[0419] First, the video decoding device 100 can apply a horizontal 8-tap signal filter (also called an interpolation filter) and a vertical 8-tap signal filter to the pixel values ​​of the integer pixels of the reference pixel to determine the pixel values ​​of the reference pixel in the reference block.

[0420] The video decoding device 100 can determine pixel values ​​of reference pixels having fractional pixel position components in the horizontal direction by applying a horizontal 8-tap signal filter to integer pixel values ​​of the reference block and store the pixel values ​​in a buffer. The video decoding device 100 can determine pixel values ​​of reference pixels having fractional pixel position components in the vertical direction by applying a vertical 8-tap signal filter to pixel values ​​of reference pixels having integer pixel position components in the vertical direction.

[0421] The video decoding device 100 can determine the horizontal gradient values ​​of the reference pixels in the reference block by first applying a vertical 6-tap signal filter to the integer pixel values ​​of the reference block, and then applying a horizontal 6-tap gradient filter to perform filtering.

[0422] That is, the video decoding device 100 can determine pixel values ​​of reference pixels having position components in fractional pixel units in the vertical direction by applying a vertical 6-tap signal filter to the integer pixel values ​​of the reference block and store them in a buffer.

[0423] The video decoding device 100 applies a horizontal 6-tap gradient filter to pixel values ​​of reference pixels having integer-unit position components in the horizontal direction, and can also determine horizontal gradient values ​​of reference pixels having fractional-pixel unit position components in the horizontal direction.

[0424] The video decoding device 100 can determine the vertical gradient value of the reference pixel in the reference block by first applying a vertical 6-tap gradient filter to the integer pixel values ​​of the reference block, and then applying a horizontal 6-tap signal filter to perform filtering.

[0425] That is, the video decoding device 100 can determine gradient values ​​of reference pixels having position components in the vertical direction in fractional pixel units by applying a vertical 6-tap gradient filter to the integer pixel values ​​of the reference block to perform filtering, and store the gradient values ​​in a buffer.

[0426] The video decoding device 100 can apply a horizontal 6-tap signal filter to the vertical gradient values ​​of reference pixels having integer-unit position components in the horizontal direction, and determine vertical gradient values ​​of reference pixels having fractional-pixel unit position components in the horizontal direction as well.

[0427] That is, the video decoding apparatus 100 applies two one-dimensional filters to determine the pixel value of a reference pixel in a reference block, the horizontal gradient value of the reference pixel in the reference block, and the vertical gradient value of the reference pixel in the reference block, respectively, and performs multiplication operations between the coefficients of each filter and the value related to the corresponding pixel. For example, two 6-tap signal / gradient filters are used to determine the horizontal gradient value of a reference pixel in the reference block, and a total of 12 multiplication operations are performed per pixel. Also, two 6-tap signal / gradient filters are used to determine the vertical gradient value of a reference pixel in the reference block, and a total of 12 multiplication operations are performed per pixel.

[0428] Figure 9L is a diagram illustrating a process of using a filter to determine pixel values ​​of reference pixels in a reference block and horizontal and vertical gradient values ​​to perform motion compensation for a current block according to another embodiment.

[0429] 9L, the video decoding apparatus 100 may perform pixel-based and block-based motion compensation for the current block using pixel values ​​and gradient values ​​of reference pixels in a reference block corresponding to the current block. Therefore, to perform pixel-based and block-based motion compensation for the current block, pixel values ​​and gradient values ​​of reference pixels in a reference block corresponding to the current block must be determined. Here, the pixel values ​​and gradient values ​​of the reference pixels in the reference block may be expressed in fractional pixel units. For example, the pixel values ​​and gradient values ​​of the reference pixels in the reference block may be expressed in 1 / 16 pixel (1 / 16 pel) units.

[0430] The video decoding device 100 may perform filtering on pixel values ​​of integer pixels of the reference block to determine pixel values ​​and gradient values ​​of reference pixels in the fractional pixel unit reference block.

[0431] 9K, the video decoding apparatus 100 may first apply a horizontal 8-tap signal filter and a vertical 8-tap signal filter to integer pixel values ​​of the reference block to determine pixel values ​​of reference pixels in the reference block, and then apply a horizontal 5-tap gradient filter to the pixel values ​​of the reference pixels in the reference block to determine horizontal gradient values ​​of the reference pixels in the reference block. Also, the video decoding apparatus 100 may apply a vertical 5-tap gradient filter to the pixel values ​​of the reference pixels in the reference block to determine vertical gradient values ​​of the reference pixels in the reference block.

[0432] The video decoding device 100 applies two one-dimensional signal (interpolation) filters to determine the pixel value of a reference pixel having a position in fractional pixel units, and then applies two one-dimensional gradient filters in parallel to the pixel value of the reference pixel located in fractional pixel units, thereby determining the horizontal gradient value or vertical gradient value of the reference pixel in the reference block.

[0433] The video decoding device 100 may apply a 5-tap horizontal gradient filter (with filter coefficients of {9, -48, 0, 48, 9}, but the filter coefficients are not limited thereto) to the pixel value of the reference pixel as shown in the following Equation (42) to determine the horizontal gradient value Ix(k) of the reference pixel in the reference block. Here, k may have a value of 0 or 1, indicating reference pictures 0 and 1, respectively. I(k)[i,j] is also the pixel value of the reference pixel in the reference block at position (i,j). i is the horizontal position component of the pixel, and j is the vertical position component of the pixel, and the units are also fractional pixels.

[0434]

number

[0435] Furthermore, the video decoding device 100 may apply a 5-tap vertical gradient filter (with filter coefficients of {9, -48, 0, 48, 9}, but is not limited thereto) to the pixel value of the reference pixel as shown in the following Equation (43), to determine a vertical gradient value Iy(k) of the reference pixel in the reference block. Here, k may have a value of 0 or 1, indicating reference pictures 0 and 1, respectively. I(k)[i,j] is also the pixel value of the reference pixel in the reference block at position (i,j). i is the horizontal position component of the pixel, and j is the vertical position component of the pixel, and the units are also fractional pixels.

[0436]

number

[0437] When the video decoding device 100 performs pixel-based motion compensation together with block extension according to the window size, and when the gradient filter length is reduced and pixel-based motion compensation is performed, the memory access operations and multiplication operations corresponding to the respective numbers of memory accesses and multiplication operations are performed in the video decoding device 100 as shown in Table 7 below. In this case, it is assumed that the signal filter length T is 8, the gradient filter length T is 6, the simplified gradient filter length T is 5, the block size is NxN, and the pixel-by-pixel window size 2M+1 is 5.

[0438] [Table 7] That is, the video decoding device 100 performs two multiplication operations per one-dimensional gradient filter according to equations (41) and 42, and the gradient filters are applied to two reference blocks. The gradient filters are applied to the (N+4)x(N+4) reference blocks expanded based on the window size, so a total of 2*2*{(N+4)x(N+4)}*2 multiplication operations are also performed to determine the gradient values ​​of the reference pixels in the horizontal and vertical directions.

[0439] When the video decoding device 100 performs pixel-based motion compensation with block extension according to the window size, or when the video decoding device 100 reduces the gradient filter length without block extension and performs pixel group-based motion compensation, the memory access operations and multiplication operations corresponding to the respective numbers of memory accesses and multiplication operations are performed in the video decoding device 100 as shown in Table 8. In this case, it is assumed that the signal filter length T is 8, the gradient filter length T is 6, the reduced gradient filter length T is 5, the pixel group size is LxL, the block size is NxN, and the pixel-based window size 2M+1 is 5.

[0440] [Table 8] Therefore, when the video decoding device 100 performs pixel group-based motion compensation by reducing the gradient filter length without block extension, the number of memory accesses, multiplication operations, and multiplication operations are much reduced compared to when pixel-based motion compensation is performed with block extension.

[0441] Hereinafter, a method for determining data units that can be used in the video decoding process of the video decoding device 100 according to an embodiment will be described with reference to Figures 10 to 23. The operation of the video encoding device 150 may be similar to or opposite to various embodiments relating to the operation of the video decoding device 100, which will be described later.

[0442] FIG. 10 illustrates a process in which the video decoding device 100 divides the current coding unit and determines at least one coding unit, according to one embodiment.

[0443] According to one embodiment, the video decoding device 100 may determine the type of coding unit using block type information and may determine the type of division of the coding unit using the partition type information. That is, the division method of the coding unit indicated by the partition type information may be determined depending on the block type indicated by the block type information used by the video decoding device 100.

[0444] According to one embodiment, the video decoding apparatus 100 may use block shape information indicating that the current coding unit is square. For example, the video decoding apparatus 100 may determine whether to not split the square coding unit, to split it vertically, to split it horizontally, or to split it into four coding units, based on the partition shape information. Referring to FIG. 10, when the block shape information of the current coding unit 1000 indicates a square shape, the video decoding apparatus 100 may not split a coding unit 1010a having the same size as the current coding unit 1000 based on the partition shape information indicating no partition, or may determine divided coding units 1010b, 1010c, and 1010d based on the partition shape information indicating a predetermined partitioning method.

[0445] 10 , according to one embodiment, the video decoding apparatus 100 may determine two coding units 1010b obtained by vertically dividing the current coding unit 1000 based on the division type information indicating vertical division. The video decoding apparatus 100 may determine two coding units 1010c obtained by horizontally dividing the current coding unit 1000 based on the division type information indicating horizontal division. The video decoding apparatus 100 may determine four coding units 1010d obtained by vertically and horizontally dividing the current coding unit 1000 based on the division type information indicating vertical and horizontal division. However, the division type by which a square coding unit is divided should not be construed as being limited to the above-described type, and may include various types that can be indicated by the division type information. Predetermined division types by which a square coding unit is divided will be described in detail below through various embodiments.

[0446] FIG. 11 illustrates a process in which video decoding device 100 divides a coding unit having a non-square shape to determine at least one coding unit, according to one embodiment.

[0447] According to an embodiment, the video decoding apparatus 100 may use block shape information indicating that the current coding unit is non-square. The video decoding apparatus 100 may determine whether to not split the non-square current coding unit or to split it in a predetermined manner based on the partition shape information. Referring to FIG. 11, when the block shape information of the current coding unit 1100 or 1150 indicates a non-square shape, the video decoding apparatus 100 may not split the coding unit 1110 or 1160 having the same size as the current coding unit 1100 or 1150 based on the partition shape information indicating no partition, or may determine the split coding units 1120a, 1120b, 1130a, 1130b, 1130c, 1170a, 1170b, 1180a, 1180b, and 1180c based on the partition shape information indicating a predetermined partition method. The predetermined partition method for splitting a non-square coding unit will be described in detail below through various embodiments.

[0448] According to an embodiment, the video decoding apparatus 100 may determine a manner in which a coding unit is divided using the partitioning type information, and in this case, the partitioning type information may indicate the number of coding units to be generated by dividing the coding unit. Referring to FIG. 11, if the partitioning type information indicates that the current coding unit 1100 or 1150 is to be divided into two coding units, the video decoding apparatus 100 may divide the current coding unit 1100 or 1150 based on the partitioning type information and determine two coding units 1120a, 1120b, or 1170a, 1170b included in the current coding unit.

[0449] According to an embodiment, when the video decoding apparatus 100 divides the non-square current coding unit 1100 or 1150 based on the division type information, the video decoding apparatus 100 may divide the current coding unit by considering the position of the long side of the non-square current coding unit 1100 or 1150. For example, the video decoding apparatus 100 may divide the current coding unit 1100 or 1150 in the direction of dividing the long side of the current coding unit 1100 or 1150 by considering the shape of the current coding unit 1100 or 1150, thereby determining a plurality of coding units.

[0450] According to one embodiment, if the partitioning type information indicates that a coding unit is to be divided into an odd number of blocks, the video decoding device 100 may determine an odd number of coding units to be included in the current coding unit 1100 or 1150. For example, if the partitioning type information indicates that the current coding unit 1100 or 1150 is to be divided into three coding units, the video decoding device 100 may divide the current coding unit 1100 or 1150 into three coding units 1130a, 1130b, 1130c, 1180a, 1180b, and 1180c. According to one embodiment, the video decoding device 100 may determine an odd number of coding units to be included in the current coding unit 1100 or 1150, and the determined coding units may not all be the same size. For example, among the determined odd number of coding units 1130a, 1130b, 1130c, 1180a, 1180b, and 1180c, a certain coding unit 1130b or 1180b may have a size different from the other coding units 1130a, 1130c, 1180a, and 1180c. That is, the coding units determined by dividing the current coding unit 1100 or 1150 may have a variety of sizes, and in some cases, the odd number of coding units 1130a, 1130b, 1130c, 1180a, 1180b, and 1180c may have different sizes.

[0451] According to an embodiment, if the partitioning type information indicates that a coding unit is to be partitioned into an odd number of blocks, the video decoding apparatus 100 may determine an odd number of coding units included in the current coding unit 1100 or 1150, and may further impose a predetermined restriction on at least one of the odd number of coding units generated by the partitioning. Referring to Figure 11, the video decoding apparatus 100 may perform a different decoding process for the central coding units 1130b and 1180b among three coding units 1130a, 1130b, 1130c, 1180a, 1180b, and 1180c generated by partitioning the current coding unit 1100 or 1150 from the other coding units 1130a, 1130c, 1180a, and 1180c. For example, the video decoding device 100 can limit the centrally located coding units 1130b and 1180b from being further divided, unlike the other coding units 1130a, 1130c, 1180a, and 1180c, or limit them to being divided only a predetermined number of times.

[0452] FIG. 12 illustrates a process in which the video decoding apparatus 100 divides a coding unit based on at least one of block format information and partition format information, according to an embodiment.

[0453] According to one embodiment, the video decoding apparatus 100 may determine whether or not to divide the square-shaped first coding unit 1200 into coding units based on at least one of block shape information and partition shape information. According to one embodiment, if the partition shape information indicates that the first coding unit 1200 is to be divided horizontally, the video decoding apparatus 100 may divide the first coding unit 1200 horizontally to determine the second coding unit 1210. According to one embodiment, the terms "first coding unit," "second coding unit," and "third coding unit" used are terms used to understand the division relationship between coding units. For example, if the first coding unit is divided, the second coding unit is determined, and if the second coding unit is divided, the third coding unit is determined. Hereinafter, the relationship between the first coding unit, the second coding unit, and the third coding unit used may be understood to be in accordance with the above-described characteristics.

[0454] According to an embodiment, the video decoding apparatus 100 may determine whether to divide the determined second coding unit 1210 into coding units based on at least one of block shape information and partition shape information. Referring to FIG. 12, the video decoding apparatus 100 may divide the non-square second coding unit 1210 determined by dividing the first coding unit 1200 into at least one third coding unit 1220a, 1220b, 1220c, and 1220d based on at least one of block shape information and partition shape information, or may not divide the second coding unit 1210. The video decoding apparatus 100 may acquire at least one of block topology information and partition topology information, and may divide the first coding unit 1200 into a plurality of second coding units (e.g., 1210) of various types based on the acquired at least one of the block topology information and the partition topology information. The second coding units 1210 may be divided according to the division method of the first coding unit 1200 based on at least one of the block topology information and the partition topology information. According to an embodiment, when the first coding unit 1200 is divided into the second coding units 1210 based on at least one of the block topology information and the partition topology information related to the first coding unit 1200, the second coding units 1210 may also be divided into third coding units (e.g., 1220a, 1220b, 1220c, and 1220d) based on at least one of the block topology information and the partition topology information related to the second coding unit 1210. That is, the coding units may be recursively divided based on at least one of the division type information and the block type information associated with each coding unit. Thus, for a non-square coding unit, a square coding unit may be determined, and the square coding unit may be recursively divided to determine a non-square coding unit. Referring to Figure 12, for an odd number of third coding units 1220b, 1220c, and 1220d determined by dividing the non-square second coding unit 1210, a predetermined coding unit (e.g., a coding unit located in the middle or a square coding unit) may be recursively divided.According to one embodiment, the square-shaped third coding unit 1220c, which is one of the odd-numbered third coding units 1220b, 1220c, and 1220d, may be divided horizontally into a plurality of fourth coding units. The non-square-shaped fourth coding unit 1240, which is one of the plurality of fourth coding units, may also be divided into a plurality of coding units. For example, the non-square-shaped fourth coding unit 1240 may be further divided into odd-numbered coding units 1250a, 1250b, and 1250c.

[0455] Methods used for recursive division of coding units will be described below through various embodiments.

[0456] According to one embodiment, the video decoding apparatus 100 may determine whether to divide each of the third coding units 1220a, 1220b, 1220c, and 1220d into coding units or not to divide the second coding unit 1210 based on at least one of the block shape information and the partition shape information. According to one embodiment, the video decoding apparatus 100 may divide the non-square second coding unit 1210 into an odd number of third coding units 1220b, 1220c, and 1220d. The video decoding apparatus 100 may impose a certain restriction on certain third coding units among the odd number of third coding units 1220b, 1220c, and 1220d. For example, the video decoding apparatus 100 may restrict the middle coding unit 1220c of the odd number of third coding units 1220b, 1220c, and 1220d from being further divided or from being divided a configurable number of times. 12, the video decoding apparatus 100 may restrict the middle coding unit 1220c of the odd number of third coding units 1220b, 1220c, and 1220d included in the non-square second coding unit 1210 to not be further divided, or to be divided into a predetermined division pattern (e.g., into only four coding units, or into a pattern corresponding to the division pattern of the second coding unit 1210), or to be divided only a predetermined number of times (e.g., divided only n times, n>0). However, the above restriction on the middle coding unit 1220c is merely a simple embodiment, and should not be construed as being limited to the above embodiment, but should be construed as including various restrictions such as the middle coding unit 1220c being decoded differently from the other coding units 1220b and 1220d.

[0457] According to an embodiment, the video decoding apparatus 100 may acquire at least one of block configuration information and partition configuration information used to partition the current coding unit at a predetermined position within the current coding unit.

[0458] 13 illustrates a method for the video decoding apparatus 100 to determine a predetermined coding unit among an odd number of coding units, according to an embodiment. Referring to FIG. 13, at least one of block shape information and partition shape information of a current coding unit 1300 may be obtained from a sample at a predetermined position (e.g., a sample 1340 located in the middle) among a plurality of samples included in the current coding unit 1300. However, the predetermined position within the current coding unit 1300 from which at least one of the block shape information and partition shape information is obtained should not be interpreted as being limited to the center position shown in FIG. 13, but may include various positions within the current coding unit 1300 (e.g., top, bottom, left, right, top left, bottom left, top right, bottom right, etc.). The video decoding apparatus 100 may obtain at least one of the block shape information and partition shape information obtained from the predetermined position and determine whether to divide the current coding unit into coding units of various shapes and sizes, or not to divide the current coding unit.

[0459] According to an embodiment, when a current coding unit is divided into a predetermined number of coding units, the video decoding apparatus 100 may select one of the coding units. There are various methods for selecting one of the plurality of coding units, and such methods will be described later with reference to various embodiments.

[0460] According to an embodiment, the video decoding apparatus 100 may divide a current coding unit into a plurality of coding units and determine a coding unit at a predetermined position.

[0461] FIG. 13 illustrates a method by which video decoding device 100 determines a coding unit at a predetermined position among an odd number of coding units, according to one embodiment.

[0462] According to an embodiment, the video decoding apparatus 100 may use information indicating the positions of each odd-numbered coding unit to determine a middle coding unit among the odd-numbered coding units. Referring to FIG. 13, the video decoding apparatus 100 may divide a current coding unit 1300 to determine odd coding units 1320a, 1320b, and 1320c. The video decoding apparatus 100 may determine the middle coding unit 1320b using information regarding the positions of the odd-numbered coding units 1320a, 1320b, and 1320c. For example, the video decoding apparatus 100 may determine the positions of the coding units 1320a, 1320b, and 1320c based on information indicating the positions of certain samples included in the coding units 1320a, 1320b, and 1320c to determine the middle coding unit 1320b. Specifically, the video decoding device 100 can determine the coding unit 1320b located in the middle by determining the positions of the coding units 1320a, 1320b, and 1320c based on information indicating the positions of the samples 1330a, 1330b, and 1330c at the top left of the coding units 1320a, 1320b, and 1320c.

[0463] According to an embodiment, the information indicating the positions of the top left samples 1330a, 1330b, and 1330c included in the coding units 1320a, 1320b, and 1320c, respectively, may include information related to the positions or coordinates of the coding units 1320a, 1320b, and 1320c within the picture. According to an embodiment, the information indicating the positions of the top left samples 1330a, 1330b, and 1330c included in the coding units 1320a, 1320b, and 1320c, respectively, may include information indicating the width or height of the coding units 1320a, 1320b, and 1320c included in the current coding unit 1300, where such width or height corresponds to information indicating the difference between the coordinates of the coding units 1320a, 1320b, and 1320c within the picture. That is, the video decoding device 100 can determine the coding unit 1320b located in the middle by directly using information related to the positions or coordinates of the coding units 1320a, 1320b, and 1320c within the picture, or by using information related to the width or height of the coding unit corresponding to the difference between the coordinates.

[0464] According to an embodiment, information indicating the position of the top left sample 1330a of the top coding unit 1320a may indicate (xa, ya) coordinates, information indicating the position of the top left sample 1330b of the middle coding unit 1320b may indicate (xb, yb) coordinates, and information indicating the position of the top left sample 1330c of the bottom coding unit 1320c may indicate (xc, yc) coordinates. The video decoding apparatus 100 may determine the middle coding unit 1320b using the coordinates of the top left samples 1330a, 1330b, and 1330c included in the coding units 1320a, 1320b, and 1320c, respectively. For example, when the coordinates of the upper left samples 1330a, 1330b, and 1330c are sorted in ascending or descending order, the coding unit 1320b containing the coordinates (xb, yb) of the middle sample 1330b can be determined as the middle coding unit among the coding units 1320a, 1320b, and 1320c determined by dividing the current coding unit 1300. However, the coordinates indicating the positions of the top left samples 1330a, 1330b, and 1330c may indicate coordinates indicating absolute positions within a picture, or may use (dxb, dyb) coordinates indicating the relative position of the top left sample 1330b of the middle coding unit 1320b based on the position of the top left sample 1330a of the top coding unit 1320a, or (dxc, dyc) coordinates indicating the relative position of the top left sample 1330c of the bottom coding unit 1320c. Furthermore, the method of determining a coding unit at a predetermined position by using the coordinates of a sample as information indicating the position of the sample included in the coding unit should not be construed as being limited to the above-mentioned method, but should be construed as various arithmetic methods that can use the coordinates of the samples.

[0465] According to an embodiment, the video decoding apparatus 100 may divide the current coding unit 1300 into a plurality of coding units 1320a, 1320b, and 1320c, and may select a coding unit from among the coding units 1320a, 1320b, and 1320c according to a predetermined criterion. For example, the video decoding apparatus 100 may select a coding unit 1320b having a different size from the coding units 1320a, 1320b, and 1320c.

[0466] According to an embodiment, the video decoding device 100 may determine the width or height of each of the coding units 1320a, 1320b, and 1320c using the (xa, ya) coordinates indicating the position of the top left sample 1330a of the top coding unit 1320a, the (xb, yb) coordinates indicating the position of the top left sample 1330b of the middle coding unit 1320b, and the (xc, yc) coordinates indicating the position of the top left sample 1330c of the bottom coding unit 1320c. The video decoding device 100 may determine the size of each of the coding units 1320a, 1320b, and 1320c using the (xa, ya), (xb, yb), and (xc, yc) coordinates indicating the positions of the coding units 1320a, 1320b, and 1320c.

[0467] According to an embodiment, the video decoding apparatus 100 may determine the width of the top coding unit 1320a as xb-xa and the height as yb-ya. According to an embodiment, the video decoding apparatus 100 may determine the width of the middle coding unit 1320b as xc-xb and the height as yc-yb. According to an embodiment, the video decoding apparatus 100 may determine the width or height of the bottom coding unit using the width or height of the current coding unit and the widths and heights of the top coding unit 1320a and the middle coding unit 1320b. The video decoding apparatus 100 may determine a coding unit having a different size from the other coding units based on the determined widths and heights of the coding units 1320a, 1320b, and 1320c. Referring to FIG. 13, the video decoding apparatus 100 may determine the middle coding unit 1320b, which has a different size from the top coding unit 1320a and the bottom coding unit 1320c, as a coding unit of a predetermined position. However, the process in which the video decoding device 100 determines coding units having different sizes from different coding units is merely one embodiment of determining a coding unit at a predetermined position using the size of the coding unit determined based on sample coordinates, and various other processes may also be used to determine a coding unit at a predetermined position by comparing the size of the coding unit determined based on predetermined sample coordinates.

[0468] However, the position of the sample considered to determine the position of the coding unit is not limited to the upper left end as mentioned above, but may be interpreted as information relating to any sample position included in the coding unit being used.

[0469] According to an embodiment, the video decoding apparatus 100 may select a coding unit at a predetermined position from among an odd number of coding units determined by dividing the current coding unit, taking into account the shape of the current coding unit. For example, if the current coding unit has a non-square shape in which the width is longer than the height, the video decoding apparatus 100 may determine a coding unit at a predetermined position along the horizontal direction. That is, the video decoding apparatus 100 may determine one of the coding units at a different position in the horizontal direction and set a constraint on the coding unit. If the current coding unit has a non-square shape in which the height is longer than the width, the video decoding apparatus 100 may determine a coding unit at a predetermined position along the vertical direction. That is, the video decoding apparatus 100 may determine one of the coding units at a different position in the vertical direction and set a constraint on the coding unit.

[0470] According to an embodiment, the video decoding apparatus 100 may use information indicating the positions of each of the even-numbered coding units to determine a coding unit at a predetermined position among the even-numbered coding units. The video decoding apparatus 100 may divide the current coding unit to determine the even-numbered coding units, and may determine a coding unit at a predetermined position using information regarding the positions of the even-numbered coding units. Specific steps related to this process correspond to the process of determining a coding unit at a predetermined position (e.g., the middle position) among the odd-numbered coding units described with reference to FIG. 13, and therefore will not be described further.

[0471] According to an embodiment, when a non-square current coding unit is divided into a plurality of coding units, predetermined information about the coding unit at a predetermined position may be used in the division process to determine a coding unit at a predetermined position among the plurality of coding units. For example, the video decoding apparatus 100 may use at least one of block shape information and partition shape information stored in a sample included in a middle coding unit in the division process to determine a coding unit at a middle position among the plurality of coding units into which the current coding unit is divided.

[0472] 13, the video decoding apparatus 100 may divide a current coding unit 1300 into a plurality of coding units 1320a, 1320b, and 1320c based on at least one of block shape information and partition shape information, and may determine a central coding unit 1320b among the plurality of coding units 1320a, 1320b, and 1320c. Furthermore, the video decoding apparatus 100 may determine the central coding unit 1320b by considering a position where at least one of block shape information and partition shape information is obtained. That is, when at least one of the block shape information and partition shape information of the current coding unit 1300 is obtained from a sample 1340 located in the middle of the current coding unit 1300 and the current coding unit 1300 is divided into a plurality of coding units 1320a, 1320b, and 1320c based on at least one of the block shape information and the partition shape information, the video decoding apparatus 100 may determine the coding unit 1320b including the sample 1340 as the central coding unit. However, the information used to determine the coding unit located in the middle is not limited to at least one of block type information and division type information, and various types of information may be used in the process of determining the coding unit located in the middle.

[0473] According to an embodiment, predetermined information for identifying a coding unit at a predetermined position may also be obtained from a predetermined sample included in the coding unit to be determined. Referring to Figure 13, the video decoding apparatus 100 may use at least one of block shape information and partition shape information obtained from a sample at a predetermined position within the current coding unit 1300 (e.g., a sample located at the center of the current coding unit 1300) to determine a coding unit at a predetermined position (e.g., a coding unit located at the center of the current coding unit 1300) among a plurality of coding units 1320a, 1320b, and 1320c obtained by dividing the current coding unit 1300. That is, the video decoding apparatus 100 may determine the sample at the predetermined position by considering the block type of the current coding unit 1300. The video decoding apparatus 100 may determine a coding unit 1320b including a sample from which predetermined information (e.g., at least one of block type information and partition type information) is acquired among a plurality of coding units 1320a, 1320b, and 1320c determined by dividing the current coding unit 1300, and may set a predetermined restriction. Referring to FIG. 13, according to one embodiment, the video decoding apparatus 100 may determine a sample 1340 located in the middle of the current coding unit 1300 as the sample from which predetermined information is acquired, and may set a predetermined restriction in the decoding process for the coding unit 1320b including such sample 1340. However, the position of the sample from which predetermined information is acquired is not limited to the above position, but may also be a sample at any position included in the coding unit 1320b to be determined for setting the restriction.

[0474] According to one embodiment, the position of the sample from which the predetermined information is acquired is also determined depending on the shape of the current coding unit 1300. According to one embodiment, the block shape information may determine whether the shape of the current coding unit is square or non-square, and the position of the sample from which the predetermined information is acquired may be determined depending on the shape. For example, the video decoding apparatus 100 may use at least one of information related to the width and information related to the height of the current coding unit to determine that a sample located on a boundary that divides at least one of the width and height of the current coding unit in half is the sample from which the predetermined information is acquired. For another example, if the block shape information related to the current coding unit indicates that the current coding unit is non-square, the video decoding apparatus 100 may determine that one of the samples adjacent to a boundary that divides the long side of the current coding unit in half is the sample from which the predetermined information is acquired.

[0475] According to an embodiment, when the video decoding apparatus 100 divides a current coding unit into a plurality of coding units, the video decoding apparatus 100 may use at least one of block shape information and partition shape information to determine a coding unit at a predetermined position among the plurality of coding units. According to an embodiment, the video decoding apparatus 100 may acquire at least one of block shape information and partition shape information from samples at predetermined positions included in the coding unit, and may divide the plurality of coding units generated by dividing the current coding unit using at least one of partition shape information and block shape information acquired from samples at predetermined positions included in each of the plurality of coding units. That is, the coding unit may also be recursively divided using at least one of block shape information and partition shape information acquired from samples at predetermined positions included in each of the plurality of coding units. The recursive division process of a coding unit has been described with reference to FIG. 12, and therefore, a detailed description thereof will be omitted.

[0476] According to one embodiment, the video decoding device 100 can divide the current coding unit and determine at least one coding unit, and can determine the order in which such at least one coding unit is decoded by a predetermined block (e.g., the current coding unit).

[0477] FIG. 14 illustrates an order in which multiple coding units are processed when the video decoding device 100 divides the current coding unit and determines multiple coding units, according to one embodiment.

[0478] According to one embodiment, the video decoding device 100 can divide the first coding unit 1400 vertically and determine the second coding units 1410a and 1410b, or divide the first coding unit 1400 horizontally and determine the second coding units 1430a and 1430b, or divide the first coding unit 1400 vertically and horizontally and determine the second coding units 1450a, 1450b, 1450c, and 1450d, based on the block shape information and the partition shape information.

[0479] 14, the video decoding apparatus 100 may determine the order of processing second coding units 1410a and 1410b, which are determined by dividing the first coding unit 1400 in the vertical direction, in the horizontal direction (1410c). The video decoding apparatus 100 may determine the processing order of second coding units 1430a and 1430b, which are determined by dividing the first coding unit 1400 in the horizontal direction, in the vertical direction (1430c). The video decoding apparatus 100 may determine the processing order of second coding units 1450a, 1450b, 1450c, and 1450d, which are determined by dividing the first coding unit 1400 in the vertical and horizontal directions, in a predetermined order (e.g., raster scan order or z scan order 1450e) in which coding units located in one row are processed before coding units located in the next row are processed.

[0480] According to an embodiment, the video decoding apparatus 100 may recursively divide a coding unit (CU). Referring to FIG. 14, the video decoding apparatus 100 may divide a first CU 1400 to determine a plurality of CUs 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d, and may recursively divide each of the determined CUs 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d. The method of dividing the plurality of CUs 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d corresponds to the method of dividing the first CU 1400. 14, the video decoding apparatus 100 may divide the first coding unit 1400 vertically to determine the second coding units 1410a and 1410b, and may further determine whether to divide the second coding units 1410a and 1410b independently.

[0481] In one embodiment, the video decoding device 100 can horizontally divide the second coding unit 1410a on the left side into third coding units 1420a and 1420b, and the second coding unit 1410b on the right side is not divided.

[0482] According to an embodiment, the processing order of the coding units is also determined based on the division process of the coding units. In other words, the processing order of the divided coding units is also determined based on the processing order of the coding units immediately before the division. The video decoding apparatus 100 may determine the processing order of the third coding units 1420a and 1420b, which are determined by dividing the second coding unit 1410a on the left side, independently of the second coding unit 1410b on the right side. Since the second coding unit 1410a on the left side is divided horizontally and the third coding units 1420a and 1420b are determined, the third coding units 1420a and 1420b may also be processed vertically (1420c). In addition, since the processing order of the second coding unit 1410a on the left side and the second coding unit 1410b on the right side corresponds to the horizontal direction 1410c, the third coding units 1420a and 1420b included in the second coding unit 1410a on the left side are processed in the vertical direction (1420c), and then the right coding unit 1410b is processed. The above content is intended to explain a process in which the processing order of coding units is determined based on the coding units before division, and should not be construed as being limited to the above embodiment, but should be construed as being applicable to various methods in which coding units determined by division in various forms are processed independently in a predetermined order.

[0483] FIG. 15 illustrates a process in which, according to one embodiment, the video decoding device 100 determines that the current coding unit is to be divided into an odd number of coding units when the coding units cannot be processed in a predetermined order.

[0484] According to an embodiment, the video decoding apparatus 100 may determine that a current coding unit is divided into an odd number of coding units based on the acquired block shape information and partition shape information. Referring to Figure 15, a square-shaped first coding unit 1500 may be divided into non-square-shaped second coding units 1510a and 1510b, and the second coding units 1510a and 1510b may be independently divided into third coding units 1520a, 1520b, 1520c, 1520d, and 1520e. According to an embodiment, the video decoding apparatus 100 may divide the left coding unit 1510a of the second coding unit horizontally to determine a plurality of third coding units 1520a and 1520b, and may divide the right coding unit 1510b into an odd number of third coding units 1520c, 1520d, and 1520e.

[0485] According to an embodiment, the video decoding apparatus 100 may determine whether there is an odd number of coding units by determining whether the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e are processed in a predetermined order. Referring to FIG. 15, the video decoding apparatus 100 may recursively divide the first coding unit 1500 to determine the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e. The video decoding apparatus 100 may determine whether the first coding unit 1500, the second coding unit 1510a, 1510b, or the third coding unit 1520a, 1520b, 1520c, 1520d, and 1520e are divided into an odd number of coding units according to the division format based on at least one of block format information and partition format information. For example, the coding units located on the right side of the second coding units 1510a and 1510b may be divided into an odd number of third coding units 1520c, 1520d, and 1520e. The order in which the coding units included in the first coding unit 1500 are processed may be a predetermined order (e.g., z-scan order 1530), and the video decoding apparatus 100 may determine whether the third coding units 1520c, 1520d, and 1520e determined by dividing the right second coding unit 1510b into an odd number of coding units satisfy the condition for processing in the predetermined order.

[0486] According to one embodiment, the video decoding device 100 can determine whether the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e included in the first coding unit 1500 satisfy a condition to be processed in a predetermined order, and the condition relates to whether at least one of the width and height of the second coding units 1510a and 1510b is divided in half by the boundary of the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e. For example, the third coding units 1520a and 1520b, which are determined by dividing the height of the non-square left second coding unit 1510a in half, satisfy the condition, but the boundaries of the third coding units 1520c, 1520d, and 1520e, which are determined by dividing the right second coding unit 1510b into three coding units, do not allow the width or height of the right second coding unit 1510b to be divided in half, so it is determined that the third coding units 1520c, 1520d, and 1520e cannot satisfy the condition.If such a condition is not satisfied, the video decoding device 100 determines that there is a disconnection in the scan order, and based on the determination result, it can determine that the right second coding unit 1510b is to be divided into an odd number of coding units. According to one embodiment, when a video decoding device 100 is divided into an odd number of coding units, it may impose a certain restriction on a coding unit at a predetermined position among the divided coding units. The content of such a restriction or the predetermined position has been described through various embodiments, so detailed description thereof will be omitted.

[0487] 16 illustrates a process in which the video decoding apparatus 100 divides a first coding unit 1600 and determines at least one coding unit, according to an embodiment. According to an embodiment, the video decoding apparatus 100 may divide the first coding unit 1600 based on at least one of block shape information and partition shape information acquired via the acquisition unit 110. The square-shaped first coding unit 1600 may be divided into four square-shaped coding units or into a plurality of non-square coding units. For example, referring to FIG. 16, if the block shape information indicates that the first coding unit 1600 is square and the partition shape information indicates that the first coding unit 1600 is to be divided into non-square coding units, the video decoding apparatus 100 divides the first coding unit 1600 into a plurality of non-square coding units. Specifically, if the division form information indicates that the first coding unit 1600 is to be divided horizontally or vertically to determine an odd number of coding units, the video decoding device 100 can divide the square-shaped first coding unit 1600 into second coding units 1610a, 1610b, and 1610c determined by dividing it vertically, or second coding units 1620a, 1620b, and 1620c determined by dividing it horizontally, as an odd number of coding units.

[0488] According to an embodiment, the video decoding apparatus 100 may determine whether the second coding units 1610a, 1610b, 1610c, 1620a, 1620b, and 1620c included in the first coding unit 1600 satisfy the condition of being processed in a predetermined order, where the condition relates to whether at least one of the width and height of the first coding unit 1600 is divided in half by the boundaries of the second coding units 1610a, 1610b, 1610c, 1620a, 1620b, and 1620c. Referring to Figure 16, the boundaries of the second coding units 1610a, 1610b, and 1610c determined by dividing the square-shaped first coding unit 1600 vertically do not divide the width of the first coding unit 1600 in half, so it may be determined that the first coding unit 1600 does not satisfy the condition of being processed in a predetermined order. In addition, since the boundaries of the second coding units 1620a, 1620b, and 1620c determined by dividing the square-shaped first coding unit 1600 horizontally do not divide the width of the first coding unit 1600 in half, it is determined that the first coding unit 1600 does not satisfy the condition for being processed in a predetermined order. If such a condition is not satisfied, the video decoding apparatus 100 determines that the scan order is broken and may determine to divide the first coding unit 1600 into an odd number of coding units based on the determination result. According to an embodiment, when dividing the first coding unit 1600 into an odd number of coding units, the video decoding apparatus 100 may impose a predetermined restriction on coding units at predetermined positions among the divided coding units. The content of such a restriction or the predetermined position has been described in various embodiments, and therefore a detailed description thereof will be omitted.

[0489] According to an embodiment, the video decoding apparatus 100 may divide the first coding unit and determine various types of coding units.

[0490] Referring to FIG. 16, the video decoding apparatus 100 may divide a square-shaped first coding unit 1600, a non-square-shaped first coding unit 1630, or a non-square-shaped first coding unit 1650 into various types of coding units.

[0491] Figure 17 illustrates that, according to one embodiment, the video decoding device 100 restricts the manner in which the second coding unit may be divided when the non-square second coding unit determined by dividing the first coding unit 1700 satisfies certain conditions.

[0492] According to an embodiment, the video decoding apparatus 100 may determine to divide the square-shaped first coding unit 1700 into non-square-shaped second coding units 1710a, 1710b, 1720a, and 1720b based on at least one of the block shape information and the partition shape information acquired via the acquisition unit 105. The second coding units 1710a, 1710b, 1720a, and 1720b may be divided independently. Thus, the video decoding apparatus 100 may determine whether to divide the first coding unit 1700 into multiple coding units based on at least one of the block shape information and the partition shape information associated with each of the second coding units 1710a, 1710b, 1720a, and 1720b. According to an embodiment, the video decoding apparatus 100 may determine third coding units 1712a and 1712b by horizontally dividing the non-square left second coding unit 1710a, which is determined by vertically dividing the first coding unit 1700. However, when the left second coding unit 1710a is horizontally divided, the video decoding apparatus 100 may restrict the right second coding unit 1710b so that it is not divided horizontally in the same direction as the left second coding unit 1710a. If the right second coding unit 1710b is divided in the same direction to determine third coding units 1714a and 1714b, the left second coding unit 1710a and the right second coding unit 1710b may be horizontally divided independently to determine third coding units 1712a, 1712b, 1714a, and 1714b. However, this is the same result as if the video decoding device 100 had divided the first coding unit 1700 into four square-shaped second coding units 1730a, 1730b, 1730c, and 1730d based on at least one of the block shape information and the partition shape information, which is inefficient in terms of video decoding.

[0493] According to one embodiment, the video decoding apparatus 100 may vertically divide the non-square second coding unit 1720a or 1720b, which is determined by horizontally dividing the first coding unit 11300, to determine the third coding units 1722a, 1722b, 1724a, and 1724b. However, if the video decoding apparatus 100 vertically divides one of the second coding units (e.g., the top second coding unit 1720a), for the reasons described above, the video decoding apparatus 100 may restrict the other second coding units (e.g., the bottom coding unit 1720b) ​​from being vertically divided in the same direction as the top second coding unit 1720a.

[0494] FIG. 18 illustrates a process in which the video decoding device 100 divides a square-shaped coding unit when the division form information cannot indicate that the square-shaped coding unit is divided into four square-shaped coding units, according to one embodiment.

[0495] According to an embodiment, the video decoding apparatus 100 may divide the first coding unit 1800 into second coding units 1810a, 1810b, 1820a, and 1820b based on at least one of block shape information and partition shape information. The partition shape information may include information regarding various types of division of the coding unit, but the information regarding the various types may not include information for division into four square-shaped coding units. According to such partition shape information, the video decoding apparatus 100 may not divide the square-shaped first coding unit 1800 into four square-shaped second coding units 1830a, 1830b, 1830c, and 1830d. ​​Based on the partition shape information, the video decoding apparatus 100 may determine non-square-shaped second coding units 1810a, 1810b, 1820a, and 1820b.

[0496] According to an embodiment, the video decoding apparatus 100 may independently divide the non-square second coding units 1810a, 1810b, 1820a, and 1820b, respectively, in a recursive manner, in which the second coding units 1810a, 1810b, 1820a, and 1820b are divided in a predetermined order, which corresponds to the division of the first coding unit 1800 based on at least one of the block shape information and the partition shape information.

[0497] For example, the video decoding device 100 may horizontally divide the left-side second coding unit 1810a into square-shaped third coding units 1812a and 1812b, and may horizontally divide the right-side second coding unit 1810b into square-shaped third coding units 1814a and 1814b. Furthermore, the video decoding device 100 may horizontally divide both the left-side second coding unit 1810a and the right-side second coding unit 1810b into square-shaped third coding units 1816a, 1816b, 1816c, and 1816d. In this case, the coding units may be determined in the same manner as when the first coding unit 1800 is divided into four square-shaped second coding units 1830a, 1830b, 1830c, and 1830d.

[0498] For another example, the video decoding device 100 may vertically divide the top second coding unit 1820a to determine square third coding units 1822a and 1822b, and may vertically divide the bottom second coding unit 1820b to determine square third coding units 1824a and 1824b. Furthermore, the video decoding device 100 may vertically divide both the top second coding unit 1820a and the bottom second coding unit 1820b to determine square third coding units 1822a, 1822b, 1824a, and 1824b. In this case, the coding units may be determined in the same manner as when the first coding unit 1800 is divided into four square second coding units 1830a, 1830b, 1830c, and 1830d.

[0499] FIG. 19 illustrates that the processing order of multiple coding units may vary depending on the division process of the coding units, according to an embodiment.

[0500] According to an embodiment, the video decoding apparatus 100 may partition the first coding unit 1900 based on the block shape information and the partition shape information. If the block shape information indicates a square shape and the partition shape information indicates that the first coding unit 1900 is to be partitioned in at least one of the horizontal and vertical directions, the video decoding apparatus 100 may partition the first coding unit 1900 to determine, for example, second coding units 1910a, 1910b, 1920a, 1920b, 1930a, 1930b, 1930c, and 1930d. Referring to FIG. 19, the non-square second coding units 1910a, 1910b, 1920a, and 1920b determined by partitioning the first coding unit 1900 only in the horizontal or vertical direction may be independently partitioned based on the block shape information and partition shape information associated therewith. For example, the video decoding apparatus 100 may horizontally divide second coding units 1910a and 1910b, which are generated by vertically dividing the first coding unit 1900, into third coding units 1916a, 1916b, 1916c, and 1916d, and may horizontally divide second coding units 1920a and 1920b, which are generated by horizontally dividing the first coding unit 1900, into third coding units 1926a, 1926b, 1926c, and 1926d. The division process for the second coding units 1910a, 1910b, 1920a, and 1920b has been described with reference to FIG. 17, and therefore, detailed description thereof will be omitted.

[0501] According to an embodiment, the video decoding device 100 may process coding units in a predetermined order. The characteristics related to processing coding units in a predetermined order have been described with reference to FIG. 14, and therefore detailed description thereof will be omitted. Referring to FIG. 19, the video decoding device 100 may divide a square-shaped first coding unit 1900 to determine four square-shaped third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d. According to an embodiment, the video decoding device 100 may determine the processing order of the third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d depending on the division form of the first coding unit 1900.

[0502] According to one embodiment, the video decoding device 100 can horizontally divide the second coding units 1910a and 1910b, which are generated by vertical division, to determine the third coding units 1916a, 1916b, 1916c, and 1916d. The video decoding device 100 can process the third coding units 1916a, 1916b, 1916c, and 1916d in an order (1917) of first vertically processing the third coding units 1916a and 1916b included in the left second coding unit 1910a, and then vertically processing the third coding units 1916c and 1916d included in the right second coding unit 1910b.

[0503] According to one embodiment, the video decoding device 100 can vertically divide the second coding units 1920a and 1920b, which are generated by dividing them horizontally, to determine the third coding units 1926a, 1926b, 1926c, and 1926d. The video decoding device 100 can process the third coding units 1926a, 1926b, 1926c, and 1926d in the order (1927) of first horizontally processing the third coding units 1926a and 1926b included in the top second coding unit 1920a, and then horizontally processing the third coding units 1926c and 1926d included in the bottom second coding unit 1920b.

[0504] 19, second coding units 1910a, 1910b, 1920a, and 1920b are each divided to determine square-shaped third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d. Although the second coding units 1910a and 1910b determined by vertical division and the second coding units 1920a and 1920b determined by horizontal division are divided in different ways, the third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d determined later ultimately result in the division of the first coding unit 1900 into coding units of the same shape. As a result, the video decoding device 100 recursively divides the coding units through different processes based on at least one of block type information and division type information, and as a result, even if coding units of the same type are determined, multiple coding units determined to be of the same type can be processed in different orders from each other.

[0505] FIG. 20 illustrates a process of determining the depth of a coding unit according to one embodiment when a coding unit is recursively divided to determine multiple coding units, as the shape and size of the coding unit change.

[0506] According to an embodiment, the video decoding apparatus 100 may determine the depth of a coding unit based on a predetermined criterion. For example, the predetermined criterion may be the length of a long side of the coding unit. When the length of the long side of the current coding unit is 2n (n>0) times the length of the long side of the coding unit before division, the video decoding apparatus 100 may determine that the depth of the current coding unit is increased by n from the depth of the coding unit before division. Hereinafter, a coding unit whose depth has been increased will be referred to as a coding unit of a lower depth.

[0507] 20, according to an embodiment, the video decoding apparatus 100 may divide the square-shaped first coding unit 2000 based on block shape information indicating a square shape (e.g., the block shape information may indicate "0:SQUARE"), and determine a second coding unit 2002, a third coding unit 2004, etc., at a lower depth. If the size of the square-shaped first coding unit 2000 is 2Nx2N, the width and height of the first coding unit 2000 may be halved. 1 The second coding unit 2002 determined by dividing the width and height of the second coding unit 2002 by 1 / 2 may have a size of NxN. Furthermore, the third coding unit 2004 determined by dividing the width and height of the second coding unit 2002 by 1 / 2 may have a size of N / 2xN / 2. In this case, the width and height of the third coding unit 2004 are 1 / 2 of the width and height of the first coding unit 2000. 2 If the depth of the first coding unit 2000 is D, then 1 / 2 of the width and height of the first coding unit 2000 1 The depth of the second coding unit 2002 is also D+1, which is half the width and height of the first coding unit 2000. 2 The depth of the third coding unit 2004, which is double the depth, is also D+2.

[0508] In one embodiment, based on block shape information indicating a non-square shape (for example, the block shape information may indicate "1:NS_VER", indicating a non-square shape in which the height is longer than the width, or "2:NS_HOR", indicating a non-square shape in which the width is longer than the height), the video decoding device 100 may divide the first coding unit 2010 or 2020, which is non-square, and determine the second coding unit 2012 or 2022 and the third coding unit 2014 or 2024 of a lower depth.

[0509] The video decoding apparatus 100 may divide at least one of the width and height of the first coding unit 2010 having a size of Nx2N to determine, for example, second coding units 2002, 2012, and 2022. That is, the video decoding apparatus 100 may divide the first coding unit 2010 horizontally to determine the second coding unit 2002 having a size of NxN or the second coding unit 2022 having a size of NxN / 2, or may divide the first coding unit 2010 horizontally and vertically to determine the second coding unit 2012 having a size of N / 2xN.

[0510] According to an embodiment, the video decoding apparatus 100 may divide at least one of the width and height of the first coding unit 2020 having a size of 2NxN to determine, for example, second coding units 2002, 2012, and 2022. That is, the video decoding apparatus 100 may divide the first coding unit 2020 vertically to determine the second coding unit 2002 having a size of NxN or the second coding unit 2012 having a size of N / 2xN, or may divide the first coding unit 2020 horizontally and vertically to determine the second coding unit 2022 having a size of NxN / 2.

[0511] According to an embodiment, the video decoding apparatus 100 may divide at least one of the width and height of the second coding unit 2002 having an NxN size, for example, to determine third coding units 2004, 2014, and 2024. That is, the video decoding apparatus 100 may divide the second coding unit 2002 vertically and horizontally to determine the third coding unit 2004 having an N / 2xN / 2 size, or may divide the second coding unit 2002 vertically and horizontally to determine the third coding unit 2004 having an N / 2xN / 2 size. 2 Determine the third coding unit 2014 of size N / 2xN / 2, or 2 A third coding unit of size 2024 can be determined.

[0512] According to an embodiment, the video decoding apparatus 100 may divide at least one of the width and height of the second coding unit 2012 having a size of N / 2xN, for example, to determine third coding units 2004, 2014, and 2024. That is, the video decoding apparatus 100 may divide the second coding unit 2012 horizontally to determine third coding units 2004, 2014, and 2024 having a size of N / 2xN / 2. 2 Determine the third coding unit of size 2024 or divide it vertically and horizontally and N / 2 2 A third coding unit 2014 of size xN / 2 can be determined.

[0513] According to an embodiment, the video decoding apparatus 100 may divide at least one of the width and height of the second coding unit 2014 having a size of NxN / 2 to determine, for example, third coding units 2004, 2014, and 2024. That is, the video decoding apparatus 100 may divide the second coding unit 2012 vertically to determine the third coding unit 2004 having a size of N / 2xN / 2 or the third coding unit 2014 having a size of N / 2xN / 2. 2 The third coding unit 2014 of size N / 2xN / 2 is determined, or divided vertically and horizontally, and 2 A third coding unit of size 2024 can be determined.

[0514] According to one embodiment, the video decoding device 100 may divide, for example, square coding units 2000, 2002, and 2004 horizontally or vertically. For example, the first coding unit 2000 having a size of 2Nx2N may be divided vertically to determine a first coding unit 2010 having a size of Nx2N, or may be divided horizontally to determine a first coding unit 2020 having a size of 2NxN. According to one embodiment, if the depth is determined based on the length of the longest side of the coding unit, the depth of the coding unit determined by dividing the first coding unit 2000, 2002, or 2004 having a size of 2Nx2N horizontally or vertically may be the same as the depth of the first coding unit 2000, 2002, or 2004.

[0515] In one embodiment, the width and height of the third coding unit 2014 or 2024 are half that of the first coding unit 2010 or 2020. 2 If the depth of the first coding unit 2010 or 2020 is D, the depth of the second coding unit 2012 or 2014, which is 1 / 2 times the width and height of the first coding unit 2010 or 2020, is also D+1, which is 1 / 2 the width and height of the first coding unit 2010 or 2020. 2 The depth of the third coding unit 2014 or 2024, which is double the depth, is also D+2.

[0516] FIG. 21 illustrates a depth determined depending on the type and size of a coding unit and an index (PID: part index) for distinguishing the coding unit, according to one embodiment.

[0517] According to an embodiment, the video decoding apparatus 100 may determine various types of second coding units by dividing a square-shaped first coding unit 2100. Referring to Figure 21, the video decoding apparatus 100 may divide the first coding unit 2100 in at least one of the vertical and horizontal directions according to the division type information, and determine second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d. That is, the video decoding apparatus 100 may determine the second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d based on the division type information related to the first coding unit 2100.

[0518] According to one embodiment, the depths of the second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d determined by the division type information for the square-shaped first coding unit 2100 may be determined based on the lengths of their long sides. For example, since the length of one side of the square-shaped first coding unit 2100 is the same as the length of the long sides of the non-square-shaped second coding units 2102a, 2102b, 2104a, and 2104b, the depths of the first coding unit 2100 and the non-square-shaped second coding units 2102a, 2102b, 2104a, and 2104b can be considered to be the same as D. In contrast, when the video decoding device 100 divides the first coding unit 2100 into four square-shaped second coding units 2106a, 2106b, 2106c, and 2106d based on the division form information, the length of one side of the square-shaped second coding units 2106a, 2106b, 2106c, and 2106d is half the length of one side of the first coding unit 2100, so the depth of the second coding units 2106a, 2106b, 2106c, and 2106d is also a depth of D+1, which is one depth lower than the depth D of the first coding unit 2100.

[0519] According to an embodiment, the video decoding apparatus 100 may divide a first coding unit 2110, whose height is greater than its width, horizontally according to the division format information and divide the first coding unit 2110 into a plurality of second coding units 2112a, 2112b, 2114a, 2114b, and 2114c. According to an embodiment, the video decoding apparatus 100 may divide a first coding unit 2120, whose width is greater than its height, vertically according to the division format information and divide the first coding unit 2120 into a plurality of second coding units 2122a, 2122b, 2124a, 2124b, and 2124c.

[0520] According to an embodiment, the depths of the second coding units 2112a, 2112b, 2114a, 2114b, 2116a, 2116b, 2116c, and 2116d determined by the division type information related to the non-square first coding unit 2110 or 2120 may be determined based on the length of their long sides. For example, since the length of one side of the square second coding units 2112a and 2112b is half the length of one side of the non-square first coding unit 2110, whose height is longer than its width, the depths of the square second coding units 2102a, 2102b, 2104a, and 2104b are D+1, which is one depth lower than the depth D of the non-square first coding unit 2110.

[0521] Furthermore, the video decoding device 100 may divide the non-square first coding unit 2110 into an odd number of second coding units 2114a, 2114b, and 2114c based on the division format information. The odd number of second coding units 2114a, 2114b, and 2114c may include the non-square second coding units 2114a and 2114c and the square second coding unit 2114b. In this case, the length of the long sides of the non-square second coding units 2114a and 2114c and the length of one side of the square second coding unit 2114b are half the length of one side of the first coding unit 2110. Therefore, the depths of the second coding units 2114a, 2114b, and 2114c are also D+1, which is one depth lower than D, the depth of the first coding unit 2110. The video decoding device 100 can determine the depth of a coding unit associated with a non-square first coding unit 2120 whose width is greater than its height in a manner corresponding to the above-mentioned manner for determining the depth of a coding unit associated with the first coding unit 2110.

[0522] According to an embodiment, when determining an index (PID) for partitioning a divided coding unit, the video decoding apparatus 100 may determine the index based on the size ratio between the coding units when the odd number of divided coding units are not the same size. Referring to FIG. 21, the middle coding unit 2114b among the odd number of divided coding units 2114a, 2114b, and 2114c has the same width as the other coding units 2114a and 2114c but is twice as high as the other coding units 2114a and 2114c. That is, the middle coding unit 2114b may include both the other coding units 2114a and 2114c. Therefore, if the index (PID) of the middle coding unit 2114b in the scanning order is 1, the next coding unit 2114c has an index (PID) of 3, which is two higher than the other coding units. That is, a discontinuity in the index values ​​exists. According to one embodiment, the video decoding device 100 can determine whether coding units divided into an odd number of parts are not the same size as each other based on whether there is a discontinuity in the indexes for partitions between such divided coding units.

[0523] According to one embodiment, the video decoding apparatus 100 may determine whether a current coding unit has been divided into a specific division type based on the value of an index for distinguishing the plurality of coding units determined by dividing the current coding unit. Referring to FIG. 21, the video decoding apparatus 100 may divide a rectangular first coding unit 2110, whose height is greater than its width, to determine an even number of coding units 2112a and 2112b or an odd number of coding units 2114a, 2114b, and 2114c. The video decoding apparatus 100 may use an index (PID) indicating each coding unit to distinguish each of the plurality of coding units. According to one embodiment, the PID may also be obtained from a sample at a predetermined position (e.g., the top left sample) of each coding unit.

[0524] According to an embodiment, the video decoding apparatus 100 may determine a coding unit at a predetermined position among the coding units determined by division using an index for dividing the coding units. According to an embodiment, if division type information related to a rectangular first coding unit 2110 whose height is greater than its width indicates that the first coding unit 2110 is to be divided into three coding units, the video decoding apparatus 100 may divide the first coding unit 2110 into three coding units 2114a, 2114b, and 2114c. The video decoding apparatus 100 may assign indexes related to the three coding units 2114a, 2114b, and 2114c, respectively. The video decoding apparatus 100 may compare the indexes related to each coding unit to determine a middle coding unit among the odd number of coding units. Based on the indexes of the coding units, the video decoding apparatus 100 may determine the coding unit 2114b having an index corresponding to the middle value among the indexes as the middle coding unit among the coding units determined by dividing the first coding unit 2110. According to an embodiment, when determining indexes for partitioning divided coding units, the video decoding apparatus 100 may determine indexes based on the size ratio between the coding units if the coding units are not the same size. Referring to FIG. 21, a coding unit 2114b generated by dividing a first coding unit 2110 has the same width as other coding units 2114a and 2114c but is twice the height of the other coding units 2114a and 2114c, which have different heights. In this case, if the index (PID) of the middle coding unit 2114b is 1, the next coding unit 2114c has an index of 3, which is increased by 2. In such a case, when the indexes increase uniformly but by different increments, the video decoding apparatus 100 may determine that the coding unit has been divided into multiple coding units, including coding units having different sizes from the other coding units.According to one embodiment, when the division type information indicates division into an odd number of coding units, the video decoding apparatus 100 may divide the current coding unit such that a coding unit at a predetermined position among the odd number of coding units (e.g., a middle coding unit) has a different size from the other coding units. In this case, the video decoding apparatus 100 may determine the middle coding unit having a different size using an index (PID) associated with the coding unit. However, the index and the size or position of the coding unit at the predetermined position to be determined are specific for purposes of describing one embodiment, and should not be construed as being limited thereto, and various indexes, positions, and sizes of coding units may be used.

[0525] According to one embodiment, the video decoding device 100 can utilize a predetermined data unit from which the recursive division of the coding units begins.

[0526] FIG. 22 illustrates a plurality of coding units determined from a plurality of predetermined data units included in a picture, according to one embodiment.

[0527] According to an embodiment, the predetermined data unit is also defined as a data unit from which a coding unit begins to be recursively divided using at least one of block type information and partition type information. That is, the predetermined data unit corresponds to a coding unit of the highest depth used in the process of determining a plurality of coding units for dividing a current picture. Hereinafter, for convenience of explanation, such a predetermined data unit will be referred to as a reference data unit.

[0528] According to one embodiment, the reference data unit may have a predetermined size and shape. According to one embodiment, the reference coding unit may include MxN samples, where M and N may be the same or may be integers expressed as a power of 2. That is, the reference data unit may have a square or non-square shape and may then be divided into an integer number of coding units.

[0529] According to an embodiment, the video decoding apparatus 100 may divide a current picture into a plurality of reference data units. According to an embodiment, the video decoding apparatus 100 may divide the current picture into a plurality of reference data units using partition information associated with each reference data unit. Such a division process of the reference data units corresponds to a division process using a quad-tree structure.

[0530] According to an embodiment, the video decoding apparatus 100 may determine in advance a minimum size that a reference data unit included in a current picture may have, and may determine reference data units of various sizes that are equal to or larger than the minimum size, and may determine at least one coding unit based on the determined reference data unit using block shape information and partition shape information.

[0531] 22, the video decoding apparatus 100 may use a square-shaped reference coding unit 2200 or a non-square-shaped reference coding unit 2202. According to an embodiment, the shape and size of the reference coding unit may be determined based on various data units (e.g., a sequence, a picture, a slice, a slice segment, a maximum coding unit, etc.) that include at least one reference coding unit.

[0532] According to an embodiment, the acquiring unit 105 of the video decoding apparatus 100 may acquire at least one of information regarding the type of the reference coding unit and information regarding the size of the reference coding unit from the bitstream for each of the various data units. The process of determining at least one coding unit included in the square-shaped reference coding unit 2200 has been described through the process of dividing the current coding unit 300 in Fig. 10, and the process of determining at least one coding unit included in the non-square-shaped reference coding unit 2200 has been described through the process of dividing the current coding unit 1100 or 1150 in Fig. 11, so detailed descriptions thereof will be omitted.

[0533] According to an embodiment, the video decoding apparatus 100 may use an index for identifying the size and type of a reference coding unit to determine the size and type of the reference coding unit according to some data units that are predetermined based on a predetermined condition. That is, the acquiring unit 105 may acquire only an index for identifying the size and type of the reference coding unit for each slice, slice segment, or maximum coding unit as a data unit that satisfies a predetermined condition (e.g., a data unit having a size equal to or smaller than a slice) among the various data units (e.g., a sequence, a picture, a slice, a slice segment, a maximum coding unit, etc.) from the bitstream. The video decoding apparatus 100 may determine the size and type of the reference data unit for each data unit that satisfies the predetermined condition by using the index. If information related to the type of the reference coding unit and information related to the size of the reference coding unit were acquired from the bitstream for each data unit of a relatively small size, the bitstream utilization efficiency would be poor. Therefore, instead of directly acquiring information related to the type of the reference coding unit and information related to the size of the reference coding unit, only the index may be acquired and used. In this case, at least one of the size and type of the reference coding unit corresponding to the index indicating the size and type of the reference coding unit is predetermined. That is, the video decoding apparatus 100 may determine at least one of the size and type of the reference coding unit included in the data unit that is the basis for obtaining the index by selecting at least one of the size and type of the predetermined reference coding unit according to the index.

[0534] According to one embodiment, the video decoding apparatus 100 may use at least one reference coding unit included in one maximum coding unit. That is, the maximum coding unit into which an image is divided may include at least one reference ...

Claims

1. 1. A method for decoding a video, comprising: obtaining a first extended reference block in a first reference picture and a second extended reference block in a second reference picture; the first extended reference block includes a first reference block and a first portion extended from the first reference block; the second extended reference block includes a second reference block and a second portion extended from the second reference block; determining a displacement vector of a pixel group including at least one pixel adjacent to an inner boundary of the current block using a gradient value of at least one reference pixel in the first extended reference block and a gradient value of at least one reference pixel in the second extended reference block; the first portion of the first extended reference block is used to calculate a gradient value of at least one reference pixel in the first reference block; the second portion of the second extended reference block is used to calculate a gradient value of at least one reference pixel in the second reference block; obtaining a predicted pixel value of the current block by performing optical flow-based compensation on the current block using the gradient value of the at least one reference pixel in the first reference block, the gradient value of the at least one reference pixel in the second reference block, and a displacement vector of the pixel group; reconstructing the current block based on the predicted pixel values; The current picture is divided into a plurality of maximum coding units, any one of the plurality of largest coding units is divided into at least one coding unit including the current block; Video decoding methods.

2. 1. A method for encoding video, comprising: obtaining a predicted pixel value of the current block by performing optical flow-based compensation on the current block using a gradient value of at least one reference pixel in a first reference block, a gradient value of at least one reference pixel in a second reference block, and a displacement vector of a pixel group of the current block; the pixel group includes at least one pixel adjacent to an inner boundary of the current block; generating a bitstream including a result of encoding the current block based on the predicted pixel values; a first extended reference block is obtained in a first reference picture, a second extended reference block is obtained in a second reference picture, the first extended reference block includes the first reference block and a first portion extended from the first reference block, and the second extended reference block includes the second reference block and a second portion extended from the second reference block; the displacement vector of the pixel group is determined using a gradient value of at least one reference pixel in the first extended reference block and a gradient value of at least one reference pixel in the second extended reference block; the first portion of the first extended reference block is used to calculate a gradient value of the at least one reference pixel in the first reference block; the second portion of the second extended reference block is used to calculate a gradient value of the at least one reference pixel in the second reference block; The current picture is divided into multiple maximum coding units, any one of the plurality of largest coding units is divided into at least one coding unit including the current block; Video coding methods.

3. A method for transmitting the bitstream produced by the video encoding method of claim 2.

Images