Video decoding method and apparatus, and video encoding method and apparatus

Abstract

To provide a video decoding method for obtaining a prediction block of a current block.SOLUTION: A method uses a value related to a reference pixel included in a block without using a value stored relative to a pixel located outside a boundary of first and second reference blocks and determines a displacement vector per unit time in a horizontal direction or a vertical direction of a pixel of a current block including a pixel adjacent to an inside of a boundary of the current block. Based on a horizontal or vertical gradient value of a second corresponding reference pixel in the first / second reference blocks corresponding to a current pixel included in a current pixel group in the current block, the pixel values of the first / second corresponding reference pixels, and the horizontal or vertical displacement vector per unit time of the current pixel, motion compensation in units of blocks and motion compensation in units of pixel groups of the current block are performed. The pixel group may include at least one pixel.SELECTED DRAWING: Figure 1B

Description

[Technical field]

[0001] The present invention relates to a video decoding method and video encoding, and more particularly to a video decoding method and video encoding method that perform inter prediction in a bi-directional motion prediction mode. [Background technology]

[0002] Due to the development and spread of hardware capable of playing and storing high-resolution or high-quality video content, the need for a video codec that can effectively encode and decode high-resolution or high-quality video content is increasing. According to existing video codecs, video is encoded by a restricted encoding scheme based on a tree-structured encoding unit.

[0003] Using frequency transformation, spatial domain image data is transformed into frequency domain coefficients. In order to perform a fast calculation of frequency transformation, the video codec divides an image into blocks of a predetermined size, performs DCT transformation on each block, and encodes frequency coefficients on a block-by-block basis. Compared to spatial domain image data, frequency domain coefficients are easily compressed. In particular, since spatial domain image pixel values ​​are expressed as prediction errors through inter-prediction or intra-prediction of the video codec, many data are converted to 0 when frequency transformation is performed on the prediction errors. The video codec saves data by replacing data that is generated continuously and repeatedly with small-sized data. Summary of the Invention [Problem to be solved by the invention]

[0004] According to various embodiments, in a bidirectional motion prediction mode, a predicted pixel value of a current block is generated by using not only a pixel value of a first reference block of a first reference picture and a pixel value of a second reference block of a second reference picture, but also a first gradient value of the first reference block and a second gradient value of the second reference block together. Therefore, a predicted block similar to an original block is generated, so that encoding / decoding efficiency can be improved.

[0005] The pixel value of the first reference block, the pixel value of the second reference block, the first gradient value of the first reference block, and the second gradient value of the second reference block are used to determine a horizontal or vertical displacement vector of the current block when performing pixel group unit motion compensation. In particular, in order to determine a horizontal or vertical displacement vector of a current pixel in the current block, not only the pixel value and gradient value of the first reference pixel in the first reference block corresponding to the current pixel and the pixel value and gradient value of the second reference pixel in the second reference block are used, but also the pixel values ​​and gradient values ​​of surrounding pixels included in a window of a predetermined size centered on the first and second reference pixels are used. Therefore, when the current pixel is located on a boundary, the surrounding pixels of the reference pixel corresponding to the current pixel are located outside the reference block, so that the pixel values ​​and gradient values ​​of the pixels located outside the reference block must be referenced, which causes a problem of additional memory access.

[0006] According to various embodiments, the number of memory accesses can be minimized by determining a horizontal or vertical displacement vector of the current block by referring only to pixel values ​​and gradient values ​​of pixels located inside the reference block, without referring to pixel values ​​and gradient values ​​stored for pixels located outside the reference block.

[0007] According to various embodiments, instead of using pixel values ​​of integer pixels as input and horizontal and vertical gradient filters and interpolation filters to determine the horizontal or vertical gradient value of a reference pixel, an interpolation filter is applied to the pixel values ​​of the integer pixels to determine pixel values ​​of pixels located at fractional pixel positions, and a horizontal or vertical gradient filter having a relatively short filter length is applied to the pixel values ​​of the pixels located at fractional pixel positions to determine the horizontal or vertical gradient value of the reference pixel, thereby minimizing the need for complex multiplication operations.

[0008] According to various embodiments, by performing motion compensation on a pixel group basis, it is possible to minimize the need for more complex multiplication operations that would be required when performing motion compensation on a pixel basis.

[0009] The present invention may include a computer readable recording medium having a program recorded thereon for implementing the methods according to various embodiments.

[0010] Here, the technical problems of the various embodiments are not limited to the features mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0011] The technical problem of the present invention is not limited to the features mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

[0012] According to various embodiments, a video decoding method includes the steps of: obtaining motion prediction mode information for a current block in a current picture from a bitstream; obtaining, from the bitstream, information on a first motion vector indicating a first reference block of a current block in a first reference picture and information on a second motion vector indicating a second reference block of the current block in a second reference picture, when the obtained motion prediction mode information indicates a bi-directional motion prediction mode; determining a displacement vector per unit time in a horizontal or vertical direction of a pixel of a current block including an adjacent pixel inside a boundary of the current block by using values ​​related to reference pixels included in the first reference block and the second reference block without using values ​​stored for pixels located outside the boundaries of the first reference block and the second reference block; performing block-based motion compensation and pixel group-based motion compensation on the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to a current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel; obtaining information about a residual block of a current block from the bitstream; reconstructing the current block based on the predicted block and the residual block; The pixel group may include at least one pixel.

[0013] In various embodiments of the video decoding method, the step of obtaining a prediction block of the current block includes: The method further includes applying a horizontal or vertical gradient filter to a pixel value of a sub-position pixel in the first reference block or the second reference block to calculate a horizontal or vertical gradient value of the first corresponding reference pixel or the second corresponding reference pixel; the gradient filter is a 5-tap filter; The fractional position pixel is also a pixel in which at least one of the horizontal and vertical components of the coordinates indicating the position of the pixel has a fractional value.

[0014] In the video decoding method according to various embodiments, the pixel value of the sub-position pixel in the first reference block or the second reference block may be calculated by applying a horizontal or vertical interpolation filter to the pixel value of the integer position pixel.

[0015] In the video decoding method according to various embodiments, the size of the pixel group is also determined based on the minimum value of the height and width of the current block.

[0016] In various embodiments of the video decoding method, the horizontal or vertical displacement vector per unit time for the current pixel group is also a displacement vector per unit time determined using a value determined based on pixel value gradient values ​​of a first corresponding reference pixel and its surrounding pixels included in a first corresponding reference pixel group in a first reference picture corresponding to the current pixel group and a second corresponding reference pixel and its surrounding pixels included in a second corresponding reference pixel group in a second reference picture, a first POC (picture order count) difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture.

[0017] In the video decoding method according to various embodiments, the step of determining a displacement vector per unit time in a horizontal or vertical direction of a pixel of the current block includes: When the first corresponding reference pixel or the second corresponding reference pixel is a boundary peripheral pixel adjacent to the inside of the boundary of the first reference block or the second reference block, deriving pixel values ​​and gradient values ​​of pixels located outside the boundary of the first reference block or the second reference block using the pixel values ​​of the boundary peripheral pixels; and determining a displacement vector per unit time in a horizontal or vertical direction of the current pixel based on pixel values ​​and gradient values ​​of the boundary-periphery pixels, and pixel values ​​and gradient values ​​of pixels located outside the boundary of the current block derived using the pixel values ​​of the boundary-periphery pixels.

[0018] In the video decoding method according to various embodiments, the step of determining a displacement vector per unit time in a horizontal or vertical direction of a pixel of the current block includes: calculating a value related to the current pixel using a pixel value of a first corresponding reference pixel included in the first reference block, a pixel value of a second corresponding reference pixel included in the second reference block, a gradient value of the first corresponding reference pixel, a gradient value of the second corresponding reference pixel, a first POC difference between the first reference picture and a current picture, and a second POC difference between the second reference picture and a current picture; calculating values ​​related to the neighboring pixels calculated using a pixel value of a first corresponding neighboring pixel of the first corresponding reference pixel, a pixel value of a second corresponding neighboring pixel of the second corresponding reference pixel, a gradient value of the first corresponding neighboring pixel, a gradient value of the second corresponding neighboring pixel, a first POC difference between the first reference picture and a current picture, and a second POC difference between the second reference picture and a current picture; calculating a weighted average value for the current pixel, which is necessary to calculate a displacement vector per unit time in a horizontal or vertical direction, using a value for the current pixel, a value for the neighboring pixels, and a weight value; and determining a displacement vector per unit time in a horizontal or vertical direction of the current pixel by using the calculated weighted average value for the current pixel.

[0019] In various embodiments of the video decoding method, the weighted average value for the current pixel is also a value calculated by applying an exponential smoothing technique to values ​​related to pixels included in the first reference block and the second reference block in the up, down, left and right directions.

[0020] According to various embodiments, a video decoding apparatus includes: acquiring motion prediction mode information related to a current block in a current picture from a bitstream; an acquiring unit for acquiring information on a first motion vector indicating a first reference block of a current block in a first reference picture and information on a second motion vector indicating a second reference block of the current block in a second reference picture from the bitstream when the acquired motion prediction mode information indicates a bidirectional motion prediction mode, and acquiring information on a residual block of the current block from the bitstream; an inter prediction unit for determining a displacement vector per unit time in a horizontal or vertical direction of a pixel of a current block including an adjacent pixel inside a boundary of the current block by using values ​​related to reference pixels included in the first and second reference blocks without using values ​​stored for pixels located outside boundaries of the first and second reference blocks, and performing block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to a current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector of the current pixel, thereby obtaining a prediction block of the current block; a decoding unit for decoding the current block based on the prediction block and the residual block, The pixel group may include at least one pixel.

[0021] In various embodiments of the video decoding apparatus, the inter prediction unit includes: Applying a horizontal or vertical gradient filter to a pixel value of a sub-position pixel in the first reference block or the second reference block to calculate a horizontal or vertical gradient value of the first corresponding reference pixel or the second corresponding reference pixel; the gradient filter is a 5-tap filter; The fractional position pixel is also a pixel in which at least one of the horizontal and vertical components of the coordinates indicating the position of the pixel has a fractional value.

[0022] In various embodiments of the video decoding apparatus, the inter prediction unit includes: The horizontal or vertical displacement vector per unit time for the current pixel group is also a displacement vector per unit time determined using a value determined based on pixel value gradient values ​​of a first corresponding reference pixel and its surrounding pixels included in a first corresponding reference pixel group in a first reference picture corresponding to the current pixel group and a second corresponding reference pixel and its surrounding pixels included in a second corresponding reference pixel group in a second reference picture, a first POC difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture.

[0023] In various embodiments of the video decoding apparatus, the inter prediction unit includes: calculating a value related to the current pixel by using a pixel value of a first corresponding reference pixel included in the first reference block, a pixel value of a second corresponding reference pixel included in the second reference block, a gradient value of the first corresponding reference pixel, a gradient value of the second corresponding reference pixel, a first POC difference between the first reference picture and a current picture, and a second POC difference between the second reference picture and a current picture; calculating values ​​related to the neighboring pixels calculated using a pixel value of a first corresponding neighboring pixel of the first corresponding reference pixel, a pixel value of a second corresponding neighboring pixel of the second corresponding reference pixel, a gradient value of the first corresponding neighboring pixel, a gradient value of the second corresponding neighboring pixel, a first POC difference between the first reference picture and a current picture, and a second POC difference between the second reference picture and a current picture; calculating a weighted average value for the current pixel, which is required to calculate a displacement vector per unit time in a horizontal or vertical direction, using a value for the current pixel, a value for the neighboring pixels, and a weight value; The calculated weighted average value for the current pixel can be used to determine a displacement vector per unit time in the horizontal or vertical direction of the current pixel.

[0024] According to various embodiments, a video encoding method includes the steps of: performing block-based motion compensation and pixel group-based motion compensation on a current block to obtain a prediction block, a first motion vector, and a second motion vector of the current block; generating a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether a motion prediction mode for the current block is a bidirectional motion prediction mode; The pixel group includes at least one pixel, the first motion vector is a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block; the second motion vector is a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block; A block-based motion compensation and a pixel group-based motion compensation for the current block are performed based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel, to obtain a prediction block of the current block; The horizontal or vertical displacement vector per unit time of a pixel of the current block, including adjacent pixels inside the boundary of the current block, may be determined using values ​​related to reference pixels included in the first reference block and the second reference block, without using values ​​stored for pixels located outside the boundaries of the first reference block and the second reference block.

[0025] According to various exemplary embodiments, a video encoding apparatus includes an inter prediction unit that performs block-based motion compensation and pixel group-based motion compensation on a current block to obtain a prediction block, a first motion vector, and a second motion vector of the current block; a bitstream generator configured to generate a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether a motion prediction mode for the current block is a bidirectional motion prediction mode, The pixel group includes at least one pixel, the first motion vector is a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block, and the second motion vector is a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block; A block-based motion compensation and a pixel group-based motion compensation for the current block are performed based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel, to obtain a prediction block of the current block; The horizontal or vertical displacement vector per unit time of a pixel of the current block, including adjacent pixels inside the boundary of the current block, may be determined using values ​​related to reference pixels included in the first reference block and the second reference block, without using values ​​stored for pixels located outside the boundaries of the first reference block and the second reference block. The present invention may include a computer readable recording medium having a program recorded thereon for implementing the methods according to various embodiments. Effect of the Invention

[0026] According to various embodiments, in a bidirectional motion prediction mode, inter prediction for a current block can be performed using gradient values ​​of a reference block of a reference picture, and a value similar to the value of the original block of the current block can be predicted, thereby improving encoding / decoding efficiency. [Brief description of the drawings]

[0027] [Figure 1A] 1 is a block diagram of a video decoding device according to various embodiments. [Figure 1B] 4 is a flowchart of a video decoding method according to various embodiments. [Figure 1C] 1 is a block diagram of a video encoding device according to various embodiments. [Figure 1D] 4 is a flowchart of a video encoding method according to various embodiments. [Figure 1E] 4A and 4B are block diagrams of a video decoder according to various embodiments. [Figure 1F] 4A and 4B are block diagrams of a video decoder according to various embodiments. [Diagram 2] 2 is a reference diagram for explaining a process of block-based bidirectional motion prediction and compensation thereof according to an embodiment; [Figure 3A] 1 is a reference diagram for explaining a process of performing pixel-based motion compensation according to an embodiment; [Figure 3B] 1 is a reference diagram for explaining a process of performing pixel-based motion compensation according to an embodiment; [Figure 3C] 1 is a reference diagram for explaining a process of performing pixel-based motion compensation according to an embodiment; [Figure 4] 1 is a reference diagram for explaining a process of calculating gradient values ​​in the horizontal and vertical directions according to an embodiment; [Diagram 5] 11 is a reference diagram for explaining a process of calculating horizontal and vertical gradient values ​​according to another embodiment; FIG. [Figure 6A] 11 is a diagram illustrating a process of determining horizontal and vertical gradient values ​​using a one-dimensional filter according to an embodiment; [Figure 6B] 11 is a diagram illustrating a process of determining horizontal and vertical gradient values ​​using a one-dimensional filter according to an embodiment; [Figure 7A] 4 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values ​​according to one embodiment. [Figure 7B] 4 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values ​​according to one embodiment. [Figure 7C] 4 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values ​​according to one embodiment. [Figure 7D] 4 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values ​​according to one embodiment. [Figure 7E] 4 is a table illustrating filter coefficients for filters utilized to determine pixel values ​​at fractional-pixel locations and horizontal and vertical gradient values ​​according to one embodiment. [Figure 8A] 1 is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel according to an embodiment; [Figure 8B] 1 is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel group according to an embodiment; [Figure 8C]1 is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel according to an embodiment; [Figure 8D] 1 is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel group according to an embodiment; [Figure 9A] 11 is a diagram illustrating a process of determining a gradient value in a horizontal or vertical direction by performing filtering, adding an offset, and performing inverse scaling, according to an embodiment. [Figure 9B] 13 is a diagram illustrating a process of determining a horizontal or vertical gradient value by performing inverse scaling after filtering and adding an offset according to another embodiment. [Figure 9C] 11 is a diagram illustrating an input range required for determining a horizontal displacement vector and a vertical displacement vector in a process of performing pixel-by-pixel motion compensation on a current block. [Figure 9D] 1 is a diagram illustrating an input range of a region used in a process of performing pixel-based motion compensation according to various embodiments; [Figure 9E] 1 is a diagram illustrating an input range of a region used in a process of performing pixel-based motion compensation according to various embodiments; [Figure 9F] 11 is a diagram illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block. [Figure 9G] 13 is a diagram illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block according to another embodiment. [Figure 9H] 13 is a diagram illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block according to another embodiment. [Figure 9I]13 is a diagram illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block according to another embodiment. [Figure 9J] This is a diagram illustrating a process of determining horizontal and vertical displacement vectors for each pixel by applying exponential smoothing techniques up, down, left, and right, without extending the block, by referring to pixel values ​​and gradient values ​​of a reference block in one embodiment. [Figure 9K] 11 is a diagram illustrating a process of determining pixel values ​​of reference pixels in a reference block and horizontal and vertical gradient values ​​using a filter to perform motion compensation for a current block, according to an embodiment. [Figure 9L] 13 is a diagram illustrating a process of determining pixel values ​​of reference pixels in a reference block and horizontal and vertical gradient values ​​using a filter to perform motion compensation for a current block according to another embodiment. [Figure 10] 1 is a diagram illustrating a process of dividing a current coding unit and determining at least one coding unit, according to an embodiment; [Figure 11] 1 is a diagram illustrating a process in which a coding unit having a non-square shape is divided and at least one coding unit is determined, according to an embodiment. [Figure 12] 1 is a diagram illustrating a process of dividing a coding unit based on at least one of block configuration information and division configuration information, according to an embodiment; [Figure 13] 1 is a diagram illustrating a method of determining a predetermined coding unit among an odd number of coding units, according to an embodiment; [Figure 14] 1 is a diagram illustrating an order in which a plurality of coding units are processed when a current coding unit is divided to determine a plurality of coding units, according to an embodiment; [Figure 15]11 is a diagram illustrating a process of determining to divide a current coding unit into an odd number of coding units when the coding units cannot be processed in a predetermined order, according to one embodiment; [Figure 16] 1 is a diagram illustrating a process in which a first coding unit is divided and at least one coding unit is determined, according to an embodiment. [Figure 17] 11 is a diagram illustrating that, according to one embodiment, when a non-square second coding unit determined by dividing a first coding unit satisfies a certain condition, the manner in which the second coding unit may be divided is restricted. [Figure 18] 11 is a diagram illustrating a process of dividing a square-shaped coding unit when division form information does not indicate that the coding unit is divided into four square-shaped coding units, according to one embodiment. [Figure 19] 1 is a diagram illustrating that a processing order among a plurality of coding units may differ depending on a division process of the coding units, according to an embodiment. [Figure 20] 11 is a diagram illustrating a process of determining a depth of a coding unit according to changes in the shape and size of the coding unit when a coding unit is recursively divided to determine a plurality of coding units, according to one embodiment. [Figure 21] 1 is a diagram illustrating a depth determined according to a type and size of a coding unit and an index (PID: part index) for distinguishing the coding unit, according to an embodiment. [Figure 22] 1 is a diagram illustrating a plurality of coding units being determined based on a plurality of predetermined data units included in a picture, according to an embodiment; [Diagram 23] 1 is a diagram illustrating processing blocks that are used as a reference to determine a determination order of reference coding units included in a picture, according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] According to various embodiments, a video decoding method includes the steps of: obtaining motion prediction mode information for a current block in a current picture from a bitstream; obtaining, from the bitstream, information on a first motion vector indicating a first reference block of a current block in a first reference picture and information on a second motion vector indicating a second reference block of the current block in a second reference picture, when the obtained motion prediction mode information indicates a bi-directional motion prediction mode; determining a displacement vector per unit time in a horizontal or vertical direction of a pixel of a current block including an adjacent pixel inside a boundary of the current block by using values ​​related to reference pixels included in the first reference block and the second reference block without using values ​​stored for pixels located outside the boundaries of the first reference block and the second reference block; performing block-based motion compensation and pixel group-based motion compensation on the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to a current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel; obtaining information about a residual block of a current block from the bitstream; reconstructing the current block based on the predicted block and the residual block; The pixel group may include at least one pixel.

[0029] According to various embodiments, a video decoding apparatus includes: acquiring motion prediction mode information related to a current block in a current picture from a bitstream; an acquiring unit for acquiring information on a first motion vector indicating a first reference block of a current block in a first reference picture and information on a second motion vector indicating a second reference block of the current block in a second reference picture from the bitstream when the acquired motion prediction mode information indicates a bidirectional motion prediction mode, and acquiring information on a residual block of the current block from the bitstream; an inter prediction unit for determining a displacement vector per unit time in a horizontal or vertical direction of a pixel of a current block including an adjacent pixel inside a boundary of the current block by using values ​​related to reference pixels included in the first and second reference blocks without using values ​​stored for pixels located outside boundaries of the first and second reference blocks, and performing block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to a current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector of the current pixel, thereby obtaining a prediction block of the current block; a decoding unit for decoding the current block based on the prediction block and the residual block, The pixel group may include at least one pixel.

[0030] According to various embodiments, a video encoding method includes the steps of: performing block-based motion compensation and pixel group-based motion compensation on a current block to obtain a prediction block, a first motion vector, and a second motion vector of the current block; generating a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether a motion prediction mode for the current block is a bidirectional motion prediction mode; The pixel group includes at least one pixel, the first motion vector is a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block; the second motion vector is a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block; A block-based motion compensation and a pixel group-based motion compensation for the current block are performed based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel, to obtain a prediction block of the current block; The horizontal or vertical displacement vector per unit time of a pixel of the current block, including adjacent pixels inside the boundary of the current block, may be determined using values ​​related to reference pixels included in the first reference block and the second reference block, without using values ​​stored for pixels located outside the boundaries of the first reference block and the second reference block.

[0031] According to various exemplary embodiments, a video encoding apparatus includes an inter prediction unit that performs block-based motion compensation and pixel group-based motion compensation on a current block to obtain a prediction block, a first motion vector, and a second motion vector of the current block; a bitstream generator configured to generate a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether a motion prediction mode for the current block is a bidirectional motion prediction mode, The pixel group includes at least one pixel, the first motion vector is a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block, and the second motion vector is a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block; A block-based motion compensation and a pixel group-based motion compensation for the current block are performed based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel, to obtain a prediction block of the current block; The horizontal or vertical displacement vector per unit time of a pixel of the current block, including adjacent pixels inside the boundary of the current block, may be determined using values ​​related to reference pixels included in the first reference block and the second reference block, without using values ​​stored for pixels located outside the boundaries of the first reference block and the second reference block.

[0032] The present invention may include a computer readable recording medium having a program recorded thereon for implementing the methods according to various embodiments.

[0033] Hereinafter, "image" can refer to still or moving video images, i.e., the video itself.

[0034] Hereinafter, a "sample" refers to data assigned to a sampling position of an image and to be processed. For example, in a spatial domain image, a pixel is also a sample.

[0035] Hereinafter, the term "current block" refers to a block of an image to be encoded or decoded.

[0036] FIG. 1A illustrates a block diagram of a video decoding device according to various embodiments.

[0037] The video decoding device 100 according to various embodiments includes an acquisition unit 105, an inter prediction unit 110, and a reconstruction unit 125.

[0038] The acquiring unit 105 receives a bitstream including information on a prediction mode of a current block, information indicating a motion prediction mode of the current block, and information on a motion vector.

[0039] The acquiring unit 105 may acquire information on a prediction mode of a current block, information indicating a motion prediction mode of the current block, and information on a motion vector from a received bitstream. The acquiring unit 105 may also acquire a reference picture index indicating a reference picture among previously decoded pictures from the received bitstream.

[0040] The inter prediction unit 110 performs inter prediction on the current block when the prediction mode of the current block is the inter prediction mode. That is, the inter prediction unit 110 may generate a predicted pixel value of the current block using at least one of pictures decoded before the current picture including the current block. For example, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the inter prediction unit 110 may generate a predicted pixel value of the current block using two pictures decoded before the current picture. That is, when information on the motion prediction mode acquired from the bitstream indicates a bidirectional motion prediction mode, the inter prediction unit 110 may generate a predicted pixel value of the current block using two pictures decoded before the current picture.

[0041] The inter prediction unit 110 may include a block-based motion compensation unit 115 and a pixel group-based motion compensation unit 120 .

[0042] The block-based motion compensation unit 115 may perform block-based motion compensation on a current block.

[0043] The block-based motion compensation unit 115 may determine at least one reference picture from among previously decoded pictures using a reference picture index acquired from a bitstream. In this case, the reference picture index refers to a reference picture index related to each prediction direction including an L0 direction and an L1 direction. Here, the reference picture index related to the L0 direction refers to an index indicating a reference picture from among pictures included in an L0 reference picture list, and the reference picture index related to the L1 direction refers to an index indicating a reference picture from among pictures included in an L1 reference picture list.

[0044] The block-based motion compensation unit 115 may determine a reference block of a current block located in at least one reference picture using information about a motion vector received from a bitstream. Here, a corresponding block in a reference picture corresponding to a current block in a current picture is also a reference block. That is, the block-based motion compensation unit 115 may determine a reference block of a current block using a motion vector indicating a reference block from a current block. Here, the motion vector means a vector indicating a displacement between a reference coordinate of a current block in a current picture and a reference coordinate of a reference block in a reference picture. For example, if the upper left coordinate of a current block is (1,1) and the upper left coordinate of a reference block in a reference picture is (3,3), the motion vector is also (2,2).

[0045] Here, the information on the motion vector may include a difference value of the motion vector, and the block-based motion compensation unit 115 may restore the motion vector using a motion vector predictor and a difference value of the motion vector acquired from a bitstream, and determine a reference block of the current block located in at least one reference picture using the restored motion vector. Here, the difference value of the motion vector means a difference value of the motion vector related to the reference picture related to each prediction direction including the L0 direction and the L1 direction. Here, the difference value of the motion vector related to the L0 direction means a difference value of the motion vector indicating a reference block in a reference picture included in the L0 reference picture list, and the difference value of the motion vector related to the L1 direction means a difference value of the motion vector indicating a reference block in a reference picture included in the L1 reference picture list.

[0046] The block-based motion compensation unit 115 may perform block-based motion compensation for a current block using pixel values ​​of a reference block. The block-based motion compensation unit 115 may perform block-based motion compensation for a current block using reference pixel values ​​in a reference block corresponding to a current pixel in the current block. Here, the reference pixel is a pixel included in the reference block, and a corresponding pixel in the current block corresponding to the current pixel is also the reference pixel.

[0047] The block-based motion compensation unit 115 may perform block-based motion compensation on the current block using a plurality of reference blocks included in a plurality of reference pictures, respectively. For example, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the block-based motion compensation unit 115 may determine two reference pictures among previously coded pictures and determine two reference blocks included in the two pictures.

[0048] The block-based motion compensation unit 115 may perform block-based motion compensation for the current block using pixel values ​​of two reference pixels in two reference blocks. The block-based motion compensation unit 115 may perform block-based motion compensation for the current block using an average value or a weighted sum of pixel values ​​of two reference pixels to generate a block-based motion compensation value.

[0049] The reference position of the reference block may be, but is not limited to, an integer pixel position, and may also be a fractional pixel position. Here, an integer pixel is a pixel whose position component is an integer and located at an integer pixel position. A fractional pixel is a pixel whose position component is a decimal and located at a fractional pixel position.

[0050] For example, if the upper left coordinate of the current block is (1,1) and the motion vector is (2.5,2.5), the upper left coordinate of the reference block in the reference picture is also (3.5,3.5). In this case, the position of the fractional pixel is determined in 1 / 4 pel (pixel element) units or 1 / 16 pel units. Without being limited thereto, the position of the fractional pixel may be determined in various fractional pel units.

[0051] When the reference position of the reference block is a decimal pixel position, the block-based motion compensation unit 115 can apply an interpolation filter to a first surrounding area including a first pixel among the pixels of the first reference block indicated by the first motion vector, and a second surrounding area including a second pixel among the pixels of the second reference block indicated by the second motion vector, to generate a pixel value of the first pixel and a pixel value of the second pixel.

[0052] That is, the reference pixel value in the reference block is also determined using pixel values ​​of surrounding pixels whose components in a predetermined direction are integers, where the predetermined direction may be the horizontal direction or the vertical direction.

[0053] For example, the block-based motion compensation unit 115 may use an interpolation filter to filter pixel values ​​of pixels whose predetermined directional components are integers, determine a reference pixel value using the resultant value, and determine a block-based motion compensation value for a current block using the reference pixel value. The block-based motion compensation value may also be determined using an average value or a weighted sum of reference pixels. In this case, the interpolation filter may be a DCT-based M-tap interpolation filter. The coefficients of the DCT-based M-tap interpolation filter are also derived from DCT and IDCT (inverse discrete cosine transform). In this case, the coefficients of the interpolation filter may be filter coefficients scaled to integer coefficients to reduce real number operations during filtering. In this case, the interpolation filter may be a one-dimensional interpolation filter in the horizontal or vertical direction. For example, when expressing the position of a pixel using x, y orthogonal coordinate components, the horizontal direction means a direction parallel to the x-axis. The vertical direction means a direction parallel to the y-axis.

[0054] The block-based motion compensation unit 115 first uses a one-dimensional interpolation filter in the vertical direction to filter pixel values ​​at integer positions, and then uses a one-dimensional interpolation filter in the horizontal direction to filter the values ​​generated after the filtering to determine reference pixel values ​​at fractional pixel positions.

[0055] On the other hand, when scaled filter coefficients are used, values ​​generated by filtering are larger than values ​​generated by non-scaled filter coefficients, so the block-based motion compensation unit 115 can perform de-scaling on values ​​generated by filtering.

[0056] The block-based motion compensation unit 115 may perform inverse scaling after filtering pixel values ​​at integer positions using a vertical one-dimensional interpolation filter. In this case, the inverse scaling may include bit shifting to the right by the inverse scaling bit count. The inverse scaling bit count is also determined based on the bit depth of the sample of the input image. For example, the inverse scaling bit count is a value obtained by subtracting 8 from the bit depth of the sample.

[0057] Also, the block-based motion compensation unit 115 may filter pixel values ​​at integer positions using a vertical one-dimensional interpolation filter, filter the filtered values ​​using a horizontal one-dimensional interpolation filter, and then perform inverse scaling. In this case, the inverse scaling may include bit shifting to the right by the inverse scaling bit number. The inverse scaling bit number is also determined based on the scaling bit number of the vertical one-dimensional interpolation filter, the scaling bit number of the horizontal one-dimensional interpolation filter, and the bit depth of the sample. For example, if the scaling bit number p of the vertical one-dimensional interpolation filter is 6, the scaling bit number q of the horizontal one-dimensional interpolation filter is 6, and the bit depth of the sample is b, the inverse scaling bit number is p+q+8-b, which is also 20-b.

[0058] If the block-based motion compensation unit 115 performs filtering on pixels whose components in a predetermined direction are integers using a one-dimensional interpolation filter and then performs only bit shifting to the right by the number of inverse scaling bits, a rounding error may occur, so the block-based motion compensation unit 115 may perform filtering on pixels whose components in a predetermined direction are integers using a one-dimensional interpolation filter, add an offset, and then perform inverse scaling. In this case, the offset is 2^(number of inverse scaling bits-1).

[0059] The pixel group unit motion compensation unit 120 may perform motion compensation for a pixel group unit related to a current block and generate a motion compensation value for a pixel group unit. When a motion prediction mode of a current block is a bidirectional motion prediction mode, the pixel group unit motion compensation unit 120 may perform motion compensation for a pixel group unit on the current block and generate a motion compensation value for a pixel group unit.

[0060] The pixel group unit motion compensation unit 120 may perform pixel group unit motion compensation for the current block based on the optical flow of pixel groups of the first and second reference pictures, and generate pixel group unit motion compensation values. The optical flow will be described later in relation to FIG. 3A.

[0061] The pixel group unit motion compensation unit 120 may perform pixel group unit motion compensation on pixel groups included in a reference block of a current block to generate pixel group unit motion compensation values. The pixel group may include at least one pixel. For example, the pixel group may be one pixel. Or, the pixel group may be a plurality of pixels including two or more pixels. The pixel group may be a plurality of pixels included in a block of a size of KxK (K is an integer).

[0062] The pixel group unit motion compensation unit 120 may determine a pixel group and perform pixel group unit motion compensation for a current block based on the determined pixel group.

[0063] The pixel group-based motion compensation unit 120 may determine the size of a pixel group based on the size of a current block. For example, the pixel group-based motion compensation unit 120 may determine the maximum value of 2 and a value obtained by dividing the minimum value of the height and width of the current block by 8 as the height and width of the pixel group.

[0064] The pixel group unit motion compensation unit 120 performs motion compensation in units of pixel groups each including a plurality of pixels, thereby enabling a reduction in encoding / decoding complexity compared to when motion compensation is performed in units of pixels from a high video resolution. Also, the pixel group unit motion compensation unit 120 performs motion compensation in units of pixel groups each including a plurality of pixels, thereby enabling a reduction in encoding / decoding complexity compared to when motion compensation is performed in units of pixels from a high frame rate.

[0065] The acquiring unit 105 may acquire information about the size of the pixel group included in the bitstream. The information about the size of the pixel group is also information indicating a height or width K when the size of the pixel group is KxK. The information about the size of the pixel group is also included in a high level syntax carrier.

[0066] The pixel group unit motion compensation unit 120 may determine at least one pixel group partition including pixels having similar pixel values ​​among a plurality of pixels included in a pixel group, and perform motion compensation on the pixel group partition. At this time, since pixel group partitions including pixels having similar pixel values ​​are likely to be the same object and have similar motions, the pixel group unit motion compensation unit 120 may perform finer pixel group unit motion compensation.

[0067] Meanwhile, motion compensation in units of pixel groups is performed when the motion prediction mode information indicates a bidirectional motion prediction mode, but even in this case, it is not performed all the time but is performed selectively.

[0068] The pixel group unit motion compensation unit 120 may determine a reference pixel group in a reference block corresponding to a current pixel group of a current block, and determine a gradient value of the reference pixel group. For example, the pixel group unit compensation unit 120 may determine the gradient value of the reference pixel group using a gradient value of at least one pixel value included in the reference pixel group.

[0069] The pixel group unit motion compensation unit 120 may perform pixel group unit motion compensation for the current block using the gradient values ​​of the reference pixel group, thereby generating pixel group unit motion compensation values.

[0070] The pixel group-based motion compensation unit 120 can apply a filter to a first peripheral area of ​​a first pixel group including a first pixel group among pixel groups of a first reference block indicated by a first motion vector, and a second peripheral area of ​​a second pixel group including a second pixel group among pixel groups of a second reference block indicated by a second motion vector, to generate gradient values ​​of the first pixel group and gradient values ​​of the second pixel group.

[0071] The pixel group-based motion compensation unit 120 may determine pixel values ​​and pixel gradient values ​​of pixels in a first window of a predetermined size including a first pixel group in a first reference picture, centered on the first pixel group, and may determine pixel values ​​and pixel gradient values ​​of pixels in a second window of a predetermined size including the second reference pixels, centered on a second reference pixel group in a second reference picture.

[0072] The pixel group-based motion compensation unit 120 may determine a displacement vector per unit time for the current pixel group using pixel values ​​and gradient values ​​of pixels in the first window and pixel values ​​and gradient values ​​of pixels in the second window. At this time, the displacement vector per unit time for the current pixel group may be adjusted according to a regularization parameter. The regularization parameter is a parameter introduced to prevent an error from occurring when an ill-posed displacement vector per unit time for the current pixel group is determined for performing motion compensation for the pixel group. The pixel group-based motion compensation unit 120 may perform motion compensation for the current block based on a regularization parameter for a displacement vector per unit time in the horizontal or vertical direction. The regularization parameter will be described later in the description of FIG. 8A.

[0073] The pixel group unit motion compensation unit 120 may perform pixel group unit motion compensation for the current block using a displacement vector per unit time for the current pixel group and gradient values ​​of reference pixels.

[0074] The reference position of the reference block may be, but is not limited to, an integer pixel position, and may also be a fractional pixel position.

[0075] When the reference position of the reference block is a sub-pixel position, the gradient value of the reference pixel in the reference block is also determined using pixel values ​​of surrounding pixels whose predetermined directional components are integers.

[0076] For example, the pixel group unit motion compensation unit 120 may use a gradient filter to filter pixel values ​​of surrounding pixels whose predetermined directional components are integers, and may determine a gradient value of a reference pixel from the resultant value. At this time, the filter coefficients of the gradient filter may be determined using coefficients determined in advance for a DCT-based interpolation filter. The coefficients of the gradient filter may also be filter coefficients scaled to integer coefficients to reduce real-number operations when filtering is performed.

[0077] In this case, the gradient filter is also a one-dimensional gradient filter in the horizontal or vertical direction.

[0078] The pixel group unit motion compensation unit 120 may use a one-dimensional horizontal or vertical gradient filter to determine a horizontal or vertical gradient value for a reference pixel, and may perform filtering on surrounding pixels whose corresponding directional components are integers.

[0079] For example, the pixel group unit motion compensation unit 120 may use a horizontal one-dimensional gradient filter to perform filtering on pixels located horizontally from pixels located around a reference pixel whose horizontal components are integers, thereby determining a horizontal gradient value related to the reference pixel.

[0080] If the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are decimals), the pixel group unit motion compensation unit 120 performs filtering using a vertical one-dimensional interpolation filter on the pixel at the (x, y) position and pixels located vertically from the pixel at the (x, y) position whose vertical components are integers, and can determine the pixel value of (x, y+β) using the resulting value.

[0081] The pixel group unit motion compensation unit 120 performs filtering using a horizontal gradient filter on the pixel value at the (x, y+β) position and the pixels located horizontally from the (x, y+β) position whose horizontal components are integers, and can determine the horizontal gradient value at the (x+α, y+β) position using the resulting value.

[0082] The order of using the one-dimensional gradient filter and the one-dimensional interpolation filter is not limited. In the above, the vertical interpolation filter is used to perform filtering on pixels at integer positions, a vertical interpolation filtering value is generated, and a one-dimensional horizontal gradient filter is used to perform filtering on the vertical interpolation filtering value. However, the one-dimensional horizontal gradient filter can be used to perform filtering on pixels at integer positions, a horizontal interpolation filtering value is generated, and a one-dimensional vertical interpolation filter is used to perform filtering on the horizontal interpolation filtering value.

[0083] The pixel group unit motion compensation unit 120 has been described in detail above for determining the horizontal gradient value at the (x+α, y+β) position. The pixel group unit motion compensation unit 120 determines the vertical gradient value at the (x+α, y+β) position in a similar manner to the horizontal gradient value, so a detailed description will be omitted.

[0084] In the above, the pixel group unit motion compensation unit 230 has been described in detail using a one-dimensional gradient filter and a one-dimensional interpolation filter to determine gradient values ​​at fractional pixel positions. However, the present invention is not limited thereto, and a gradient filter and an interpolation filter may also be used to determine gradient values ​​at integer pixel positions. However, in the case of integer pixels, pixel values ​​are determined without using an interpolation filter. However, in order to be consistent with fractional pixel processing, the integer pixel value may be determined by filtering the integer pixel and surrounding pixels whose predetermined directional components are integers using an interpolation filter. For example, the interpolation filter coefficients for integer pixels may be {0,0,64,0,0}, and since the interpolation filter coefficients for surrounding integer pixels are 0, filtering is performed using only the pixel value of the current integer pixel. As a result, the interpolation filter may be used to filter the current integer pixel and surrounding integer pixels to determine the pixel value of the current integer pixel.

[0085] The pixel group unit motion compensation unit 120 may perform inverse scaling after filtering pixels at integer positions using a vertical one-dimensional interpolation filter. In this case, the inverse scaling may include bit shifting to the right by the inverse scaling bit number. The inverse scaling bit number may also be determined based on the bit depth of the sample. The inverse scaling bit number may also be determined based on specific input data in the block.

[0086] For example, the bit shifting number is the bit depth of the sample minus 8.

[0087] The pixel group unit motion compensation unit 120 may perform inverse scaling after filtering the value generated by performing the inverse scaling using a horizontal gradient filter. Similarly, here, the inverse scaling may include bit shifting to the right by the inverse scaling bit number. The inverse scaling bit number is also determined based on the scaled bit number of the vertical one-dimensional interpolation filter, the scaled bit number of the horizontal one-dimensional gradient filter, and the bit depth of the sample. For example, if the scaling bit number p of the one-dimensional interpolation filter is 6, the scaling bit number q of the one-dimensional gradient filter is 4, and the bit depth of the sample is b, the inverse scaling bit number is p+q+8-b, which is also 18-b.

[0088] If the pixel group unit motion compensation unit 120 performs only bit shifting to the right by the number of inverse scaling bits on the value generated by the filtering after the filtering, a rounding error may occur, so an offset may be added to the value generated by the filtering before performing inverse scaling. In this case, the offset may be 2^(number of inverse scaling bits-1).

[0089] The inter prediction unit 110 may generate a predicted pixel value of the current block using a block-based motion compensation value related to the current block and a pixel group-based motion compensation value. For example, the inter prediction unit 110 may generate a predicted pixel value of the current block by combining a block-based motion compensation value related to the current block and a pixel group-based motion compensation value. Here, the block-based motion compensation value means a value generated by performing block-based motion compensation, and the pixel group-based motion compensation value is a value generated by performing pixel group-based motion compensation, and the block-based motion compensation value may be an average value or a weighted sum of reference pixels, and the pixel group-based motion compensation value may be a value determined based on a displacement vector per unit time for the current pixel and a gradient value of the reference pixels.

[0090] The pixel group unit motion compensation unit 120 may obtain a shift value for descaling after the interpolation or gradient operation based on at least one of a bit depth of a sample, an input range of a filter used for the interpolation or gradient operation, and a coefficient of the filter. The pixel group unit motion compensation unit 120 may perform descaling after the interpolation or gradient operation on pixels included in a first reference block and a second reference block using the shift value for descaling.

[0091] When performing block-based motion compensation, the inter prediction unit 110 may use a motion vector and store the motion vector. In this case, the motion vector unit may be a block of 4x4 size. Meanwhile, when storing a motion vector after block-based motion compensation, the motion vector storage unit may be a block of various sizes other than 4x4 size (e.g., a block of RxR size; R is an integer). In this case, the motion vector storage unit may be a block larger than 4x4 size, e.g., a block of 16x16 size.

[0092] Meanwhile, when performing motion compensation in units of pixel groups, the size of a target block for performing motion compensation in units of pixel groups is expanded based on the size of the current block as well as the window size and the length of the interpolation filter. The reason that the size of the target block is expanded from the size of the current block based on the window size is that in the case of a pixel located at an edge of the current block, a window is used to perform motion compensation in units of pixel groups related to the current block based on the pixel located at the current edge and the surrounding pixels.

[0093] Therefore, in order to reduce the number of memory accesses and the number of multiplication operations, the pixel group-based motion compensation unit 120 adjusts the positions of pixels within the window that are outside the current block to the positions of adjacent pixels within the current block in the process of performing motion compensation on a pixel group basis using a window, and determines pixel values ​​and gradient values ​​at the adjusted pixel positions, thereby reducing the number of memory accesses and the number of multiplication operations.

[0094] The pixel group unit motion compensation unit 120 does not use pixel values ​​of integer position pixels to determine gradient values ​​of reference pixels, which are values ​​required for pixel group unit motion compensation. That is, the pixel group unit compensation unit 120 may apply a horizontal or vertical gradient filter to pixel values ​​of fractional position pixels to calculate horizontal or vertical gradient values ​​of a first corresponding reference pixel in a first reference block or a second corresponding reference pixel in a second reference block. In this case, the gradient filter length may be 5. In this case, the filter coefficients may have coefficients that are symmetrical about a center coefficient of the filter. The fractional position pixel is also a pixel in which at least one of the horizontal and vertical components indicating the position of the pixel has a fractional value.

[0095] The pixel value of the sub-position pixel in the first reference block or the second reference block is also calculated by applying a horizontal or vertical interpolation filter to the pixel value of the integer position pixel.

[0096] The horizontal or vertical displacement vector per unit time for the current pixel group is also a displacement vector per unit time determined using a value determined based on a first corresponding reference pixel included in a first corresponding reference pixel group in a first reference picture corresponding to the current pixel group, a second corresponding reference pixel included in a second corresponding reference pixel group in a second reference picture, and pixel values ​​and gradient values ​​of their surrounding pixels, a first POC (picture order count) difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture.

[0097] When the first corresponding reference pixel or the second corresponding reference pixel is a boundary peripheral pixel adjacent to the inside of the boundary of the first reference block or the second reference block, the pixel group unit compensation unit 120 can derive a pixel value of a pixel located outside the boundary of the first reference block or the second reference block using a pixel value of the boundary peripheral pixel.

[0098] The pixel group unit compensation unit 120 may determine a displacement vector per unit time in the horizontal or vertical direction of the current block based on the pixel values ​​of the boundary surrounding pixels and the pixel values ​​of the pixels located outside the boundary of the current block derived using the pixel values ​​of the boundary surrounding pixels. That is, there are pixels located outside the boundary among the pixels included in the window centered on the boundary surrounding pixels, and in this case, the pixel values ​​and gradient values ​​of the pixels located outside the boundary are also pixel values ​​and gradient values ​​of the pixels derived from the boundary surrounding pixels that are not values ​​stored in the memory.

[0099] The pixel group unit compensation unit 120 may calculate a value related to the current pixel by using a pixel value of a first corresponding reference pixel included in a first reference block, a pixel value of a second corresponding reference pixel included in a second reference block, a gradient value of the first corresponding reference pixel, a gradient value of the second corresponding reference pixel, a first POC difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture. That is, the value related to the current pixel is also a result value of a function based on the pixel value, gradient value, of the corresponding reference pixel of each reference picture, and the POC difference between each reference picture and the current picture.

[0100] The pixel group unit compensation unit 120 may calculate values ​​related to the surrounding pixels by using the pixel values ​​of the first corresponding surrounding pixels of the first corresponding reference pixels, the gradient values ​​of the first corresponding surrounding pixels, the pixel values ​​of the second corresponding surrounding pixels of the second corresponding reference pixels, the gradient values ​​of the second corresponding surrounding pixels, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture. That is, the values ​​related to the surrounding pixels are also the result values ​​of a function based on the pixel values ​​and gradient values ​​of the corresponding reference pixels of each reference picture and the POC difference between each reference picture and the current picture. That is, the values ​​related to the surrounding pixels are also the result values ​​of a function based on the pixel values ​​and gradient values ​​of the corresponding surrounding pixels of each reference picture and the POC difference between each reference picture and the current picture.

[0101] The pixel group unit compensation unit 120 may calculate a weighted average value for the current pixel required to calculate a horizontal displacement vector per unit time using the value for the current pixel, the value for the neighboring pixels, and a weight value. At this time, the weight value may be determined based on the distance between the current pixel and the neighboring pixels, the distance between the pixel and the block boundary, the number of pixels located outside the boundary, or whether the pixel is located inside or outside the boundary.

[0102] The weighted average value for the current pixel is also a value calculated by applying exponential smoothing to the values ​​of pixels included in the first and second reference blocks in the up, down, left and right directions. By applying exponential smoothing to the values ​​of pixels in the up, down, left and right directions, the value calculated for the current pixel has the largest weighting for the value of the current pixel, and the weighting for the values ​​of the neighboring pixels is a value that is exponentially decreased according to the distance from the current pixel.

[0103] The pixel group unit compensation unit 120 may determine a displacement vector per unit time in the horizontal or vertical direction of the current pixel by using the weighted average value for the current pixel.

[0104] The reconstruction unit 125 may obtain a residual block of the current block from the bitstream, and reconstruct the current block using the residual block and a predicted block of the current block. For example, the reconstruction unit 125 may combine pixel values ​​of the residual block of the current block and pixel values ​​of the predicted block of the current block from the bitstream to generate pixel values ​​of a reconstructed block.

[0105] The video decoding device 100 may include a video decoding unit (not shown), which may include an obtaining unit 105, an inter predicting unit 110, and a restoring unit 125. The video decoding unit will be described with reference to FIG. 1E.

[0106] FIG. 1B illustrates a flowchart of a video decoding method according to various embodiments.

[0107] In operation S105, the video decoding apparatus 100 may acquire motion prediction mode information related to a current block in a current picture from a bitstream. The video decoding apparatus 100 may receive a bitstream including motion prediction mode information related to a current block in a current picture, and acquire the motion prediction mode information related to the current block from the received bitstream. The video decoding apparatus 100 may acquire information on a prediction mode of the current block from the bitstream, and may determine a prediction mode of the current block based on the information on the prediction mode of the current block. In this case, if the prediction mode of the current block is an inter prediction mode, the video decoding apparatus 100 may acquire motion prediction mode information related to the current block.

[0108] For example, the video decoding apparatus 100 may determine the prediction mode of the current block to be the inter prediction mode based on information about the prediction mode of the current block. If the prediction mode of the current block is the inter prediction mode, the video decoding apparatus 100 may obtain motion prediction mode information related to the current block from the bitstream.

[0109] At step S110, if the motion prediction mode information indicates a bidirectional motion prediction mode, the video decoding device 100 may obtain from the bitstream a first motion vector indicating a first reference block of the current block in a first reference picture, and a second motion vector indicating a second reference block of the current block in a second reference picture.

[0110] That is, the video decoding device 100 may receive a bitstream including information about the first motion vector and the second motion vector, and may obtain the first motion vector and the second motion vector from the received bitstream. The video decoding device 100 may obtain a reference picture index from the bitstream, and may determine the first reference picture and the second reference picture among previously decoded pictures based on the reference picture index.

[0111] In operation S115, the video decoding apparatus 100 may determine a horizontal or vertical displacement vector of a pixel of the current block including a pixel adjacent to an inner boundary of the current block by using values ​​related to reference pixels included in the first and second reference blocks without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks. In this case, the values ​​stored for pixels located outside the boundaries of the first and second reference blocks and the values ​​related to reference pixels included in the first and second reference blocks may be pixel values ​​of related pixels, or horizontal gradient values ​​or vertical gradient values ​​of related pixels. Alternatively, the values ​​stored for pixels located outside the boundaries of the first and second reference blocks and the values ​​related to reference pixels included in the first and second reference blocks may be values ​​determined by using pixel values ​​or gradient values ​​of related pixels.

[0112] In step S120, the video decoding device 100 may perform block-based motion compensation and pixel group-based motion compensation of the current block using a horizontal or vertical gradient value of a first corresponding reference pixel in a first reference block corresponding to a current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in a second reference block, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector of the current pixel, to obtain a prediction block of the current block.

[0113] That is, the video decoding apparatus 100 may perform block-based motion compensation based on a first motion vector and a second motion vector, and pixel group-based motion compensation on a current block to generate a predicted block of the current block. The video decoding apparatus 100 may perform block-based motion compensation on the current block using a pixel value of a first reference block indicated by a first motion vector and a pixel value of a second reference block indicated by a second motion vector. In addition, the video decoding apparatus 100 may perform pixel group-based motion compensation on the current pixel group using a horizontal or vertical gradient value of at least one first corresponding reference pixel in a first reference block corresponding to at least one pixel included in a current pixel group, a horizontal or vertical gradient value of at least one second corresponding reference pixel in a second reference block, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector of the current pixel.

[0114] The video decoding device 100 can obtain a prediction block for the current block by using a block-based motion compensation value generated by performing block-based motion compensation on the current block, and a pixel group-based motion compensation value generated by performing pixel group-based motion compensation on the current pixel group.

[0115] In operation S125, the video decoding apparatus 100 may obtain information about the residual block of the current block from the bitstream.

[0116] In operation S130, the video decoding apparatus 100 may reconstruct the current block based on the predictive block and the residual block. That is, the video decoding apparatus 100 may combine pixel values ​​of the residual block indicated by the residual block related to the current block with predicted pixel values ​​of the predictive block to generate pixel values ​​of a reconstructed block of the current block.

[0117] FIG. 1C illustrates a block diagram of a video encoding device according to various embodiments.

[0118] A video encoding device 150 according to various embodiments includes an inter prediction unit 155 and a bitstream generation unit 170 .

[0119] The inter prediction unit 155 performs inter prediction with reference to various blocks based on a rate and a distortion cost for the current block. That is, the inter prediction unit 155 may generate a predicted pixel value of the current block by using at least one of pictures encoded before the current picture including the current block.

[0120] The inter prediction unit 155 may include a block-based motion compensation unit 160 and a pixel group-based motion compensation unit 165 .

[0121] The block-based motion compensation unit 160 may perform block-based motion compensation on the current block to generate a block-based motion compensation value.

[0122] The block-based motion compensation unit 160 may determine at least one reference picture from among previously decoded pictures, and determine a reference block of the current block located within the at least one reference picture.

[0123] The block-based motion compensation unit 160 may perform block-based motion compensation for the current block using pixel values ​​of a reference block to generate block-based motion compensation values. The block-based motion compensation unit 160 may perform block-based motion compensation for the current block using reference pixel values ​​of a reference block corresponding to a current pixel of the current block to generate block-based motion compensation values.

[0124] The block-based motion compensation unit 160 may perform block-based motion compensation for the current block using a plurality of reference blocks included in a plurality of reference pictures, and generate a block-based motion compensation value. For example, when the motion prediction mode of the current block is a bidirectional prediction mode, the block-based motion compensation unit 160 may determine two reference pictures among previously encoded pictures and determine two reference blocks included in the two pictures. Here, the bidirectional prediction is not limited to performing inter-prediction using a picture that is earlier than the current picture in display order and a picture that is later than the current picture in display order, but may also mean performing inter-prediction using two pictures that were encoded before the current picture regardless of the display order.

[0125] The block-based motion compensation unit 160 may perform block-based motion compensation for the current block using two reference pixel values ​​in two reference blocks to generate a block-based motion compensation value. The block-based motion compensation unit 160 may perform block-based motion compensation for the current block using an average pixel value or a weighted sum of two reference pixels to generate a block-based motion compensation value.

[0126] The block-based motion compensation unit 160 may output a reference picture index indicating a reference picture for motion compensation of the current block among previously encoded pictures.

[0127] The block-based motion compensation unit 160 may determine a motion vector starting from the current block and ending at the reference block of the current block, and output the motion vector. The motion vector refers to a vector indicating the displacement between the reference coordinates of the current block in the current picture and the reference coordinates of the reference block in the reference picture. For example, if the coordinates of the upper left corner of the current block are (1,1) and the coordinates of the upper left corner of the reference block in the reference picture are (3,3), the motion vector is also (2,2).

[0128] The reference position of the reference block may be, but is not limited to, an integer pixel position, or a fractional pixel position, where the fractional pixel position may be determined in 1 / 4-pel units or 1 / 16-pel units, where the fractional pixel position may be determined in various fractional pel units.

[0129] For example, if the reference position of the reference block is (1.5,1.5) and the coordinate of the upper left corner of the current block is (1,1), the motion vector is also (0.5,0.5). If the motion vector is determined to be 1 / 4 pel unit or 1 / 16 pel unit to indicate the reference position of the reference block which is a fractional pixel position, the motion vector can be scaled to determine an integer motion vector, and the upscaled motion vector can be used to determine the reference position of the reference block. If the reference position of the reference block is a fractional pixel position, the position of the reference pixel of the reference block is also a fractional pixel position. Therefore, in the reference block, the pixel value of the fractional pixel position is also determined using the pixel value of a surrounding pixel whose predetermined direction component is an integer.

[0130] For example, the block-based motion compensation unit 160 may use an interpolation filter to perform filtering on pixel values ​​of neighboring pixels whose predetermined directional components are integers, determine a reference pixel value at a sub-pixel position using the result, and determine a block-based motion compensation value for a current block using the pixel value of the reference pixel. In this case, the interpolation filter may be a DCT-based M-tap interpolation filter. The coefficients of the DCT-based M-tap interpolation filter may be derived from the DCT and IDCT. In this case, the coefficients of the interpolation filter may be filter coefficients scaled to integer coefficients to reduce real-number operations when filtering is performed.

[0131] In this case, the interpolation filter is also a one-dimensional interpolation filter in the horizontal or vertical direction.

[0132] The block-based motion compensation unit 160 may first filter the surrounding integer pixels using a one-dimensional interpolation filter in the vertical direction, and then filter the filtered value using a one-dimensional interpolation filter in the horizontal direction to determine a reference pixel value at a decimal pixel position. When using scaled filter coefficients, the block-based motion compensation unit 160 may filter the pixels at integer positions using a one-dimensional interpolation filter in the vertical direction, and then perform inverse scaling on the filtered value. In this case, the inverse scaling may include bit shifting to the right by the inverse scaling bit number. The inverse scaling bit number is also determined based on the bit depth of the sample. For example, the bit shifting number may be a value obtained by subtracting 8 from the bit depth of the sample.

[0133] In addition, the block-based motion compensation unit 160 may perform filtering on pixels whose horizontal components are integers using a horizontal one-dimensional interpolation filter, and then bit-shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits is also determined based on the number of scaled bits for the vertical one-dimensional interpolation filter coefficients, the number of scaled bits for the horizontal one-dimensional interpolation filter coefficients, and the bit depth of the samples.

[0134] If the block-based motion compensation unit 160 only performs bit shifting to the right by the number of inverse scaling bits, a rounding error may occur, so the block-based motion compensation unit 160 may use a one-dimensional interpolation filter in a predetermined direction to filter pixels whose components in the predetermined direction are integers, add an offset to the filtered value, and perform inverse scaling on the value to which the offset has been added. In this case, the offset is 2^(number of inverse scaling bits-1).

[0135] As described above, after filtering using a vertical one-dimensional interpolation filter, the number of inverse scaling bits is determined based on the bit depth of the sample, but the present invention is not limited thereto and may be determined by considering not only the bit depth of the sample but also the number of bits scaled for the interpolation filter coefficient. That is, the number of inverse scaling bits may be determined based on the bit depth of the sample and the number of bits scaled for the interpolation filter coefficient within a range where no overflow occurs, taking into consideration the size of the register used when filtering and the size of the buffer that stores the value generated during filtering.

[0136] The pixel group unit motion compensation unit 165 may perform motion compensation for a pixel group unit related to a current block and generate a motion compensation value for a pixel group unit. For example, when the motion prediction mode is a bidirectional motion prediction mode, the pixel group unit motion compensation unit 165 may perform motion compensation for a pixel group unit on the current block and generate a motion compensation value for a pixel group unit.

[0137] The pixel group unit motion compensation unit 165 may perform pixel group unit motion compensation for the current block using gradient values ​​of pixels included in a reference block of the current block, and generate pixel group unit motion compensation values.

[0138] The pixel group unit motion compensation unit 165 may apply a filter to a first surrounding area of ​​a first pixel among pixels of a first reference block in a first reference picture, and a second surrounding area of ​​a second pixel among pixels of a second reference block in a second reference picture, to generate a gradient value of the first pixel and a gradient value of the second pixel.

[0139] The pixel group unit motion compensation unit 165 may determine a pixel value and a pixel gradient value of a pixel in a first window of a predetermined size including a first reference pixel in a first reference picture, and may determine a pixel value and a pixel gradient value of a pixel in a second window of a predetermined size including a second reference pixel in a second reference picture, and may determine a pixel value and a pixel gradient value of a pixel in a second window of a predetermined size including a second reference pixel in a second reference picture. The pixel group unit motion compensation unit 165 may determine a displacement vector per unit time for a current pixel using the pixel value and pixel gradient value of a pixel in the first window and the pixel value and pixel gradient value of a pixel in the second window.

[0140] The pixel group unit motion compensation unit 165 may perform pixel group unit motion compensation for the current block using the displacement vector per unit time and the gradient value of the reference pixel, and generate a pixel group unit motion compensation value.

[0141] The reference pixel location may be, but is not limited to, an integer pixel location, and may also be a fractional pixel location.

[0142] When the reference position of the reference block is a sub-pixel position, the gradient value of the reference pixel in the reference block is also determined using pixel values ​​of surrounding pixels whose predetermined directional components are integers.

[0143] For example, the pixel group unit motion compensation unit 165 may filter pixel values ​​of surrounding pixels having integer directional components using a gradient filter, and may determine a gradient value of a reference pixel using the resultant value. In this case, the filter coefficients of the gradient filter may be determined using coefficients determined in advance for a DCT-based interpolation filter.

[0144] The coefficients of the gradient filter are also filter coefficients scaled to integer coefficients in order to reduce real-number operations during filtering, and the gradient filter used here is also a one-dimensional gradient filter in the horizontal or vertical direction.

[0145] The pixel group unit motion compensation unit 165 may use a one-dimensional horizontal or vertical gradient filter to determine a horizontal or vertical gradient value for a reference pixel, and perform filtering on surrounding pixels whose corresponding directional components are integers.

[0146] For example, the pixel group unit motion compensation unit 165 can use a one-dimensional vertical interpolation filter to perform filtering on pixels whose vertical components are integers among the pixels in the vertical direction from the integer pixels adjacent to the reference pixel, and determine pixel values ​​of pixels whose vertical components are decimals.

[0147] The pixel group unit motion compensation unit 165 may also determine pixel values ​​of fractional pixel positions in other columns adjacent to the integer pixel adjacent to the reference pixel by filtering the neighboring integer pixels in the vertical direction using a vertical one-dimensional interpolation filter, where the positions of the pixels in the other columns are fractional pixel positions in the vertical direction and integer pixel positions in the horizontal direction.

[0148] That is, when the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are decimals), the pixel group unit motion compensation unit 165 can determine the pixel value at the (x, y+β) position by filtering the surrounding integer pixels in the vertical direction from the (x, y) position using a vertical interpolation filter.

[0149] The pixel group unit motion compensation unit 165 can determine a horizontal gradient value at the (x+α,y+β) position by filtering the pixel values ​​of pixels located horizontally whose horizontal components are integers using a horizontal gradient filter from the pixel values ​​at the (x,y+β) position and the pixel values ​​at the (x,y+β) position.

[0150] The order of using the one-dimensional gradient filter and the one-dimensional interpolation filter is not limited. As described above, the vertical interpolation filter can be used first to perform filtering on pixels at integer positions, to generate vertical interpolation filtering values, and the one-dimensional horizontal gradient filter can be used to perform filtering on the vertical interpolation filtering values, but the order is not limited thereto. The one-dimensional horizontal gradient filter can be used first to perform filtering on pixels at integer positions, to generate horizontal gradient filtering values, and the one-dimensional vertical interpolation filter can be used to perform filtering on the horizontal gradient filtering values.

[0151] The above has described in detail how pixel group unit motion compensation section 165 determines the horizontal gradient value at position (x+α, y+β).

[0152] The pixel group unit motion compensation unit 165 may determine the vertical gradient value at the (x+α, y+β) position in a manner similar to that in which the horizontal gradient value is determined.

[0153] The pixel group unit motion compensation unit 165 may determine a vertical gradient value for the reference pixel by filtering integer pixels surrounding the reference pixel in the vertical direction from the integer pixels surrounding the reference pixel using a vertical one-dimensional gradient filter. The pixel group unit motion compensation unit 165 may also determine a vertical gradient value for a pixel located in another column adjacent to the reference pixel by filtering integer pixels in the vertical direction using a vertical one-dimensional gradient filter. Here, the pixel position is a fractional pixel position in the vertical direction and an integer pixel position in the horizontal direction.

[0154] That is, when the position of the reference pixel is (x+α, y+β) (where x, y are integers and α, β are decimals), the pixel group unit motion compensation unit 165 can determine a vertical gradient value at the (x, y+β) position by filtering the surrounding integer pixels in the vertical direction from the (x, y) position using a vertical gradient filter.

[0155] The pixel group-based motion compensation unit 165 can determine a vertical gradient value at a position (x+α,y+β) by filtering the gradient value at the position (x,y+β) and the gradient values ​​of surrounding integer pixels located horizontally from the position (x,y+β) using a horizontal interpolation filter.

[0156] The order of using the one-dimensional gradient filter and the one-dimensional interpolation filter is not limited. As described above, the vertical gradient filter can be used to first filter pixels at integer positions, generate vertical gradient filtering values, and then use the one-dimensional horizontal interpolation filter to filter the vertical gradient filtering values. However, the order is not limited thereto. The one-dimensional horizontal interpolation filter can be used to first filter pixels at integer positions, generate horizontal interpolation filtering values, and then use the one-dimensional vertical gradient filter to filter the horizontal interpolation filtering values.

[0157] The above describes in detail how the pixel group unit motion compensation unit 165 uses gradient filters and interpolation filters to determine gradient values ​​at sub-pixel positions. However, the present invention is not limited to this, and gradient filters and interpolation filters can also be used to determine gradient values ​​at integer pixel positions.

[0158] In the case of integer pixels, the pixel value can be determined without using an interpolation filter, but for processing consistent with processing for fractional pixels, filtering can be performed on the integer pixels and surrounding integer pixels using an interpolation filter. For example, the interpolation filter coefficients for integer pixels are {0,0,64,0,0}, and since the interpolation filter coefficients multiplied by the surrounding integer pixels are 0, filtering is performed using only the pixel value of the current integer pixel. As a result, a value generated by filtering the current integer pixel and surrounding integer pixels using an interpolation filter can be determined to be the same as the pixel value of the current integer pixel.

[0159] Meanwhile, when using the scaled filter coefficients, the pixel group unit motion compensation unit 165 may perform filtering on pixels at integer positions using a horizontal one-dimensional gradient filter, and perform inverse scaling on the filtered values. In this case, the inverse scaling may include bit shifting to the right by the inverse scaling bit number. The inverse scaling bit number is also determined based on the bit depth of the sample. For example, the inverse scaling bit number may be a value obtained by subtracting about 8 from the bit depth of the sample.

[0160] The pixel group unit motion compensation unit 165 may perform inverse scaling after filtering pixels whose vertical components are integers using a vertical interpolation filter. In this case, the inverse scaling may include bit shifting to the right by the inverse scaling bit number. The inverse scaling bit number is also determined based on the scaled bit number of the vertical one-dimensional interpolation filter, the scaled bit number of the horizontal one-dimensional gradient filter, and the bit depth of the sample.

[0161] If the pixel group unit motion compensation unit 165 only performs bit shifting to the right by the number of inverse scaling bits, a rounding error may occur. Therefore, after filtering is performed using a one-dimensional interpolation filter, an offset is added to the filtered value, and inverse scaling is performed on the value to which the offset is added. In this case, the offset is also 2^(number of bit shifting-1).

[0162] When performing block-based motion compensation, the inter prediction unit 110 may use motion vectors and store the motion vectors. In this case, the motion vector unit may be a block of 4x4 size. Meanwhile, when storing motion vectors after block-based motion compensation, the motion vector storage unit may be a block of various sizes other than 4x4 size (e.g., a block of RxR size; R is an integer). In this case, the motion vector storage unit may be a block larger than 4x4 size, e.g., a block of 16x16 size.

[0163] Meanwhile, when performing motion compensation in pixel group units, the size of the target block for performing motion compensation in pixel group units may be expanded based on the size of the current block as well as the window size and the interpolation filter length. The reason that the size of the target block is expanded from the size of the current block based on the window size is that in the case of a pixel located at the edge of the current block, a window is used to perform motion compensation in pixel group units related to the current block based on the pixel located at the current edge and the surrounding pixels.

[0164] Therefore, in order to reduce the number of memory accesses and the number of multiplication operations, the pixel group-based motion compensation unit 120 adjusts the positions of pixels within the window that are outside the current block to the positions of adjacent pixels within the current block in the process of performing motion compensation on a pixel group basis using a window, and determines pixel values ​​and gradient values ​​at the adjusted pixel positions, thereby reducing the number of memory accesses and the number of multiplication operations.

[0165] The pixel group unit motion compensation unit 120 does not use pixel values ​​of integer position pixels to determine gradient values ​​of reference pixels, which are values ​​required for pixel group unit motion compensation. That is, the pixel group unit compensation unit 120 may apply a horizontal or vertical gradient filter to pixel values ​​of fractional position pixels to calculate horizontal or vertical gradient values ​​of a first corresponding reference pixel in a first reference block or a second corresponding reference pixel in a second reference block. In this case, the gradient filter length may be 5. In this case, the filter coefficients may have coefficients that are symmetrical about a center coefficient of the filter. The fractional position pixel is also a pixel in which at least one of the horizontal and vertical components indicating the position of the pixel has a fractional value.

[0166] The pixel value of the sub-position pixel in the first reference block or the second reference block is also calculated by applying a horizontal or vertical interpolation filter to the pixel value of the integer position pixel.

[0167] The horizontal or vertical displacement vector per unit time for the current pixel group is also a displacement vector per unit time determined using a value determined based on a first corresponding reference pixel included in a first corresponding reference pixel group in a first reference picture corresponding to the current pixel group, a second corresponding reference pixel included in a second corresponding reference pixel group in a second reference picture, and pixel values, gradient values, and surrounding pixels thereof, a first POC difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture.

[0168] When the first corresponding reference pixel or the second corresponding reference pixel is a boundary peripheral pixel adjacent to the inside of the boundary of the first reference block or the second reference block, the pixel group unit compensation unit 120 can derive a pixel value of a pixel located outside the boundary of the first reference block or the second reference block using a pixel value of the boundary peripheral pixel.

[0169] The pixel group unit compensation unit 120 may determine a displacement vector per unit time in the horizontal or vertical direction of the current block based on the pixel values ​​of the boundary surrounding pixels and the pixel values ​​of the pixels located outside the boundary of the current block derived using the pixel values ​​of the boundary surrounding pixels. That is, there are pixels located outside the boundary among the pixels included in the window centered on the boundary surrounding pixels, and in this case, the pixel values ​​and gradient values ​​of the pixels located outside the boundary are also pixel values ​​and gradient values ​​of the pixels derived from the boundary surrounding pixels that are not values ​​stored in the memory.

[0170] The pixel group unit compensation unit 120 may calculate a value related to the current pixel by using a pixel value of a first corresponding reference pixel included in a first reference block, a pixel value of a second corresponding reference pixel included in a second reference block, a gradient value of the first corresponding reference pixel, a gradient value of the second corresponding reference pixel, a first POC difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture. That is, the value related to the current pixel is also a result value of a function based on the pixel value, gradient value, of the corresponding reference pixel of each reference picture, and the POC difference between each reference picture and the current picture.

[0171] The pixel group unit compensation unit 120 may calculate values ​​related to corresponding surrounding pixels calculated using pixel values ​​of the first corresponding surrounding pixels of the first corresponding reference pixels, gradient values ​​of the first corresponding surrounding pixels, pixel values ​​of the second corresponding surrounding pixels of the second corresponding reference pixels, gradient values ​​of the second corresponding surrounding pixels, a first POC difference between the first reference picture and the current picture, and a second POC difference between the second reference picture and the current picture. That is, the values ​​related to surrounding pixels are also a result value of a function based on pixel values ​​and gradient values ​​of corresponding reference pixels of each reference picture and POC differences between each reference picture and the current picture. That is, the values ​​related to corresponding surrounding pixels are also a result value of a function based on pixel values ​​and gradient values ​​of corresponding surrounding pixels of each reference picture and POC differences between each reference picture and the current picture.

[0172] The pixel group unit compensation unit 120 may calculate a weighted average value for the current pixel required to calculate a horizontal displacement vector per unit time using a value related to the current pixel, a value related to the corresponding surrounding pixel, and a weight value. In this case, the weight value may be determined based on a distance between the current pixel and the surrounding pixel, a distance between the pixel and the block boundary, the number of pixels located outside the boundary, or whether the pixel is located inside or outside the boundary.

[0173] The weighted average value for the current pixel is also a value calculated by applying exponential smoothing to values ​​of pixels included in the first and second reference blocks in the up, down, left and right directions. The value calculated for the current pixel by applying exponential smoothing to values ​​of pixels in the up, down, left and right directions has the largest weighting for the value of the current pixel, and the weighting for values ​​of the neighboring pixels is a value that is exponentially decreased according to the distance from the current pixel.

[0174] The pixel group unit compensation unit 120 may determine a displacement vector per unit time in the horizontal or vertical direction of the current pixel by using the weighted average value for the current pixel.

[0175] The inter prediction unit 155 may generate a predicted pixel value of the current block by using a block-based motion compensation value related to the current block and a pixel group-based motion compensation value related to the current block. For example, the inter prediction unit 155 may generate a predicted pixel value of the current block by combining a block-based motion compensation value related to the current block and a pixel group-based motion compensation value. In particular, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the inter prediction unit 155 may generate a predicted pixel value of the current block by using a block-based motion compensation value related to the current block and a pixel group-based motion compensation value related to the current block.

[0176] When the motion prediction mode of the current block is a unidirectional motion prediction mode, the inter prediction unit 155 may generate a predicted pixel value of the current block by using a block-based motion compensation value related to the current block. Here, unidirectional means using one reference picture among previously encoded pictures. The one reference picture is not limited to being a picture prior to the current picture in display order, but may also be a picture following the current picture.

[0177] The inter prediction unit 155 may determine a motion prediction mode of the current block and output information indicating the motion prediction mode of the current block. For example, the inter prediction unit 155 may determine a bidirectional motion prediction mode as the motion prediction mode of the current block and output information indicating the bidirectional motion prediction mode. Here, the bidirectional motion prediction mode refers to a mode in which motion prediction is performed using reference blocks in two decoded reference pictures.

[0178] The bitstream generator 170 may generate a bitstream including a motion vector indicating a reference block. The bitstream generator 170 may encode a motion vector indicating a reference block and generate a bitstream including the encoded motion vector. The bitstream generator 170 may encode a vector difference value of a motion vector indicating a reference block and generate a bitstream including the encoded motion vector difference value. Here, the motion vector difference value means a difference value between a motion vector and a motion vector predictor. In this case, the motion vector difference value means a motion vector difference value related to a reference picture related to each prediction direction including an L0 direction and an L1 direction. Here, the motion vector difference value related to the L0 direction means a motion vector difference value indicating a reference block in a reference picture included in an L0 reference picture list, and the motion vector difference value related to the L1 direction means a motion vector difference value indicating a reference block in a reference picture included in an L1 reference picture list.

[0179] Also, the bitstream generator 170 may generate a bitstream further including information indicating a motion prediction mode of the current block. The bitstream generator 170 may generate a bitstream including the coded reference picture index by coding a reference picture index indicating a reference picture of the current block among previously coded pictures. In this case, the reference picture index means a reference picture index related to each prediction direction including the L0 direction and the L1 direction. Here, the reference picture index related to the L0 direction means an index indicating a reference picture among pictures included in the L0 reference picture list, and the reference picture index related to the L1 direction means an index indicating a reference picture among pictures included in the L1 reference picture list.

[0180] The video encoding device 150 may include a video encoding unit (not shown), which may include an inter prediction unit 155 and a bitstream generation unit 170. The video encoding unit will be described with reference to FIG. 1F.

[0181] FIG. 1D illustrates a flowchart of a video encoding method according to various embodiments.

[0182] Referring to FIG. 1D, in step S150, the video encoding device 150 performs motion compensation for the current block and motion compensation on a pixel group basis, and can obtain a predicted block for the current block, a first motion vector, a second motion vector, and parameters related to motion compensation on a pixel group basis.

[0183] In operation S155, the video encoding device 150 may generate a bitstream including information on the first motion vector and the second motion vector, and motion prediction mode information indicating whether a motion prediction mode for a current block is a bidirectional motion prediction mode. Here, the first motion vector is also a motion vector indicating a first reference block of a first reference picture corresponding to the current block in the current picture from the current block, and the second motion vector is also a motion vector indicating a second reference block of a second reference picture corresponding to the current block in the current picture from the current block.

[0184] The video encoding apparatus 150 may encode a residual block of the current block indicating a difference between pixels of a prediction block of the current block and an original block of the current block, and generate a bitstream further including an encoded residual signal. The video encoding apparatus 150 may encode information on a prediction mode of the current block and a reference picture index, and generate a bitstream further including information on a prediction mode of the encoded current block and a reference picture index. For example, the video encoding apparatus 150 may encode information indicating whether a prediction mode of the current block is an inter prediction mode and a reference picture index indicating at least one picture among previously decoded pictures, and generate a bitstream further including information on a prediction mode of the encoded current block and a reference picture index.

[0185] The video encoding device 150 may perform block-based motion compensation and pixel group-based motion compensation for the current block based on a horizontal or vertical gradient value of a first corresponding reference pixel in the first reference block corresponding to a current pixel included in a current pixel group in the current block, a horizontal or vertical gradient value of a second corresponding reference pixel in the second reference block corresponding to the current pixel, a pixel value of the first corresponding reference pixel, a pixel value of the second corresponding reference pixel, and a horizontal or vertical displacement vector per unit time of the current pixel. The video encoding device 150 may obtain a prediction block of the current block by performing block-based motion compensation and pixel group-based motion compensation for the current block.

[0186] In this case, the horizontal or vertical displacement vector per unit time of the pixels of the current block including the adjacent pixels inside the boundary of the current block may be determined using values ​​related to the reference pixels included in the first and second reference blocks without using values ​​stored for the pixels located outside the boundaries of the first and second reference blocks, and the values ​​related to the reference pixels included in the first and second reference blocks may be pixel values ​​or gradient values ​​of the reference pixels.

[0187] FIG. 1E illustrates a block diagram of a video decoder 600 according to various embodiments.

[0188] The video decoder 600 according to various embodiments performs the same operations as those performed by a video decoder (not shown) of the video decoding device 100 to encode video data.

[0189] 1E, the entropy decoding unit 615 parses the coded video data to be decoded and coding information required for decoding from the bitstream 605. The coded video data is quantized transform coefficients, and the inverse quantization unit 620 and the inverse transform unit 625 restore residue data from the quantized transform coefficients.

[0190] The intra prediction unit 640 performs intra prediction for each block. The inter prediction unit 635 performs inter prediction for each block using a reference image acquired from the reconstructed picture buffer 630. The inter prediction unit 635 of FIG 1E corresponds to the inter prediction unit 110 of FIG 1A.

[0191] By adding the prediction data and the residue data for each block generated by the intra prediction unit 640 or the inter prediction unit 635, spatial domain data for the block of the current image 605 is restored, and the deblocking unit 645 and the SAO performing unit 650 perform loop filtering on the restored spatial domain data to output a filtered restored image 660. In addition, the restored image stored in the restored picture buffer 630 may be output as a reference image.

[0192] In a decoder (not shown) of the video decoding device 100, in order to decode image data, the step-by-step operations of the image decoder 600 according to various embodiments may be performed block by block.

[0193] FIG. 1F illustrates a block diagram of a video encoder according to various embodiments.

[0194] The image encoder 700 according to various embodiments performs the same operations as those performed in an image encoder (not shown) of the video encoder 150 to encode image data.

[0195] That is, the intra prediction unit 720 performs intra prediction for each block in the current image 705, and the inter prediction unit 715 performs inter prediction for each block using the current image 705 and a reference image acquired from the reconstructed picture buffer 710. Here, the inter prediction unit 715 of FIG 1E corresponds to the inter prediction unit 160 of FIG 1C.

[0196] Residue data is generated by subtracting prediction data for each block output from the intra prediction unit 720 or the inter prediction unit 715 from data for a block to be encoded of the current image 705, and the transform unit 725 and the quantization unit 730 may perform transform and quantization on the residue data to output quantized transform coefficients for each block. The inverse quantization unit 745 and the inverse transform unit 750 may perform inverse quantization and inverse transform on the quantized transform coefficients to restore spatial domain residue data. The restored spatial domain residue data is added to the prediction data for each block output from the intra prediction unit 720 or the inter prediction unit 715 to restore spatial domain data for the block of the current image 705. The deblocking unit 755 and the SAO performing unit 760 perform in-loop filtering on the restored spatial domain data to generate a filtered restored image. The generated restored image is stored in the restored picture buffer 710. The reconstructed image stored in the reconstructed picture buffer 710 is also used as a reference image for inter-prediction of other images. The entropy coding unit 735 performs entropy coding on the quantized transform coefficients, and the entropy coded coefficients are also output as a bitstream 740.

[0197] Since the image encoder 700 according to various embodiments is applied to the video encoding device 150, the operations of the image encoder 700 according to various embodiments may be performed on a block-by-block basis.

[0198] 2 is a reference diagram for explaining a process of block-based bidirectional motion prediction and compensation thereof according to an embodiment. Referring to FIG. 2, the video encoding device 150 performs bidirectional motion prediction to search for an area most similar to a current block 201 to be encoded in a current picture 200 in a first reference picture 210 and a second reference picture 220. Here, it is assumed that the first reference picture 210 is a picture before the current picture 200, and the second reference picture 220 is a picture after the current picture 200. The video encoding device 150 determines a first corresponding area 212 most similar to the current block 201 in the first reference picture 210 and a second corresponding area 222 most similar to the current block 201 in the second reference picture 220 as a result of the bidirectional motion prediction. Here, the first corresponding area and the second corresponding area are also reference areas of the current block.

[0199] Then, the video encoding device 150 determines a first motion vector MV1 based on the positional difference between a block 211 at the same position as the current block 201 in the first reference picture 210 and the first corresponding area 212, and determines a second motion vector MV2 based on the positional difference between a block 221 at the same position as the current block 201 in the second reference picture 220 and the second corresponding area 222.

[0200] The video encoding device 150 performs block-based bidirectional motion compensation on the current block 201 using the first motion vector MV1 and the second motion vector MV2.

[0201] For example, if the pixel value located at (i,j) (i, j are integers) in the first reference picture 210 is P0(i,j), the pixel value located at (i,j) in the second reference picture 220 is P1(i,j), MV1 = (MVx1,MVy1) and MV2 = (MVx2,MVy2), the block-wise bidirectional motion compensation value P_BiPredBlock(i,j) of the pixel located at (i,j) of the current block 201 can be calculated as follows: P_BiPredBlock(i,j) = {P0(i + MVx1,j + MVy1) + P1(i + MVx2,j + MVy2)} / 2. In this manner, the video encoding device 150 can perform block-based motion compensation on the current block 201 by using the average value or weighted sum of pixels in the first corresponding area 212 and the second corresponding area 222 indicated by the first motion vector MV1 and the second motion vector MV2, and generate a block-based motion compensation value.

[0202] 3A to 3C are reference diagrams for explaining a process of performing motion compensation in units of pixel groups according to an embodiment.

[0203] In FIG. 3A, a first corresponding region 310 and a second corresponding region 320 correspond to the first corresponding region 212 and the second corresponding region 222 in FIG. 2, respectively, and are assumed to have been shifted using bidirectional motion vectors (MV1, MV2) so as to overlap with the current block 300.

[0204] In addition, the pixel at the bidirectionally predicted position (i,j) (i, j are integers) of the current block 300 is defined as P(i,j), the first reference pixel value of the first reference picture corresponding to the bidirectionally predicted pixel P(i,j) of the current block 300 is defined as P0(i,j), and the second reference pixel value of the second reference picture corresponding to the bidirectionally predicted pixel P(i,j) of the current block 300 is defined as P1(i,j).

[0205] In other words, the first reference pixel value P0(i,j) is the pixel corresponding to pixel P(i,j) of the current block 300 determined by bidirectional motion vector MV1 indicating the first reference picture, and the pixel value P1(i,j) of the second reference pixel is the pixel corresponding to pixel P(i,j) of the current block 300 determined by bidirectional motion vector MV2 indicating the second reference picture.

[0206] Also, the horizontal gradient value of the first reference pixel is

[0207]

number

[0208]

number

[0209]

number

[0210]

number

[0211] Assuming that there is a small motion determined in the video sequence, the pixel in the first corresponding region 310 of the first reference picture that is most similar to the current pixel P(i,j) bidirectionally motion compensated in pixel group units is not the first reference pixel P0(i,j), but is assumed to be the first displaced reference pixel PA obtained by moving the first reference pixel P0(i,j) by a predetermined displacement vector. Assuming that there is a small motion determined in the video sequence as described above, the pixel in the second corresponding region 320 of the second reference picture that is most similar to the current pixel P(i,j) can be estimated to be the second displaced reference pixel PB obtained by moving the second reference pixel P1(i,j) by a predetermined displacement vector.

[0212] The displacement vector is also composed of the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction described above. Therefore, the pixel group-based motion compensation unit 165 calculates the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction that constitute such a displacement vector, and performs motion compensation in pixel group units using the calculated vectors.

[0213] Optical flow refers to the pattern of apparent motion of an object or surface induced by the relative motion between an observer (eye or a video image capturing device such as a camera) and a scene. In a video sequence, the optical flow can also be expressed by calculating the motion between frames captured at any time t and t+△t. The pixel value located at (x,y) in a frame at time t is also defined as I(x,y,t). That is, I(x,y,t) is also a value that changes spatiotemporally. When I(x,y,t) is differentiated with respect to time t, it is given by the following Equation (1).

[0214]

number

[0215]

number

[0216] The pixel group unit motion compensation unit 165 calculates a displacement vector Vx in the x-axis direction and a displacement vector Vy in the y-axis direction according to Equation (2), and performs motion compensation in pixel group units using the displacement vectors Vx and Vy. In Equation (2), since the pixel value I(x,y,t) is the value of the original signal, using the original signal value as it is would cause a lot of overhead during encoding. Therefore, the pixel group unit motion compensation unit 165 can calculate the displacement vectors Vx and Vy according to Equation (2) using pixels of the first and second reference pictures determined as a result of the bidirectional motion prediction in block units. That is, the pixel group unit motion compensation unit 165 determines the x-axis displacement vector Vx and the y-axis displacement vector Vy that minimize Δ within a window (Ωij) of a predetermined size including surrounding pixels centered on the current pixel P(i,j) to be bidirectionally motion compensated. It is most desirable that Δ is 0, but since there is no x-axis direction displacement vector Vx or y-axis direction displacement vector Vy that satisfies the case where Δ is 0 for all pixels in the window (Ωij), the x-axis direction displacement vector Vx and y-axis direction displacement vector Vy that minimize Δ are determined. The process of obtaining the displacement vectors Vx and Vy will be described in detail with reference to FIG. 8A.

[0217] In order to determine the predicted pixel value of the current pixel, a function P(t) related to t may be determined as shown in Equation (3) below.

[0218]

number

[0219] The temporal distance from the first reference picture (assuming that the first reference picture is located temporally prior to the current picture) to the current picture is τ0 and the temporal distance from the second reference picture (assuming that the second reference picture is located later in time than the current picture) to the current picture is τ 1 Then, the reference pixel value in the first reference picture is P(-τ 0 ), and the reference pixel value in the second reference picture is equal to P(τ 1 ) is the same as τ 0 and τ 1 Assume that both are identical as τ.

[0220] The coefficients of each degree of P(t) are also determined by the following formula (4): where P0(i,j) means the pixel value at the (i,j) position of the first reference picture, and P1(i,j) means the pixel value at the (i,j) position of the second reference picture.

[0221]

number

[0222]

number

[0223]

number

[0224] For the sake of convenience, the process of determining a predicted pixel value of a current pixel has been described in the above in which the temporal distance from a first reference picture to a current picture is τ and the temporal distance from a second reference picture to a current picture is also the same as τ. 0 and the temporal distance from the second reference picture to the current picture is τ 1 In this case, the predicted pixel value P(0) of the current pixel can also be determined as shown in Equation (7).

[0225]

number

[0226]

number

[0227] For example, as shown in FIG. 3B, a first reference picture including a first corresponding area and a second reference picture including a second corresponding area may both be located temporally before a current picture including a current block in display order.

[0228] In this case, in Equation (8) derived with reference to FIG. 3A, τ 1 , -τ 1 The predicted pixel value P(0) of the current pixel is determined by Equation (9) substituted with .

[0229]

number

[0230] In this case, in Equation (8) derived with reference to FIG. 3A, τ 0 , -τ 0 The predicted pixel value of the current pixel is also determined by Equation (10) substituted with

[0231]

number

[0232] 4 is a reference diagram illustrating a process of calculating horizontal and vertical gradient values ​​according to an embodiment. Referring to FIG. 4, a horizontal gradient value of a first reference pixel P0(i,j) 410 of a first reference picture is

[0233]

number

[0234]

number

[0235]

number

[0236]

number

[0237]

number

[0238] According to one embodiment, instead of calculating the change in pixel value at the sub-pixel position and calculating the gradient value as in Equation (11), a predetermined filter can be used to calculate the gradient value at each reference pixel, where the filter coefficients of the predetermined filter are also determined from the coefficients of the interpolation filter used to calculate the pixel value at the sub-pixel position, taking into account the linearity of the filter.

[0239] FIG. 5 is a reference diagram for explaining a process of calculating horizontal and vertical gradient values ​​according to another embodiment.

[0240] According to another embodiment, the gradient value may be determined by applying a predetermined filter to the pixels of the reference picture. Referring to FIG. 5, the video decoding apparatus 100 performs a process of determining a horizontal gradient value of a reference pixel P0 500 on the left side of the reference pixel P0 500. Max 520 pixels, and |M on the right Min A predetermined filter is applied to the | pixels 510 to calculate the horizontal gradient value of P0 500. At this time, the filter coefficients used are the M 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , Max Integer pixels and M Min It is also determined by an α value indicating an interpolation position with an integer pixel (fractional pixel position). For example, referring to FIG. 7A, M Min is -2, and M Max is 3, and the pixel is about 1 / 4 away from the reference pixel P0 500, i.e., α=1 / 4, the filter coefficients {4, -17, -36, 60, -15, 4} in the second row of FIG. 7A are applied to the surrounding pixels P-2, P-1, P0, P1, P2, and P3. In this case, the horizontal gradient value of the reference pixel 500 is

[0241]

number

[0242]

number

[0243] 6A and 6B are diagrams illustrating a process of determining horizontal and vertical gradient values ​​using a one-dimensional filter according to an embodiment.

[0244] Referring to FIG. 6A, in order to determine a horizontal gradient value of a reference pixel in a reference picture, filtering is performed using a plurality of one-dimensional filters for integer pixels. Motion compensation by pixel group is additional motion compensation performed after motion compensation by block. Therefore, in the process of motion compensation by block, the reference position of the reference block of the current block indicated by the motion vector is also a fractional pixel position, and motion compensation by pixel group is performed for the reference pixel in the reference block at the fractional pixel position. Therefore, filtering is performed taking into consideration determining the gradient value of the pixel at the fractional pixel position.

[0245] 6A, the video decoding apparatus 100 may first perform filtering on pixels located in the horizontal or vertical direction from neighboring integer pixels of a reference pixel in a reference picture using a first one-dimensional filter. Similarly, the video decoding apparatus 100 may perform filtering on neighboring integer pixels located in a different row or column from the reference pixel using the first one-dimensional filter. The video decoding apparatus 100 may generate a horizontal gradient value of the reference pixel by filtering the value generated by the filtering using a second one-dimensional filter.

[0246] For example, if the position of the reference pixel is a fractional pixel position of (x+α, y+β) (x and y are integers, and α and β are fractional), then the integer pixels (x, y) and (x-1, y), (x+1, y), ..., (x+M Min ,y),(x+M Max ,y)(M Min , M Max , and , are integers), filtering is performed using a one-dimensional vertical interpolation filter as shown in the following equation (12).

[0247]

number

[0248] That is, the first one-dimensional filter is also an interpolation filter for determining fractional pixel values ​​in the vertical direction. 1 is an offset to prevent rounding errors, and shift 1 means the number of inverse scaling bits. Temp[i,j+β] means the pixel value at the fractional pixel position (i,j+β). Temp[i',j+β] (i' is the pixel value at i+M excluding i) min From i+M max , which is an integer up to , is also determined by replacing i with i' using equation (12).

[0249] Next, the video decoding device 100 may perform filtering on the pixel value at the sub-pixel position (i, j+β) and the pixel value at the sub-pixel position (i', j+β) using a second one-dimensional filter.

[0250]

number

[0251] That is, according to Equation (13), the video decoding device 100 performs filtering using a gradient filter (gradFilterα) on a pixel value (Temp[i,j+β]) at (i,j+β) and a pixel value (Temp[i',j+β]) located vertically from the pixel position (i,j+β) to obtain a horizontal gradient value at (i+α,j+β)

[0252]

number

[0253] Although the description has been given above of first applying an interpolation filter and then applying a gradient filter to determine a gradient value in the horizontal direction, the present invention is not limited thereto, and the gradient filter may be applied first and then an interpolation filter may be applied to determine a gradient value in the horizontal direction. In the following, an embodiment will be described in which a gradient filter is applied and then an interpolation filter is applied to determine a gradient value in the horizontal direction.

[0254] For example, if the position of the reference pixel is a fractional pixel position of (x+α, y+β) (x and y are integers, and α and β are fractional), then the integer pixels (x, y) and (x-1, y), (x+1, y), ..., (x+M Min ,y),(x+M Max ,y)(M Min , M Mmax A first one-dimensional filter is used for x, y, and z (all of which are integers) to perform filtering as shown in Equation (14) below.

[0255]

number

[0256] That is, the first one-dimensional filter is also an interpolation filter for determining the horizontal gradient value of a pixel at a position where the horizontal component of the pixel position is a fraction. 3 is an offset to prevent rounding errors, and shift 3 means the number of inverse scaling bits. Temp[i+α,j] means the horizontal gradient value at pixel position (i+α,j). Temp[i+α,j'] (j' is j+M excluding i) min From j+M max , j' is also determined by equation (14).

[0257] Next, the video decoding device 100 can perform filtering on the horizontal gradient value at pixel position (i+α,j) and the horizontal gradient value at pixel position (i+α,j') using a second one-dimensional filter as shown in the following equation (15).

[0258]

number

[0259] That is, according to Equation (15), the video decoding device 100 performs filtering using a gradient filter (fracFilterβ) on the horizontal gradient value (Temp[i+α,j]) at (i+α,j) and the horizontal gradient value (Temp[i+α,j']) of a pixel located vertically from the pixel position (i+α,j), to obtain a horizontal gradient value (Temp[i+α,j']) at (i+α,j+β).

[0260]

number

[0261] Referring to FIG. 6B, in order to determine a vertical gradient value of a reference pixel in a reference picture, filtering is performed using a plurality of one-dimensional filters for integer pixels. Motion compensation by pixel group is additional motion compensation performed after motion compensation by block. Therefore, in the process of motion compensation by block, the reference position of the reference block of the current block indicated by the motion vector is also a sub-pixel position, and motion compensation by pixel group is also performed for the reference pixel in the reference block at the sub-pixel position. Therefore, filtering is also performed in consideration of determining the gradient value of the pixel at the sub-pixel position.

[0262] 6B, the video decoding apparatus 100 may first perform filtering on pixels located in a horizontal or vertical direction from a neighboring integer pixel of a reference pixel in a reference picture using a first one-dimensional filter. Similarly, the video decoding apparatus 100 may perform filtering on adjacent pixels located in a different column or row from the reference pixel using the first one-dimensional filter. The video decoding apparatus 100 may generate a vertical gradient value of the reference pixel by filtering a value generated by the filtering using a second one-dimensional filter.

[0263] For example, if the position of the reference pixel is a fractional pixel position of (x+α, y+β) (x, y are integers, and α, β are fractional), then the integer pixels (x, y) and (x-1, y-1), (x+1, y+1), ..., (x+M Min ,y+M Min ),(x+M Max ,y+M max ) (MMin, MMmax are both integers) is filtered using a first one-dimensional filter as shown in the following equation (16).

[0264]

number

[0265] That is, the first one-dimensional filter is also an interpolation filter for determining a pixel value at a sub-pixel position α in the horizontal direction. 5 is an offset to prevent rounding errors, and shift 5 means the number of inverse scaling bits.

[0266] Temp[i+α,j] means the pixel value at the sub-pixel position (i+α,j). Temp[i+α,j'] (j' is j+M excluding j) min From j+M max , j' is also determined by equation (16).

[0267] Next, the video decoding device 100 can perform filtering on the pixel value at pixel position (i+α,j) and the pixel value at pixel position (i+α,j') using a second one-dimensional filter as shown in the following equation (17).

[0268]

number

[0269] That is, according to Equation (17), the video decoding device 100 applies a gradient filter (gradFilter β ) to filter the vertical gradient value at (i+α,j+β)

[0270]

number

[0271] Although the description has been given above of first applying an interpolation filter and then applying a gradient filter to determine a gradient value in the vertical direction, the present invention is not limited thereto, and the gradient filter may be applied first and then an interpolation filter may be applied to determine a gradient value in the horizontal direction. Hereinafter, an embodiment will be described in which a gradient filter is applied and then an interpolation filter is applied to determine a gradient value in the vertical direction.

[0272] For example, if the position of the reference pixel is a fractional pixel position of (x+α, y+β) (x, y are integers, and α, β are fractional), then integer pixels (x, y) and (x, y-1), (x, y+1), ..., (x, y+M Min ),(x,y+M max )(M Min , M max A first one-dimensional filter is used for x, y, and z (all of which are integers) to perform filtering as shown in Equation (18) below.

[0273]

number

[0274] That is, the first one-dimensional filter is also an interpolation filter for determining the vertical gradient value of a pixel at a position where the vertical component of the pixel position is a fraction. 7 is an offset to prevent rounding errors, and shift 7 means the number of inverse scaling bits.

[0275] Temp[i,j+β] is the vertical gradient value at pixel position (i,j+β). Temp[i',j+β] (i' is the vertical gradient value of i+M excluding i) min From i+M max (which is an integer up to 1000) is also determined by replacing i with i' using equation (18).

[0276] Next, the video decoding device 100 can perform filtering on the vertical gradient value at pixel position (i, j+β) and the vertical gradient value at pixel position (i', j+β) using a second one-dimensional filter as shown in the following equation (19).

[0277]

number

[0278] That is, according to Equation (19), the video decoding device 100 calculates an interpolation filter (fracFilter α ) to filter the vertical gradient value at (i+α,j+β)

[0279]

number

[0280] According to an embodiment, the video decoding device 100 determines the horizontal and vertical gradient values ​​at (i+α,j+β) by a combination of the various filters described above. For example, to determine the horizontal gradient value, the first one-dimensional filter may use an interpolation filter for determining pixel values ​​in the vertical direction, and the second one-dimensional filter may use a gradient filter for the horizontal gradient value. To determine the vertical gradient value, the first one-dimensional filter may use a gradient filter for determining gradient values ​​in the vertical direction, and the second one-dimensional filter may use an interpolation filter for determining pixel values ​​in the horizontal direction.

[0281] 7A-7E are tables illustrating filter coefficients for filters utilized to determine pixel values ​​at sub-pixel locations and horizontal and vertical gradient values ​​according to one embodiment.

[0282] 7A and 7B are tables showing filter coefficients for filters for determining horizontal or vertical gradient values ​​at fractional pixel positions in 1 / 4 pel units.

[0283] As described above, a one-dimensional gradient filter and a one-dimensional interpolation filter may be used to determine the horizontal or vertical gradient values. Referring to FIG. 7A, the filter coefficients of the one-dimensional gradient filter are shown. In this case, the gradient filter may be a 6-tap filter. The gradient filter coefficients are also coefficients scaled by 2^4. M min means the difference between the position of the farthest integer pixel in the negative direction that is applied to the filter, relative to the central integer pixel, and the position of the central integer pixel, and M max means the difference between the position of the farthest pixel among integer pixels in the positive direction to which the filter is applied, based on the central integer pixel, and the position of the central integer pixel. For example, the gradient filter coefficients for determining the horizontal gradient value of a pixel at a fractional pixel position α=1 / 4 in the horizontal direction are {4, -17, -36, 60, -15, -4}. The gradient filter coefficients for determining the horizontal gradient value of a pixel at a fractional pixel position α=0, 1 / 2, 3 / 4 in the horizontal direction may also be determined with reference to FIG. 7A.

[0284] Referring to FIG. 7B, the filter coefficients of a one-dimensional interpolation filter are shown. In this case, a 6-tap filter may be used as the interpolation filter. The filter coefficients of the interpolation filter are also coefficients scaled by 2^6. M minmeans the difference between the position of the farthest integer pixel in the negative direction that is applied to the filter, relative to the central integer pixel, and the position of the central integer pixel, and M max means the difference between the position of the farthest pixel among integer pixels in the positive direction applied to the filter based on the central integer pixel, and the position of the central integer pixel.

[0285] FIG. 7C is a table showing filter coefficients of a one-dimensional interpolation filter utilized to determine pixel values ​​at fractional pixel locations in 1 / 4-pel units.

[0286] As previously mentioned, two identical one-dimensional interpolation filters are also used horizontally and vertically to determine pixel values ​​at fractional pixel positions.

[0287] Referring to FIG. 7C, the filter coefficients of the one-dimensional interpolation filter are shown. At this time, the one-dimensional interpolation filter is also a 6-tap filter. The gradient filter coefficients are also coefficients scaled by 2^6. M min means the difference between the position of the farthest integer pixel in the negative direction that is applied to the filter, relative to the central integer pixel, and the position of the central integer pixel, and M max means the difference between the position of the farthest pixel among integer pixels in the positive direction applied to the filter based on the central integer pixel, and the position of the central integer pixel.

[0288] FIG. 7D is a table showing filter coefficients for a filter utilized to determine horizontal or vertical gradient values ​​at fractional pixel positions in 1 / 16 pel units.

[0289] As described above, a one-dimensional gradient filter and a one-dimensional interpolation filter may be used to determine gradient values ​​in the horizontal or vertical direction. Referring to FIG. 7D, filter coefficients of a one-dimensional gradient filter are illustrated. In this case, a 6-tap filter may be used as the gradient filter. The gradient filter coefficients may be coefficients scaled by 2^4. For example, the gradient filter coefficients for determining the horizontal gradient value of a pixel whose sub-pixel position α is 1 / 16 in the horizontal direction may be {8, -32, -13, 50, -18, 5}. The gradient filter coefficients for determining the horizontal gradient value of a pixel whose sub-pixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, 1 / 2 in the horizontal direction may also be determined using FIG. 7D. Meanwhile, the gradient filter coefficients for calculating the horizontal gradient values ​​of pixels with fractional pixel positions α of 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, and 15 / 16 may also be determined using the symmetry of the filter coefficients based on α=1 / 2. That is, the filter coefficients of the fractional pixel positions on the left side may be determined based on α=1 / 2 as shown in FIG. 7D, and the filter coefficients of the fractional pixel positions on the right side may be determined based on the remaining α=1 / 2. For example, the filter coefficients at α=15 / 16 may also be determined using the filter coefficients {8, -32, -13, 50, -18, 5} of α=1 / 16, which is a position symmetrical to α=1 / 2. That is, the filter coefficients when α=15 / 16 are determined as {5, -18, 50, -13, -32, 8} by arranging the filter coefficients {8, -32, -13, 50, -18, 5} in reverse order.

[0290] Referring to FIG. 7E, the filter coefficients of a one-dimensional interpolation filter are illustrated. In this case, a 6-tap filter may be used as the interpolation filter. The filter coefficients of the interpolation filter may be coefficients scaled by 2^6. For example, the one-dimensional interpolation filter coefficients for obtaining horizontal pixel values ​​of pixels whose sub-pixel positions α are 1 / 16 in the horizontal direction may be {1, -3, 64, 4, -2, 0}. The interpolation filter coefficients for obtaining horizontal pixel values ​​of pixels whose sub-pixel positions α are 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, 1 / 2 in the horizontal direction may also be determined using FIG. 7E. Meanwhile, the interpolation filter coefficients for obtaining horizontal pixel values ​​of pixels whose sub-pixel positions α are 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, 15 / 16 in the horizontal direction may also be determined using the symmetry of the filter coefficients based on α=1 / 2. That is, the filter coefficients of the left sub-pixel position may be determined based on α=1 / 2 as shown in FIG. 7E, and the filter coefficients of the right sub-pixel position may be determined based on the remaining α=1 / 2. For example, the filter coefficients of α=15 / 16 may be determined based on the filter coefficients {1,-3,64,4,-2,0} of α=1 / 16, which is the symmetrical position based on α=1 / 2. That is, the filter coefficients of α=15 / 16 may be determined as {0,-2,4,64,-3,1} by arranging the filter coefficients {1,-3,64,4,-2,0} in the reverse order.

[0291] FIG. 8A is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel according to an embodiment.

[0292] Referring to FIG. 8A, a window (Ωij) 800 of a predetermined size has a size of (2M+1)*(2N+1) (M and N are integers) centered on a bidirectionally predicted pixel P(i,j) in a current block.

[0293] Let P(i',j') be the bidirectionally predicted pixel of the current block in the window (where (i',j')∈Ωij, if iM≦i'≦i+M and jN≦j'≦j+N), let P0(i',j') be the pixel value of the first reference pixel in the first reference picture 810 corresponding to the bidirectionally predicted pixel P(i',j') of the current block, let P1(i',j') be the pixel value of the second reference pixel in the second reference picture 820 corresponding to the bidirectionally predicted pixel P(i',j') of the current block, and let P2(i',j') be the horizontal gradient value of the first reference pixel.

[0294]

number

[0295]

number

[0296]

number

[0297]

number

[0298]

number

[0299] The difference value Δi′j′ between the first displacement corresponding pixel PA′ and the second displacement corresponding pixel PB′ can also be determined by the following equation (21).

[0300]

number

[0301]

number

[0302]

number

[0303]

number

[0304]

number

[0305]

number

[0306]

number

[0307] A simplified solution of the above equation can be determined by first minimizing in the horizontal direction and then minimizing in the vertical direction. For example, if it is assumed that only the horizontal displacement vector is changed, it can be assumed that Vy=0 in the first equation of Equation (24), and therefore the equation: τVx=s3 / s1 can be determined.

[0308] In addition, by rearranging the second equation of equation (24) using the equation τVx=s3 / s1, it can be determined as the equation τVy=(s6-τVx*S2) / s5.

[0309] where the gradient value

[0310]

number

[0311] In the process of obtaining Vx(i,j) and Vy(i,j), adjustment parameters r and m are also introduced to prevent multiplication operations from being performed by 0 or very small values.

[0312] For convenience, it is assumed that Vx(i,j) and Vy(i,j) are in the opposite direction to that shown in Fig. 3A. For example, Vx(i,j) and Vy(i,j) derived by Equation (24) based on the direction of Vx(i,j) and Vy(i,j) shown in Fig. 3A are the same in magnitude as Vx(i,j) and Vy(i,j) determined in the opposite direction to that shown in Fig. 3A, with only a difference in sign between their values.

[0313] The first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' are also determined as shown in Equation (26) below: At this time, PA' and PB' are also determined using the first-order linear term of the local Taylor expansion.

[0314]

number

[0315]

number

[0316] [Number]

[0317] [Number] Φ(Vx, Vy) is a function with Vx and Vy as intermediate variables, and the maximum or minimum value is also determined by calculating the values that become 0 by partially differentiating Φ(Vx, Vy) with respect to Vx and Vy, as shown in the following mathematical formula (30).

[0318] [Number] That is, Vx and Vy are also determined as the Vx and Vy that minimize the Φ(Vx, Vy) value. To solve the optimization problem, first minimization is performed in the vertical direction, and then minimization is performed in the horizontal direction. By the minimization, Vx is also determined as shown in the following mathematical formula (31).

[0319] [Number] Here, the clip3(x, y, z) function is a function that outputs x if z < x, y if z > y, and z if x < z < y. According to mathematical formula (31), when s1 + r > m, Vx is clip3(-thBIO, thBIO, -s3 / (s1 + r)), and when s1 + r > m is not satisfied, Vx is also 0.

[0320] With the above minimization, Vy is also determined as in the following mathematical formula (32).

[0321]

Number

[0322] At this time, s1, s2, s3, and s5 are also determined as in the following mathematical formula (33). s4 can have the same value as s2.

[0323]

Number

[0324]

Number

[0325] However, without being limited thereto, the adjustment parameters r, m and thBIO may be determined based on information about the adjustment parameters obtained from the bitstream, which may be included in a slice header, a picture parameter set, a sequence parameter set, or various forms of high-level syntax carriers.

[0326] In addition, the adjustment parameters may be determined depending on whether or not temporally different bidirectional prediction is available. For example, when temporally different bidirectional prediction is available, thBIO diff If bidirectional predictions that are the same in time are available, then thBIO same Bigger, thBIO diff The size of thBIO same It's also twice the size.

[0327] FIG. 8B is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel group according to an embodiment.

[0328] Referring to FIG. 8B, a window (Ωij) 810 of a given size is centered on a KxK size pixel group 820 of multiple pixels that are not bidirectionally predicted pixels in the current block, and has a size of (2M+K+1)*(2N+K+1) (M and N are integers).

[0329] In this case, the difference from FIG. 8A is that the window size is larger, but apart from that, the horizontal and vertical displacement vectors for the pixel group can be determined in the same manner.

[0330] FIG. 8C is a reference diagram for explaining a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel according to an embodiment.

[0331] 8C, the video decoding apparatus 100 may determine a horizontal displacement vector and a vertical displacement vector for each pixel. Thus, a displacement vector 835 per unit time for each pixel may be determined. At this time, the horizontal displacement vector Vx[i,j] and the vertical displacement vector Vy[i,j] of the displacement vector 835 for each pixel may also be determined by the following Equation (35). Here, i,j represent the x-coordinate component and the y-coordinate component of the pixel. Here, σ 1 [i,j], σ 2 [i,j]σ 3 [i,j], σ 5 [i,j] and σ6 [i,j] are also s1, s2, s3, s5, and s6 in equation (33), respectively.

[0332]

number

[0333] 8D, the video decoding device 100 calculates σ 1 [i,j], σ 2 [i,j], σ 3 [i,j], σ 5 [i,j] and σ 6 [i,j] can be determined as shown in equation (35) above.

[0334] The video decoding device 100 calculates the horizontal displacement vector Vx[i,j] for the pixel group 840 by the pixel's σ 1 [i,j] and σ 3 It can be determined by using [i,j] as shown in Equation (36): where i,j are the x and y components of the top left coordinates of the pixel group.

[0335]

number

[0336] The video decoding device 100 is 2 [i,j], σ 5 [i,j], σ 6 Using Vx[i,j] and Vy[i,j], the vertical displacement vector Vy[i,j] for pixel group 840 can be determined as follows: where Vx[i,j] is also the value determined by equation (36):

[0337]

number

[0338] Meanwhile, while a detailed description has been given above with reference to FIG. 8D assuming that the pixel group has a size of 2x2, this is not limited thereto, and the pixel group may have a size of LxL (L is an integer).

[0339] In this case, the size L of the pixel group is also determined by the following Equation 38: W and H are the width and height of the current block, respectively.

[0340]

number

[0341] [Table 1] That is, when the video decoding device 100 performs motion compensation in units of pixel groups, the video decoding device 100 can perform division operations by reducing the number of division operations by a value obtained by dividing by L^2, which is a value corresponding to the size of the pixel group.

[0342] FIG. 9A is a diagram illustrating a process of determining a horizontal or vertical gradient value by performing inverse scaling after filtering and adding an offset, according to an embodiment.

[0343] Referring to FIG. 9A, the video decoding apparatus 100 may determine a gradient value in the horizontal or vertical direction by filtering a pixel having a predetermined directional component at an integer position using a first one-dimensional filter and a second one-dimensional filter. However, a value obtained by filtering a pixel having a predetermined directional component at an integer position using the first one-dimensional filter or the second one-dimensional filter may fall outside a predetermined range. Such a phenomenon is called an overflow phenomenon. The coefficients of the one-dimensional filter are also determined as integers to perform integer arithmetic instead of inaccurate and complicated decimal arithmetic. The coefficients of the one-dimensional filter may be scaled to be determined as integers. If filtering is performed using the coefficients of the scaled one-dimensional filter, integer arithmetic is possible, but the magnitude of the filtered value may be larger than that of a case where filtering is performed using a non-scaled one-dimensional filter, and an overflow phenomenon may occur. Therefore, in order to prevent the overflow phenomenon, inverse scaling may be performed after filtering using the one-dimensional filter. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits to the right. The number of inverse scaling bits is determined while maximizing the accuracy of calculation and taking into consideration the maximum number of bits of a register for a filtering operation and the maximum number of bits of a temporal buffer for storing a filtering operation result, In particular, the number of inverse scaling bits is determined based on an internal bit depth, a number of scaling bits for an interpolation filter, and a number of scaling bits for a gradient filter.

[0344] In the following, in order to determine a horizontal gradient value, a vertical interpolation filter is first used to perform filtering on pixels at integer positions to generate a vertical interpolation filtering value, and then a horizontal gradient filter is used to perform inverse scaling on the vertical interpolation filtering value during the filtering process.

[0345] According to the above formula (12), in order to determine the horizontal gradient value, the video decoding device 100 can first perform filtering on the pixels at integer positions using a vertical interpolation filter. 1 is also determined as b-8, where b is the internal bit depth of the input image. 1 Based on this, when actual inverse scaling is performed, the bit depth of the register (Reg Bitdepth) and the bit depth of the temporary buffer (Temp Bitdepth) will be described.

[0346] [Table 2] At this time, the values ​​of the variables in Table 2 are also determined by the following equation (39).

[0347]

number

[0348] For example, assuming the quarter-pel gradient filter fracFilter disclosed in FIG. 7C is utilized, FilterSumPos is also 88 and FilterSumNeg is also -24.

[0349] The Ceiling(x) function returns the smallest integer greater than or equal to the error x. 1 shift1 This is an offset value that is added to the filtered value to prevent rounding errors that may occur during the inverse scaling process using offset 1 is 2^(shift 1 -1) is also determined.

[0350] Referring to Table 2, when the internal bit depth b is 8, the bit depth of the register (RegBitdpeth) is 16, when the internal bit depth b is 9, the bit depth of the register is 17, and when the internal bit depth b is 10, 11, 12, and 16, the bit depth of the register is also 18, 19, and 24. If the register used for filtering is a 32-bit register, the bit depth of the register in Table 2 does not exceed 32, so that no overflow phenomenon occurs.

[0351] Similarly, if the internal bit depth b is 8, 9, 10, 11, 12, or 16, the bit depth of the temporary buffer (TempBitDepth) is 16. If the temporary buffer used to store the filtered and inverse scaled values ​​is a 16-bit buffer, the bit depth of the temporary buffer in Table 2 is 16, none of which exceeds 16, so no overflow phenomenon will occur.

[0352] According to Equation (12), in order to determine the horizontal gradient value, the video decoding device 100 first uses a vertical interpolation filter to filter pixels at integer positions to generate vertical interpolation filtering values, and then uses a horizontal gradient filter to filter the vertical interpolation filtering values ​​according to Equation (13). 2 is p+q-shift 1Here, p means the number of scaled bits for the interpolation filter including the filter coefficients shown in FIG. 7C, and q means the number of scaled bits for the gradient filter including the filter coefficients shown in FIG. 7A. For example, p is 6 and q is 4, so shift 2 =18-b.

[0353] shift 2 The reason why is determined as described above is that when the filter coefficients are upscaled and when the filter coefficients are not upscaled, the final filtering result value is the same, so the sum of the number of bits to be inversely scaled, shift 1 +shift 2 must be equal to the sum of the number of upscaled bits for the filter (p+q).

[0354] Please refer to Table 3 below. 2 Based on this, when actual inverse scaling is performed, the bit depth of the register (Reg Bitdepth) and the bit depth of the temporary buffer (Temp Bitdepth) will be described.

[0355] [Table 3] At this time, the values ​​of the variables in Table 3 are also determined by the following equation (40).

[0356]

number

[0357] offset 2 shift 2 This is an offset value that is added to the filtered value to prevent rounding errors that may occur during the inverse scaling process using offset 2 is 2^(shift 2 -1) is also determined.

[0358] shift 1 and shift 2 can be determined as described above, but is not limited thereto, and shift 1 and shift 2 shift in various ways so that the sum of and is the same as the sum of the number of scaling bits for the filter. 1 and shift 2 At this time, assuming that no overflow occurs, shift 1 Value and shift 2 The value is also determined. 1 and shift 2 is also determined based on the internal bit depth of the input image and the number of scaling bits for the filter.

[0359] However, it is not necessarily shift 1 and shift 2 shift so that the sum of and is the same as the sum of the number of scaling bits for the filter. 1 and shift 2 For example, shift1 is also determined as d-8, but shift 2 is also determined to be a fixed number.

[0360] shift 1 is the same as before, shift 2 When is a fixed integer 7, OutMax, OutMin and Temp Bitdepth may be different in the previously mentioned Table 3. Hereinafter, the bit depth of the temporary buffer (Temp Bitdepth) will be described with reference to Table 4.

[0361] [Table 4] Unlike Table 3, in the case of Table 4, the bit depth of the temporary buffer is the same as 16 for all b, and if the result data is stored using a 16-bit temporary buffer, no overflow occurs for the internal bit depth of any input image because the bit depth of the temporary buffer is smaller than 16. On the other hand, referring to Table 3, if the internal bit depth of the input image is 12 and 16, if the result data is stored using a 16-bit temporary buffer, an overflow phenomenon occurs because the bit depth of the temporary buffer is larger than 16.

[0362] shift 2 If is a fixed number, the result value when filtering is performed without using the scaled filter coefficients may differ from the result value when filtering is performed using the scaled filter coefficients and then inverse scaling is performed. In that case, it can be easily understood by a person skilled in the art that inverse scaling must be additionally performed.

[0363] In the above description, in order to determine a horizontal gradient value, a vertical interpolation filter is used to first filter pixels at integer positions to generate a vertical interpolation filtered value, and then a horizontal gradient filter is used to filter the vertical interpolation filtered value, during which inverse scaling is performed. However, the present invention is not limited to this, and those skilled in the art will easily understand that in a similar manner, inverse scaling can be performed when filtering is performed on pixels whose predetermined directional components are integers in order to determine horizontal and vertical gradient values ​​using various one-dimensional filter combinations.

[0364] FIG. 9B is a diagram illustrating a process of determining a horizontal or vertical gradient value by adding an offset and performing inverse scaling after filtering according to another embodiment.

[0365] 9B, the video decoding apparatus 100 may perform filtering using fractional pixels and integer pixels of a reference picture as input. Here, it is assumed that the fractional pixels of the reference picture are values ​​determined by applying one-dimensional filters in the horizontal and vertical directions to the integer pixels of the reference picture.

[0366] The video decoding device 100 may determine a gradient value in the horizontal or vertical direction by filtering pixels whose position components in a predetermined direction are decimal and integer pixels using a one-dimensional filter in the horizontal or vertical direction. However, a value obtained by filtering pixels whose position components in a predetermined direction are decimal and integer pixels using a first one-dimensional filter may fall outside a predetermined range. This phenomenon is called an overflow phenomenon. The coefficients of the one-dimensional filter may be determined as integers to perform integer arithmetic instead of inaccurate and complicated decimal arithmetic. The coefficients of the one-dimensional filter are also scaled to be determined as integers. If filtering is performed using the coefficients of a scaled one-dimensional filter, integer arithmetic is possible, but compared to a case where filtering is performed using a non-scaled one-dimensional filter, the magnitude of the filtered value may be larger, and an overflow phenomenon may occur. Therefore, in order to prevent the overflow phenomenon, inverse scaling may be performed after filtering using a one-dimensional filter. In this case, the inverse scaling is performed by adding an inverse scaling bit number (shift) to the right. 1 ) to the right. The number of inverse scaling bits is determined while maximizing the accuracy of the calculation and taking into consideration the maximum number of bits of a register for the filtering operation and the maximum number of bits of a temporary buffer for storing the filtering operation result. In particular, the number of inverse scaling bits is determined based on the internal bit depth and the number of scaling bits for the gradient filter.

[0367] FIG. 9C is a diagram illustrating a range required for determining a horizontal displacement vector and a vertical displacement vector in a process of performing pixel-by-pixel motion compensation on a current block.

[0368] 9C, in a process of performing pixel-by-pixel motion compensation on a reference block 910 corresponding to a current block, the video decoding apparatus 100 may determine a horizontal displacement vector per unit time and a vertical displacement vector per unit time at the pixel 915 by using a window 920 around a pixel 915 located at the upper left end of the reference block 910. In this case, the horizontal or vertical displacement vector per unit time may be determined by using pixel values ​​and gradient values ​​of pixels located outside the reference block 910. In a similar manner, in a process of determining a horizontal displacement vector and a vertical displacement vector for a pixel located at the boundary of the reference block 910, the video decoding apparatus 100 may determine pixel values ​​and gradient values ​​of pixels located outside the reference block 910. Thus, the video decoding apparatus 100 may determine a horizontal displacement vector and a vertical displacement vector per unit time by using a block 925 that is larger than the reference block 910. For example, if the size of the current block is AxB and the pixel-wise window size is (2M+1)x(2N+1), then the size of the range for determining the horizontal displacement vector and the vertical displacement vector is also (A+2M)x(B+2N).

[0369] 9D and 9E are diagrams illustrating ranges of areas used in the process of performing pixel-by-pixel motion compensation according to various embodiments.

[0370] 9D, in the process of performing pixel-based motion compensation, the video decoding apparatus 100 may determine horizontal and vertical displacement vectors per unit time for each pixel included in the reference block 930 based on a block 935 of a range extended according to the size of a window of pixels located at the boundary of the reference block 930. However, in the process of determining the horizontal and vertical displacement vectors per unit time, the video decoding apparatus 100 needs pixel values ​​and gradient values ​​of pixels located in the block 935, and may use an interpolation filter or a gradient filter to obtain the pixel values ​​and gradient values. In the process of using the interpolation filter or gradient filter for the boundary pixels of the block 935, pixel values ​​of surrounding pixels may be used, and thus pixels located outside the block boundary may be used. Therefore, pixel-based motion compensation may be performed using a block 940 of a range additionally extended by a value obtained by subtracting 1 from the number of taps of the interpolation filter or gradient filter. Thus, if the block size is NxN, the window size per pixel is (2M+1)x(2M+1), and the interpolation or gradient filter length is T, then it is also (N+2M+T-1)x(N+2M+T-1).

[0371] Referring to FIG. 9E, in the process of performing pixel-by-pixel motion compensation, the video decoding apparatus 100 may determine a horizontal displacement vector and a vertical displacement vector per unit time for each pixel by using pixel values ​​and gradient values ​​of pixels located in the reference block 945 without extending the reference block according to the window size of pixels located at the boundary of the reference block 945. In particular, a process for the video decoding apparatus 100 to determine a horizontal displacement vector and a vertical displacement vector per unit time for each pixel without extending the reference block will be described with reference to FIG. 9E. However, in order to obtain pixel values ​​and gradient values ​​of pixels, an interpolation filter or gradient filter of the reference block 945 is used, and pixel-by-pixel motion compensation may be performed by using the extended block 950. Therefore, if the size of the block is NxN, the window size for each pixel is (2M+1)x(2M+1), and the interpolation filter length or gradient filter length is T, it is also (N+T-1)x(N+T-1).

[0372] FIG. 9F is a diagram for explaining a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block.

[0373] 9F, for a pixel located outside the boundary of the reference block 955, the video decoding device 100 adjusts the position of the pixel to the position of the closest available pixel among the pixels located within the boundary of the reference block 955, and the pixel value and gradient value of the pixel located outside the boundary are also determined as the pixel value and gradient value of the closest available pixel. In this case, the video decoding device 100 determines the position of the pixel located outside the boundary of the reference block 955 according to the mathematical formula:

[0374]

number

[0375]

number

[0376] Here, i' is the x-coordinate value of the pixel, j' is the y-coordinate value of the pixel, and H and W are the height and width of the reference block. In this case, the position of the upper left corner of the reference block is assumed to be (0,0). If the position of the upper left corner of the reference block is (xP, yP), then the position of the final pixel is also (i'+xP, j'+yP).

[0377] Also, referring to FIG. 9D, in a block 935 expanded according to the size of the pixel-by-pixel window, the positions of pixels located outside the boundary of the reference block 930 are adjusted to the positions of adjacent pixels within the boundary of the reference block 930, and as shown in FIG. 9E, the video decoding device 100 can determine horizontal displacement vectors and vertical displacement vectors per unit time for each pixel within the reference block 945 using the pixel values ​​and gradient values ​​of the reference block 945.

[0378] Therefore, the video decoding device 100 does not expand the reference block 945 according to the pixel-by-pixel window size, but instead performs pixel-by-pixel motion compensation, thereby reducing the number of memory accesses for pixel value references, reducing the number of multiplication operations, and reducing the computational complexity.

[0379] When the video decoding device 100 performs block-based motion compensation (as when operating according to the HEVC standard), when it performs pixel-based motion compensation with block extension according to a window size, and when it performs pixel-based motion compensation without block extension, the memory access operations and multiplication operations corresponding to the respective numbers of memory accesses and multiplication operations may be performed in the video decoding device 100 as shown in Table 5 below. In this case, it is assumed that the signal (interpolation) filter length is 8, the gradient filter length is 6, the block size is NxN, and the pixel-by-pixel window size 2M+1 is 5.

[0380] [Table 5] In block-based motion compensation according to the HEVC standard, an 8-tap interpolation filter is used for one sample, so 8 surrounding samples are required, and therefore, when the size of the reference block is NxN, (N+7)x(N+7) reference samples are required according to 8-tap interpolation, and two reference blocks are used because of bidirectional motion prediction compensation, so block-based motion compensation according to the HEVC standard requires 2*(N+7)x(N+7) memory accesses as shown in Table 4. In the case of performing pixel-based motion compensation together with block extension, if M=2, and an 8-tap interpolation filter or gradient filter is used to perform pixel-based motion compensation for an extended block of (N+4)x(N+4), (N+4+7)x(N+4+7) reference samples are required, and two reference blocks are used because of bidirectional motion prediction compensation, so finally, pixel-based motion compensation performed together with block extension requires 2*(N+4+7)x(N+4+7) memory accesses as shown in Table 5.

[0381] When pixel-based motion compensation is performed without block extension, (N+7)x(N+7) reference samples are required, as in block-based motion compensation according to the HEVC standard, because there is no block extension, and two reference blocks are used because of bidirectional motion prediction compensation. Therefore, pixel-based motion compensation performed without block extension requires 2*(N+7)x(N+7) memory accesses, as shown in Table 5.

[0382] On the other hand, in block-based motion compensation according to the HEVC standard, an 8-tap interpolation filter is used for each sample, so the number of samples required for the first horizontal interpolation is (N+7)xN samples. The number of samples required for the second vertical interpolation is NxN samples. On the other hand, the number of multiplication operations required per 8-tap interpolation filter is 8, and since bidirectional motion prediction compensation is used, two reference blocks are used, so block-based motion compensation according to the HEVC standard requires 2*8*{(N+7)*N+N*N) multiplication operations as shown in Table 5.

[0383] When performing pixel-by-pixel motion compensation together with block extension, the size of the block is extended to perform pixel-by-pixel motion compensation, so that an 8-tap interpolation filter is used for the extended (N+4)x(N+4) size block to determine the pixel value at the decimal pixel position, as shown in Table 5, requiring a total of 2*8*{(N+4+7)x(N+4)+(N+4)x(N+4)} multiplication operations.

[0384] On the other hand, when pixel-wise motion compensation is performed together with block extension, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine horizontal or vertical gradient values. As the size of the block is extended, a total of 2*6*{(N+4+5)x(N+4)+(N+4)x(N+4)}*2 multiplication operations are required to determine gradient values ​​using a 6-tap interpolation filter and gradient filter for an extended block of size (N+4)x(N+4) as shown in Table 5.

[0385] When pixel-based motion compensation is performed without block extension, (N+7)x(N+7) reference samples are required as in block-based motion compensation according to the HEVC standard because there is no block extension, and two reference blocks are used because of bidirectional motion prediction compensation. Therefore, in pixel-based motion compensation performed without block extension, an 8-tap interpolation filter is used for an NxN size block, and 2*8*{(N+7)xN+NxN} multiplication operations are required to determine pixel values ​​at fractional pixel positions, as shown in Table 5.

[0386] On the other hand, when pixel-based motion compensation is performed without block extension, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine horizontal or vertical gradient values. For an NxN block, a total of 2*6*{(N+5)xN+NxN}*2 multiplication operations are required to determine gradient values ​​using a 6-tap interpolation filter and gradient filter, as shown in Table 5.

[0387] 9G to 9I are diagrams illustrating a process for determining a horizontal displacement vector and a vertical displacement vector without extending a reference block according to another embodiment.

[0388] As described above with reference to FIG. 9D, in the process of performing pixel-by-pixel motion compensation, the video decoding apparatus 100 may determine a horizontal displacement vector and a vertical displacement vector per unit time for each pixel included in the reference block 930 based on a block 935 of a range expanded according to a window size of a pixel located at a boundary of the reference block 930. For example, when the window size is (2M+1)×(2M+1), the video decoding apparatus 100 may determine a horizontal displacement vector and a vertical displacement vector for each pixel by referring to pixel values ​​and gradient values ​​of pixels that are off the reference block 930 by M when a window is applied to a pixel located at a boundary of the reference block 930.

[0389] Hereinafter, a method will be described in which the video decoding device 100 determines values ​​(s1 to s6 in Equation (33)) for determining horizontal and vertical displacement vectors for each pixel by using only pixel values ​​and gradient values ​​of a reference block corresponding to a current block without referring to pixel values ​​and gradient values ​​of pixels outside the reference block, according to another embodiment. Here, it is assumed that the window size is 5x5. For convenience, pixels located horizontally around the current pixel will be described. Those skilled in the art will easily understand that weights can also be determined in the same manner for pixels located vertically around the current pixel.

[0390] The video decoding apparatus 100 determines pixel values ​​P0(i',j' and P1(i',j'), horizontal or vertical gradient values ​​P0(i',j' and P1(i',j'), and horizontal or vertical gradient values ​​P1(i',j') for each pixel included in a window centered on a current pixel for which a horizontal or vertical displacement vector is to be determined.

[0391]

number

[0392] In this case, the result value of the operation performed for each pixel is multiplied by the weight value so that each pixel has the same weight value, and the resulting values ​​are added up to determine values ​​s1 to s6 for determining the horizontal and vertical displacement vectors for each pixel.

[0393] 9G, in the process of determining horizontal and vertical displacement vectors for a current pixel 961 in a current block 960, the video decoding apparatus 100 may determine weights for the pixels in the window to have the same value of 1. The video decoding apparatus 100 may multiply the resultant values ​​calculated for each pixel by the weights determined for each pixel, and add all the resultant values ​​to determine values ​​s1 to s6 for determining a horizontal displacement vector and a vertical displacement vector for the current pixel.

[0394] 9H, in a current block 970, if a current pixel 971 is immediately adjacent to the boundary of the current block 970, the video decoding device 100 may determine a weight value of a pixel adjacent to the boundary of the block 970 that is located outside the boundary of the current block 970 to be 3. The video decoding device 100 may determine a weight value of a pixel 973 other than the current block 970 to be 1.

[0395] 9H, when a current pixel 981 is located near a boundary of the current block 980 (when the current pixel is about one pixel away from the boundary), video decoding device 100 may determine a weighting value of a pixel 982 located outside the boundary of the current block 980 to be 0, and a weighting value of a pixel 983 adjacent to the boundary of the current block 980 to be 2. Video decoding device 100 may determine a weighting value of a pixel 984 other than the current pixel 981 to be 1.

[0396] As described with reference to Figures 9G to 9I, the video decoding device 100 assigns different weights to each pixel in the window depending on the position of the current pixel, thereby being able to determine values ​​s1 to s6 for determining horizontal and vertical displacement vectors for each pixel using pixel values ​​and gradient values ​​of pixels located inside the reference block rather than using pixel values ​​and gradient values ​​of pixels located outside the reference block corresponding to the current block.

[0397] FIG. 9J is a diagram illustrating a process of determining horizontal and vertical displacement vectors for each pixel by applying exponential smoothing techniques up, down, left, and right, without extending the block, by referring to pixel values ​​and gradient values ​​of a reference block, according to one embodiment.

[0398] 9J, the video decoding apparatus 100 may obtain pixel values ​​P0(i',j') and P1(i',j') of corresponding reference pixels included in a corresponding reference block for each pixel included in the current block 990, horizontal or vertical gradient values ​​of corresponding reference pixels included in the corresponding reference block, and

[0399]

number

[0400] The video decoding device 100 combines the function operation result value of the current pixel with the function operation result value of the surrounding pixels to obtain values ​​s1 to s6 (σ k(k=1,2,3,4,5,6) ) is also determined. That is, values ​​s1 to s6 for determining the horizontal and vertical displacement vectors of the current pixel are expressed as a weighted average of the operation performance values ​​for the current pixel and the surrounding pixels as shown in the following Equation (41). At this time, the position coordinates of the pixel included in the window Ω are (i',j'). Also, W[i',j'] means a weight value related to the pixel included in the window Ω. Here, the size of the window Ω is also (2M+1)x(2M+1) (M is an integer). Also, the function A k [i',j'] is the pixel value P0(i',j') and P1(i',j') (I[i',j'](0,1)) of the corresponding reference pixel for the pixel at position (i',j') included in the window Ω, and the horizontal or vertical gradient value of the corresponding reference pixel included in the pixel of the corresponding reference block.

[0401]

number

[0402]

number

[0403] [Table 6] Here I x (0) [i',j'], I x (1) [i',j'] are

[0404]

number

[0405]

number

[0406] The video decoding device 100 includes k To determine the weighted average over [i',j'], A k Exponential smoothing techniques can be applied to [i',j'] in the up, down, left and right directions to perform averaging.

[0407] Referring to FIG. 9J, the video decoding device 100 first performs a k An exponential smoothing technique can be applied to [i',j']. In the following, the video decoding device 100 performs A k Apply exponential smoothing to [i',j'] and A k The content of determining the weighted average value related to [i', j'] will be described in detail.

[0408] First, the video decoding device 100 performs averaging in the right direction by applying an exponential smoothing technique. The video decoding device 100 performs averaging in the right direction by using the following pseudo code 1. k For [i',j'], averaging can be performed to the right. In this case, H means the height of the current block, W means the width of the current block, and Stride means the distance between one line and the next line in the one-dimensional array. In other words, the two-dimensional array A[i,j] can also be expressed by the one-dimensional array A[i+j*Stride]. [Pseudocode 1]

[0409]

number

[0410] The video decoding device 100 performs left-side averaging by applying the exponential smoothing technique. k For [i',j'], averaging can be performed to the left. In this case, H means the height of the current block, W means the width of the current block, and Stride means the distance between one line and the next line in the one-dimensional array. In other words, the two-dimensional array A[i,j] can also be expressed by the one-dimensional array A[i+j*Stride]. [Pseudocode 2]

[0411]

number

[0412] The video decoding device 100 performs downward averaging by applying an exponential smoothing technique. k Averaging can be performed downward for [i',j']. In this case, H means the height of the current block, W means the width of the current block, and Stride means the distance between one line and the next line in the one-dimensional array. In other words, the two-dimensional array A[i,j] can also be expressed by the one-dimensional array A[i+j*Stride]. [Pseudocode 3]

[0413]

number

[0414]

number

[0415] Therefore, the video decoding apparatus 100 uses an exponential smoothing technique on the current block 990 to average in the up, down, left, and right directions, thereby obtaining a weighted average σ necessary for determining a horizontal or vertical displacement vector for each pixel. kThat is, the video decoding apparatus 100 may determine a horizontal or vertical displacement vector for each pixel by referring only to pixel values ​​and gradient values ​​of a reference block corresponding to the current block 990, without referring to pixel values ​​and gradient values ​​of a reference block corresponding to the block 996 obtained by extending the current block 990.

[0416] FIG. 9K is a diagram illustrating a process of determining pixel values ​​of reference pixels in a reference block and horizontal and vertical gradient values ​​using a filter to perform motion compensation for a current block according to one embodiment.

[0417] 9K, the video decoding apparatus 100 may perform pixel-based and block-based motion compensation for the current block using pixel values ​​and gradient values ​​of reference pixels in a reference block corresponding to the current block. Therefore, in order to perform pixel-based and block-based motion compensation for the current block, pixel values ​​and gradient values ​​of reference pixels in a reference block corresponding to the current block must be determined. At this time, the pixel values ​​and gradient values ​​of the reference pixels in the reference block may be in fractional pixel units. For example, the pixel values ​​and gradient values ​​of the reference pixels in the reference block may be in 1 / 16 pixel (1 / 16 pel) units.

[0418] The video decoding device 100 may perform filtering on pixel values ​​of integer pixels of the reference block to determine pixel values ​​and gradient values ​​of reference pixels in the fractional pixel unit reference block.

[0419] First, the video decoding device 100 can apply a horizontal 8-tap signal filter (also called an interpolation filter) and a vertical 8-tap signal filter to the pixel value of the integer pixel of the reference pixel to determine the pixel value of the reference pixel in the reference block.

[0420] The video decoding device 100 can determine pixel values ​​of reference pixels having fractional pixel position components in the horizontal direction by applying an 8-tap signal filter in the horizontal direction to integer pixel values ​​of the reference block to perform filtering, and store the pixel values ​​in the buffer. The video decoding device 100 can determine pixel values ​​of reference pixels having fractional pixel position components in the vertical direction by applying an 8-tap signal filter in the vertical direction to pixel values ​​of reference pixels having integer position components in the vertical direction.

[0421] The video decoding device 100 can determine the horizontal gradient value of the reference pixel in the reference block by first applying a vertical 6-tap signal filter to the integer pixel values ​​of the reference block, and then applying a horizontal 6-tap gradient filter to perform filtering.

[0422] That is, the video decoding device 100 can determine pixel values ​​of reference pixels having position components in the vertical direction in units of fractional pixels by applying a vertical 6-tap signal filter to the integer pixel values ​​of the reference block to perform filtering, and store the pixel values ​​in a buffer.

[0423] The video decoding device 100 applies a horizontal 6-tap gradient filter to pixel values ​​of reference pixels having integer-unit position components in the horizontal direction, and can also determine horizontal gradient values ​​of reference pixels having fractional-pixel unit position components in the horizontal direction.

[0424] The video decoding device 100 can determine the vertical gradient value of the reference pixel in the reference block by first applying a vertical 6-tap gradient filter to the integer pixel values ​​of the reference block, and then applying a horizontal 6-tap signal filter to perform filtering.

[0425] That is, the video decoding device 100 can determine gradient values ​​of reference pixels having fractional pixel position components in the vertical direction by applying a vertical 6-tap gradient filter to the integer pixel values ​​of the reference block to perform filtering, and store the gradient values ​​in a buffer.

[0426] The video decoding device 100 can apply a horizontal 6-tap signal filter to the vertical gradient values ​​of reference pixels having integer-unit position components in the horizontal direction, and determine vertical gradient values ​​of reference pixels having fractional-pixel unit position components in the horizontal direction as well.

[0427] That is, the video decoding apparatus 100 applies two one-dimensional filters to determine a pixel value of a reference pixel in a reference block, a horizontal gradient value of a reference pixel in a reference block, and a vertical gradient value of a reference pixel in a reference block, respectively, and performs a multiplication operation between a coefficient of each filter and a value related to a corresponding pixel. For example, two 6-tap signal / gradient filters are used to determine a horizontal gradient value of a reference pixel in a reference block, and a total of 12 multiplication operations are performed per pixel. Also, two 6-tap signal / gradient filters are used to determine a vertical gradient value of a reference pixel in a reference block, and a total of 12 multiplication operations are performed per pixel.

[0428] FIG. 9L is a diagram illustrating a process of using a filter to determine pixel values ​​of reference pixels in a reference block, as well as horizontal and vertical gradient values, to perform motion compensation for a current block according to another embodiment.

[0429] 9L, the video decoding apparatus 100 may perform pixel-based and block-based motion compensation for the current block using pixel values ​​and gradient values ​​of reference pixels in a reference block corresponding to the current block. Therefore, in order to perform pixel-based and block-based motion compensation for the current block, pixel values ​​and gradient values ​​of reference pixels in a reference block corresponding to the current block must be determined. At this time, the pixel values ​​and gradient values ​​of the reference pixels in the reference block may be in fractional pixel units. For example, the pixel values ​​and gradient values ​​of the reference pixels in the reference block may be in 1 / 16 pixel (1 / 16 pel) units.

[0430] The video decoding device 100 may perform filtering on pixel values ​​of integer pixels of the reference block to determine pixel values ​​and gradient values ​​of reference pixels in the fractional pixel unit reference block.

[0431] Unlike the description with reference to Fig. 9K, the video decoding apparatus 100 may first apply a horizontal 8-tap signal filter and a vertical 8-tap signal filter to integer pixel values ​​of the reference block to determine pixel values ​​of the reference pixels in the reference block, and then apply a horizontal 5-tap gradient filter to the pixel values ​​of the reference pixels in the reference block to determine horizontal gradient values ​​of the reference pixels in the reference block. Also, the video decoding apparatus 100 may apply a vertical 5-tap gradient filter to the pixel values ​​of the reference pixels in the reference block to determine vertical gradient values ​​of the reference pixels in the reference block.

[0432] The video decoding device 100 can apply two one-dimensional signal (interpolation) filters to determine a pixel value of a reference pixel having a fractional pixel position, and then apply two one-dimensional gradient filters in parallel to the pixel value of the reference pixel located at the fractional pixel position to determine a horizontal gradient value or a vertical gradient value of the reference pixel in the reference block.

[0433] The video decoding device 100 may apply a 5-tap horizontal gradient filter (with filter coefficients of {9, -48, 0, 48, 9}; however, the filter coefficients are not limited thereto) to the pixel value of the reference pixel as shown in the following Equation (42) to determine a horizontal gradient value Ix(k) of the reference pixel in the reference block. Here, k may have a value of 0 or 1, which may indicate reference pictures 0 and 1, respectively. I(k)[i,j] is also the pixel value of the reference pixel in the reference block at the (i,j) position. i is the horizontal position component of the pixel, and j is the vertical position component of the pixel, and the unit is also a fractional pixel unit.

[0434]

number

[0435] Also, the video decoding device 100 may apply a 5-tap vertical gradient filter (with filter coefficients of {9, -48, 0, 48, 9}; but not limited thereto) to the pixel value of the reference pixel as shown in the following Equation (43) to determine a vertical gradient value Iy(k) of the reference pixel in the reference block. Here, k may have a value of 0 or 1, and may indicate reference pictures 0 and 1, respectively. I(k)[i,j] is also the pixel value of the reference pixel in the reference block at the (i,j) position. i is the horizontal position component of the pixel, j is the vertical position component of the pixel, and the unit is also a fractional pixel unit.

[0436]

number

[0437] When the video decoding device 100 performs pixel-based motion compensation together with block extension according to the window size, when the gradient filter length is reduced and pixel-based motion compensation is performed, the memory access operations and multiplication operations corresponding to the respective memory access numbers and multiplication operation numbers are performed in the video decoding device 100 as shown in Table 7 below. In this case, it is assumed that the signal filter length T is 8, the gradient filter length T is 6, the simplified gradient filter length T is 5, the block size is NxN, and the pixel-by-pixel window size 2M+1 is 5.

[0438] [Table 7] That is, the video decoding device 100 performs two multiplication operations per one-dimensional gradient filter according to equations (41) and 42, and the gradient filter is applied to two reference blocks. The gradient filter is applied to the (N+4)x(N+4) reference block expanded based on the window size, so a total of 2*2*{(N+4)x(N+4)}*2 multiplication operations are also performed to determine the gradient values ​​of the reference pixels in the horizontal and vertical directions.

[0439] When the video decoding device 100 performs pixel-based motion compensation together with block extension according to the window size, and when the video decoding device 100 performs pixel-based motion compensation by reducing the gradient filter length without block extension, the memory access operations, multiplication operations, and multiplication operations corresponding to the respective memory access numbers and multiplication operations are performed in the video decoding device 100 as shown in Table 8 below. In this case, it is assumed that the signal filter length T is 8, the gradient filter length T is 6, the reduced gradient filter length T is 5, the pixel group size is LxL, the block size is NxN, and the pixel-based window size 2M+1 is 5.

[0440] [Table 8] Therefore, when the video decoding device 100 performs pixel group-based motion compensation by reducing the gradient filter length without block extension, the number of memory accesses, multiplication operations, and multiplication operations are much reduced compared to the number of memory accesses, multiplication operations, and multiplication operations performed when pixel-based motion compensation is performed together with block extension.

[0441] Hereinafter, a method for determining a data unit that can be used in the process of decoding an image by the video decoding device 100 according to an embodiment will be described with reference to Figures 10 to 23. The operation of the video encoding device 150 may be similar to or opposite to various embodiments related to the operation of the video decoding device 100 described below.

[0442] FIG. 10 illustrates a process in which video decoding device 100 divides a current coding unit to determine at least one coding unit, according to one embodiment.

[0443] According to an embodiment, the video decoding device 100 may determine the type of the coding unit using the block type information, and may determine the type of the coding unit to be divided using the partition type information. That is, the division method of the coding unit indicated by the partition type information may be determined depending on the block type indicated by the block type information used by the video decoding device 100.

[0444] According to an embodiment, the video decoding apparatus 100 may use block shape information indicating that the current coding unit is square. For example, the video decoding apparatus 100 may determine whether to not split the square coding unit, to split it vertically, to split it horizontally, or to split it into four coding units, etc., according to the partition shape information. Referring to FIG. 10, when the block shape information of the current coding unit 1000 indicates a square shape, the video decoding apparatus 100 may not split the coding unit 1010a having the same size as the current coding unit 1000 according to the partition shape information indicating no partition, or may determine the divided coding units 1010b, 1010c, and 1010d according to the partition shape information indicating a predetermined partition method.

[0445] 10, the video decoding apparatus 100 may determine two coding units 1010b obtained by dividing the current coding unit 1000 in the vertical direction based on the division type information indicating division in the vertical direction, according to an embodiment. The video decoding apparatus 100 may determine two coding units 1010c obtained by dividing the current coding unit 1000 in the horizontal direction based on the division type information indicating division in the horizontal direction. The video decoding apparatus 100 may determine four coding units 1010d obtained by dividing the current coding unit 1000 in the vertical and horizontal directions based on the division type information indicating division in the vertical and horizontal directions. However, the division type in which the square coding unit is divided is not limited to the above-mentioned type, and may include various types that can be indicated by the division type information. The predetermined division type in which the square coding unit is divided will be described in detail below through various embodiments.

[0446] FIG. 11 illustrates a process by which video decoding device 100 divides a coding unit having a non-square shape to determine at least one coding unit, according to one embodiment.

[0447] According to an embodiment, the video decoding apparatus 100 may use block shape information indicating that the current coding unit is non-square. The video decoding apparatus 100 may determine whether to not divide the non-square current coding unit or to divide it in a predetermined manner according to the division shape information. Referring to FIG. 11, when the block shape information of the current coding unit 1100 or 1150 indicates a non-square shape, the video decoding apparatus 100 may not divide the coding unit 1110 or 1160 having the same size as the current coding unit 1100 or 1150 according to the division shape information indicating no division, or may determine the divided coding units 1120a, 1120b, 1130a, 1130b, 1130c, 1170a, 1170b, 1180a, 1180b, and 1180c according to the division shape information indicating a predetermined division method. The predetermined division method for dividing the non-square coding unit will be described in detail below through various embodiments.

[0448] According to an embodiment, the video decoding apparatus 100 may determine a manner in which the coding unit is divided using the division type information, and in this case, the division type information may indicate the number of at least one coding unit generated by dividing the coding unit. Referring to FIG 11, when the division type information indicates that the current coding unit 1100 or 1150 is divided into two coding units, the video decoding apparatus 100 may divide the current coding unit 1100 or 1150 based on the division type information, and may determine two coding units 1120a, 11420b or 1170a, 1170b included in the current coding unit.

[0449] According to an embodiment, when the video decoding apparatus 100 divides the non-square shaped current coding unit 1100 or 1150 based on the division shape information, the video decoding apparatus 100 may divide the current coding unit by considering the position of a long side of the non-square shaped current coding unit 1100 or 1150. For example, the video decoding apparatus 100 may divide the current coding unit 1100 or 1150 in a direction dividing the long side of the current coding unit 1100 or 1150 by considering the shape of the current coding unit 1100 or 1150, to determine a plurality of coding units.

[0450] According to an embodiment, when the partitioning format information indicates that the coding unit is to be partitioned into an odd number of blocks, the video decoding apparatus 100 may determine an odd number of coding units included in the current coding unit 1100 or 1150. For example, when the partitioning format information indicates that the current coding unit 1100 or 1150 is to be partitioned into three coding units, the video decoding apparatus 100 may partition the current coding unit 1100 or 1150 into three coding units 1130a, 1130b, 1130c, 1180a, 1180b, and 1180c. According to an embodiment, the video decoding apparatus 100 may determine an odd number of coding units included in the current coding unit 1100 or 1150, and the determined coding units may not all have the same size. For example, among the determined odd number of coding units 1130a, 1130b, 1130c, 1180a, 1180b, and 1180c, a certain coding unit 1130b or 1180b may have a size different from the other coding units 1130a, 1130c, 1180a, and 1180c. That is, the coding units determined by dividing the current coding unit 1100 or 1150 may have a plurality of sizes, and in some cases, the odd number of coding units 1130a, 1130b, 1130c, 1180a, 1180b, and 1180c may have different sizes.

[0451] According to an embodiment, when the partitioning type information indicates that the coding unit is partitioned into an odd number of blocks, the video decoding apparatus 100 may determine an odd number of coding units included in the current coding unit 1100 or 1150, and may further place a predetermined restriction on at least one coding unit among the odd number of coding units generated by partitioning. Referring to FIG 11, the video decoding apparatus 100 may perform a decoding process for the coding units 1130b and 1180b located in the center among three coding units 1130a, 1130b, 1130c, 1180a, 1180b, and 1180c generated by partitioning the current coding unit 1100 or 1150 differently from the other coding units 1130a, 1130c, 1180a, and 1180c. For example, the video decoding device 100 may limit the centrally located coding units 1130b and 1180b from being further divided, unlike the other coding units 1130a, 1130c, 1180a, and 1180c, or may limit the coding units 1130b and 1180b to being divided only a predetermined number of times.

[0452] FIG. 12 illustrates a process in which the video decoding apparatus 100 divides a coding unit based on at least one of block format information and partition format information, according to an embodiment.

[0453] According to an embodiment, the video decoding apparatus 100 may determine whether or not to divide the square-shaped first coding unit 1200 into coding units based on at least one of the block shape information and the division shape information. According to an embodiment, if the division shape information indicates that the first coding unit 1200 is to be divided in the horizontal direction, the video decoding apparatus 100 may divide the first coding unit 1200 in the horizontal direction to determine the second coding unit 1210. According to an embodiment, the first coding unit, the second coding unit, and the third coding unit used are terms used to understand the division context between the coding units. For example, if the first coding unit is divided, the second coding unit is determined, and if the second coding unit is divided, the third coding unit is determined. In the following, the relationship between the first coding unit, the second coding unit, and the third coding unit used may be understood to be due to the above-mentioned characteristics.

[0454] According to an embodiment, the video decoding apparatus 100 may determine whether to divide the determined second coding unit 1210 into coding units based on at least one of block shape information and partition shape information. Referring to FIG 12, the video decoding apparatus 100 may divide the non-square second coding unit 1210 determined by dividing the first coding unit 1200 into at least one third coding unit 1220a, 1220b, 1220c, 1220d based on at least one of block shape information and partition shape information, or may not divide the second coding unit 1210. The video decoding apparatus 100 may acquire at least one of block configuration information and partition configuration information, and may divide the first coding unit 1200 based on at least one of the acquired block configuration information and partition configuration information to divide a plurality of second coding units (e.g., 1210) of various types, and the second coding unit 1210 may be divided according to the manner in which the first coding unit 1200 was divided based on at least one of the block configuration information and partition configuration information related to the first coding unit 1200. According to an embodiment, when the first coding unit 1200 is divided into the second coding units 1210 based on at least one of the block configuration information and partition configuration information related to the first coding unit 1200, the second coding unit 1210 may also be divided into third coding units (e.g., 1220a, 1220b, 1220c, 1220d) based on at least one of the block configuration information and partition configuration information related to the second coding unit 1210. That is, the coding units may be recursively divided based on at least one of the division type information and the block type information associated with each coding unit. Thus, in a non-square coding unit, a square coding unit may be determined, and the square coding unit may be recursively divided to determine a non-square coding unit. Referring to FIG. 12, in an odd number of third coding units 1220b, 1220c, and 1220d determined by dividing the non-square second coding unit 1210, a predetermined coding unit (e.g., a coding unit located in the middle or a square coding unit) may be recursively divided.According to an embodiment, the square-shaped third coding unit 1220c, which is one of the odd number of third coding units 1220b, 1220c, and 1220d, may be divided horizontally into a plurality of fourth coding units. The non-square-shaped fourth coding unit 1240, which is one of the plurality of fourth coding units, may also be divided into a plurality of coding units. For example, the non-square-shaped fourth coding unit 1240 may be further divided into odd numbers of coding units 1250a, 1250b, and 1250c.

[0455] Methods used for recursively splitting coding units will be described below in various embodiments.

[0456] According to an embodiment, the video decoding apparatus 100 may determine whether to divide each of the third coding units 1220a, 1220b, 1220c, and 1220d into coding units or not to divide the second coding unit 1210 based on at least one of the block configuration information and the partition configuration information. According to an embodiment, the video decoding apparatus 100 may divide the non-square second coding unit 1210 into an odd number of third coding units 1220b, 1220c, and 1220d. The video decoding apparatus 100 may set a predetermined restriction on a certain third coding unit among the odd number of third coding units 1220b, 1220c, and 1220d. For example, the video decoding apparatus 100 may restrict the coding unit 1220c located in the middle of the odd number of third coding units 1220b, 1220c, and 1220d so that it is not further divided or that it must be divided a configurable number of times. 12, the video decoding apparatus 100 may restrict the coding unit 1220c located in the middle of the odd number of third coding units 1220b, 1220c, and 1220d included in the non-square second coding unit 1210 to not be further divided, or to be divided in a predetermined division form (e.g., divided into only four coding units, or divided in a form corresponding to the division form of the second coding unit 1210), or to be divided only a predetermined number of times (e.g., divided only n times, n>0). However, the above restriction on the coding unit 1220c located in the middle is merely a simple embodiment, and should not be construed as being limited to the above embodiment, but should be construed as including various restrictions in which the coding unit 1220c located in the middle may be decoded differently from the different coding units 1220b and 1220d.

[0457] According to an embodiment, the video decoding apparatus 100 may obtain at least one of block configuration information and partition configuration information used to partition the current coding unit at a predetermined position within the current coding unit.

[0458] FIG. 13 illustrates a method for the video decoding apparatus 100 to determine a predetermined coding unit among an odd number of coding units according to an embodiment. Referring to FIG. 13, at least one of block shape information and partition shape information of a current coding unit 1300 may be obtained from a sample at a predetermined position (e.g., a sample 1340 located in the middle) among a plurality of samples included in the current coding unit 1300. However, the predetermined position in the current coding unit 1300 from which at least one of the block shape information and partition shape information is obtained is not limited to the center position illustrated in FIG. 13, and the predetermined position may include various positions (e.g., the top end, the bottom end, the left end, the right end, the top left end, the bottom left end, the top right end, or the bottom right end, etc.) included in the current coding unit 1300. The video decoding apparatus 100 may obtain at least one of the block shape information and partition shape information obtained from the predetermined position and determine whether to divide the current coding unit into coding units of various shapes and sizes or not to divide the current coding unit.

[0459] According to an embodiment, when a current coding unit is divided into a predetermined number of coding units, the video decoding apparatus 100 may select one of the coding units. There are various methods for selecting one of the multiple coding units, and the methods will be described below through various embodiments.

[0460] According to an embodiment, the video decoding apparatus 100 may divide a current coding unit into a plurality of coding units and determine a coding unit at a predetermined position.

[0461] FIG. 13 illustrates a method for video decoding device 100 to determine a coding unit at a given position among an odd number of coding units, according to one embodiment.

[0462] According to an embodiment, the video decoding apparatus 100 may use information indicating the positions of each of the odd-numbered coding units to determine a coding unit located in the middle of the odd-numbered coding units. Referring to FIG. 13, the video decoding apparatus 100 may divide a current coding unit 1300 to determine odd-numbered coding units 1320a, 1320b, and 1320c. The video decoding apparatus 100 may determine the middle coding unit 1320b using information related to the positions of the odd-numbered coding units 1320a, 1320b, and 1320c. For example, the video decoding apparatus 100 may determine the positions of the coding units 1320a, 1320b, and 1320c based on information indicating the positions of certain samples included in the coding units 1320a, 1320b, and 1320c to determine the coding unit 1320b located in the middle. Specifically, the video decoding device 100 can determine the positions of the coding units 1320a, 1320b, and 1320c based on information indicating the positions of the upper left samples 1330a, 1330b, and 1330c of the coding units 1320a, 1320b, and 1320c, thereby determining the coding unit 1320b located in the middle.

[0463] According to an embodiment, the information indicating the positions of the top left samples 1330a, 1330b, and 1330c included in the coding units 1320a, 1320b, and 1320c may include information related to the positions or coordinates of the coding units 1320a, 1320b, and 1320c in a picture. According to an embodiment, the information indicating the positions of the top left samples 1330a, 1330b, and 1330c included in the coding units 1320a, 1320b, and 1320c may include information indicating a width or height of the coding units 1320a, 1320b, and 1320c included in the current coding unit 1300, where such a width or height corresponds to information indicating a difference between the coordinates of the coding units 1320a, 1320b, and 1320c in a picture. That is, the video decoding device 100 can determine the coding unit 1320b located in the middle by directly using information related to the positions or coordinates of the coding units 1320a, 1320b, and 1320c within the picture, or by using information related to the width or height of the coding unit corresponding to the difference between the coordinates.

[0464] According to an embodiment, information indicating a position of the top left sample 1330a of the top coding unit 1320a may indicate (xa, ya) coordinates, information indicating a position of the top left sample 1330b of the middle coding unit 1320b may indicate (xb, yb) coordinates, and information indicating a position of the top left sample 1330c of the bottom coding unit 1320c may indicate (xc, yc) coordinates. The video decoding apparatus 100 may determine the middle coding unit 1320b using the coordinates of the top left samples 1330a, 1330b, and 1330c included in the coding units 1320a, 1320b, and 1320c, respectively. For example, when the coordinates of the upper left samples 1330a, 1330b, and 1330c are sorted in ascending or descending order, the coding unit 1320b containing the coordinates (xb, yb) of the middle sample 1330b can be determined to be the middle coding unit among the coding units 1320a, 1320b, and 1320c determined by dividing the current coding unit 1300. However, the coordinates indicating the positions of the upper left samples 1330a, 1330b, and 1330c may indicate coordinates indicating absolute positions within a picture, and furthermore, (dxb, dyb) coordinates indicating the relative position of the upper left sample 1330b of the middle coding unit 1320b based on the position of the upper left sample 1330a of the upper coding unit 1320a, and (dxc, dyc) coordinates indicating the relative position of the upper left sample 1330c of the lower coding unit 1320c may be used. Furthermore, the method of determining a coding unit at a predetermined position by using the coordinates of a sample as information indicating the position of the sample included in the coding unit should not be interpreted as being limited to the above-mentioned method, but should be interpreted as various arithmetic methods that can use the coordinates of the sample.

[0465] According to an embodiment, the video decoding apparatus 100 may divide the current coding unit 1300 into a plurality of coding units 1320a, 1320b, and 1320c, and may select a coding unit from among the coding units 1320a, 1320b, and 1320c according to a predetermined criterion. For example, the video decoding apparatus 100 may select a coding unit 1320b having a different size from the coding units 1320a, 1320b, and 1320c.

[0466] According to an embodiment, the video decoding device 100 may determine the width or height of each of the coding units 1320a, 1320b, and 1320c using (xa, ya) coordinates indicating the position of the top left sample 1330a of the top coding unit 1320a, (xb, yb) coordinates indicating the position of the top left sample 1330b of the middle coding unit 1320b, and (xc, yc) coordinates indicating the position of the top left sample 1330c of the bottom coding unit 1320c. The video decoding device 100 may determine the size of each of the coding units 1320a, 1320b, and 1320c using (xa, ya), (xb, yb), and (xc, yc) coordinates indicating the positions of the coding units 1320a, 1320b, and 1320c.

[0467] According to an embodiment, the video decoding apparatus 100 may determine the width of the top coding unit 1320a as xb-xa and the height as yb-ya. According to an embodiment, the video decoding apparatus 100 may determine the width of the middle coding unit 1320b as xc-xb and the height as yc-yb. According to an embodiment, the video decoding apparatus 100 may determine the width or height of the bottom coding unit using the width or height of the current coding unit and the widths and heights of the top coding unit 1320a and the middle coding unit 1320b. The video decoding apparatus 100 may determine a coding unit having a different size from other coding units based on the determined widths and heights of the coding units 1320a, 1320b, and 1320c. Referring to FIG. 13, the video decoding apparatus 100 may determine the middle coding unit 1320b having a different size from the top coding unit 1320a and the bottom coding unit 1320c as a coding unit at a predetermined position. However, the above-mentioned process in which the video decoding device 100 determines a coding unit having a different size from different coding units is merely one embodiment of determining a coding unit at a predetermined position using the size of the coding unit determined based on sample coordinates, and various other processes of comparing the size of the coding unit determined by a predetermined sample coordinate and determining a coding unit at a predetermined position may also be used.

[0468] However, the position of the sample considered to determine the position of the coding unit is not to be interpreted as being limited to the upper left end as mentioned above, but may be interpreted as meaning that information related to any sample position included in the coding unit may be used.

[0469] According to an embodiment, the video decoding apparatus 100 may select a coding unit at a predetermined position from among an odd number of coding units determined by dividing the current coding unit, taking into consideration the shape of the current coding unit. For example, if the current coding unit is a non-square shape in which the width is longer than the height, the video decoding apparatus 100 may determine a coding unit at a predetermined position along the horizontal direction. That is, the video decoding apparatus 100 may determine one of the coding units at different positions in the horizontal direction and set a restriction on the coding unit. If the current coding unit is a non-square shape in which the height is longer than the width, the video decoding apparatus 100 may determine a coding unit at a predetermined position along the vertical direction. That is, the video decoding apparatus 100 may determine one of the coding units at different positions in the vertical direction and set a restriction on the coding unit.

[0470] According to an embodiment, the video decoding apparatus 100 may use information indicating the positions of each of the even number of coding units to determine a coding unit at a predetermined position among the even number of coding units. The video decoding apparatus 100 may divide the current coding unit to determine the even number of coding units, and may determine the coding unit at a predetermined position using information related to the positions of the even number of coding units. A detailed process related to this process corresponds to the process of determining a coding unit at a predetermined position (e.g., a middle position) among the odd number of coding units described in FIG. 13, and therefore will not be described.

[0471] According to an embodiment, when a non-square current coding unit is divided into a plurality of coding units, in order to determine a coding unit at a predetermined position among the plurality of coding units, in the division process, the video decoding apparatus 100 may use at least one of block shape information and partition shape information stored in a sample included in a middle coding unit in the division process to determine a coding unit at a middle position among the plurality of coding units into which the current coding unit is divided.

[0472] 13, the video decoding apparatus 100 may divide a current coding unit 1300 into a plurality of coding units 1320a, 1320b, and 1320c based on at least one of block configuration information and partition configuration information, and may determine a coding unit 1320b located in the middle of the plurality of coding units 1320a, 1320b, and 1320c. Furthermore, the video decoding apparatus 100 may determine the coding unit 1320b located in the middle by considering a position where at least one of block configuration information and partition configuration information is obtained. That is, at least one of the block configuration information and partition configuration information of the current coding unit 1300 is obtained from a sample 1340 located in the middle of the current coding unit 1300, and when the current coding unit 1300 is divided into a plurality of coding units 1320a, 1320b, and 1320c based on at least one of the block configuration information and the partition configuration information, the coding unit 1320b including the sample 1340 may be determined as the coding unit located in the middle. However, the information used to determine the coding unit located in the middle is not limited to at least one of block shape information and partition shape information, and various types of information may be used in the process of determining the coding unit located in the middle.

[0473] According to an embodiment, the predetermined information for identifying the coding unit at a predetermined position is also obtained from a predetermined sample included in the coding unit to be determined. Referring to FIG 13, the video decoding apparatus 100 may use at least one of block shape information and partition shape information obtained from a sample at a predetermined position in the current coding unit 1300 (e.g., a sample located at the center of the current coding unit 1300) in order to determine a coding unit at a predetermined position (e.g., a coding unit located at the center of the multiple coding units 1320a, 1320b, and 1320c into which the current coding unit 1300 is divided). That is, the video decoding apparatus 100 may determine the sample at the predetermined position by considering the block block type of the current coding unit 1300, and may determine a coding unit 1320b including a sample from which predetermined information (e.g., at least one of block type information and division type information) is obtained among a plurality of coding units 1320a, 1320b, and 1320c determined by dividing the current coding unit 1300, and may set a predetermined restriction. Referring to FIG. 13, according to an embodiment, the video decoding apparatus 100 may determine a sample 1340 located in the middle of the current coding unit 1300 as a sample from which predetermined information is obtained, and may set a predetermined restriction in the decoding process for the coding unit 1320b including the sample 1340. However, the position of the sample from which the predetermined information is obtained is not limited to the above position, but may be interpreted as a sample at any position included in the coding unit 1320b to be determined for setting the restriction.

[0474] According to an embodiment, the position of the sample from which the predetermined information is acquired is also determined according to the shape of the current coding unit 1300. According to an embodiment, the block shape information may determine whether the shape of the current coding unit is square or non-square, and may determine the position of the sample from which the predetermined information is acquired according to the shape. For example, the video decoding apparatus 100 may determine, using at least one of information related to the width and information related to the height of the current coding unit, a sample located on a boundary that divides at least one of the width and height of the current coding unit in half as the sample from which the predetermined information is acquired. For another example, when the block shape information related to the current coding unit indicates that the current coding unit is non-square, the video decoding apparatus 100 may determine, as the sample from which the predetermined information is acquired, one of the samples adjacent to a boundary that divides the long side of the current coding unit in half.

[0475] According to an embodiment, when the video decoding apparatus 100 divides a current coding unit into a plurality of coding units, the video decoding apparatus 100 may use at least one of block shape information and partition shape information to determine a coding unit at a predetermined position among the plurality of coding units. According to an embodiment, the video decoding apparatus 100 may obtain at least one of block shape information and partition shape information from a sample at a predetermined position included in the coding unit, and the video decoding apparatus 100 may divide a plurality of coding units generated by dividing the current coding unit using at least one of partition shape information and block shape information obtained from a sample at a predetermined position included in each of the plurality of coding units. That is, the coding unit may be recursively divided using at least one of block shape information and partition shape information obtained from a sample at a predetermined position included in each of the coding units. The recursive division process of the coding unit has been described with reference to FIG. 12, and therefore a detailed description thereof will be omitted.

[0476] In one embodiment, the video decoding device 100 can divide the current coding unit and determine at least one coding unit, and can determine the order in which such at least one coding unit is decoded by a predetermined block (e.g., the current coding unit).

[0477] FIG. 14 illustrates an order in which multiple coding units are processed when video decoding device 100 divides a current coding unit and determines multiple coding units, according to one embodiment.

[0478] According to one embodiment, the video decoding device 100 can, based on the block shape information and the partition shape information, vertically divide the first coding unit 1400 and determine the second coding units 1410a and 1410b, or horizontally divide the first coding unit 1400 and determine the second coding units 1430a and 1430b, or vertically and horizontally divide the first coding unit 1400 and determine the second coding units 1450a, 1450b, 1450c, and 1450d.

[0479] Referring to FIG. 14, the video decoding apparatus 100 may determine an order in which the second coding units 1410a and 1410b determined by dividing the first coding unit 1400 in the vertical direction are processed in the horizontal direction (1410c). The video decoding apparatus 100 may determine a processing order in which the second coding units 1430a and 1430b determined by dividing the first coding unit 1400 in the horizontal direction are processed in the vertical direction (1430c). The video decoding apparatus 100 may determine an order in which the coding units located in one row are processed and then the coding units located in the next row are processed, for the second coding units 1450a, 1450b, 1450c, and 1450d determined by dividing the first coding unit 1400 in the vertical and horizontal directions, according to a predetermined order (for example, a raster scan order or a z scan order (1450e)).

[0480] According to an embodiment, the video decoding apparatus 100 may recursively divide the coding units. Referring to FIG. 14, the video decoding apparatus 100 may divide the first coding unit 1400 to determine a plurality of coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d, and may recursively divide each of the determined coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d. A method of dividing the plurality of coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d corresponds to a method of dividing the first coding unit 1400. Accordingly, the multiple coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d may be divided into multiple coding units independently. Referring to FIG 14, the video decoding apparatus 100 may divide the first coding unit 1400 vertically to determine the second coding units 1410a and 1410b, and may further determine whether or not to divide the second coding units 1410a and 1410b independently.

[0481] According to one embodiment, the video decoding apparatus 100 may horizontally split the second coding unit 1410a on the left side into third coding units 1420a and 1420b, while not splitting the second coding unit 1410b on the right side.

[0482] According to an embodiment, the processing order of the coding units is also determined based on the division process of the coding units. In other words, the processing order of the divided coding units is also determined based on the processing order of the coding units immediately before the division. The video decoding apparatus 100 may determine the processing order of the third coding units 1420a and 1420b determined by dividing the second coding unit 1410a on the left side, independently of the second coding unit 1410b on the right side. Since the second coding unit 1410a on the left side is divided horizontally and the third coding units 1420a and 1420b are determined, the third coding units 1420a and 1420b are also processed vertically (1420c). In addition, since the order in which the second coding unit 1410a on the left side and the second coding unit 1410b on the right side are processed corresponds to the horizontal direction 1410c, the third coding units 1420a and 1420b included in the second coding unit 1410a on the left side are processed in the vertical direction (1420c) before the right coding unit 1410b is processed. The above content is intended to explain a process in which the processing order of the coding units is determined based on the coding units before division, and should not be construed as being limited to the above embodiment, but should be construed as being used in various methods in which coding units determined by being divided in various forms are independently processed in a predetermined order.

[0483] FIG. 15 illustrates a process in which the video decoding device 100 determines that a current coding unit is divided into an odd number of coding units when the coding units cannot be processed in a predetermined order, according to one embodiment.

[0484] According to an embodiment, the video decoding apparatus 100 may determine that the current coding unit is divided into an odd number of coding units based on the acquired block configuration information and division configuration information. Referring to FIG. 15, a square-shaped first coding unit 1500 may be divided into non-square-shaped second coding units 1510a and 1510b, and the second coding units 1510a and 1510b may be independently divided into third coding units 1520a, 1520b, 1520c, 1520d, and 1520e. According to an embodiment, the video decoding apparatus 100 may divide the left coding unit 1510a of the second coding unit in the horizontal direction to determine a plurality of third coding units 1520a and 1520b, and may divide the right coding unit 1510b into an odd number of third coding units 1520c, 1520d, and 1520e.

[0485] According to an embodiment, the video decoding apparatus 100 may determine whether the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e are processed in a predetermined order, and may determine whether there is an odd number of coding units. Referring to FIG. 15, the video decoding apparatus 100 may recursively divide the first coding unit 1500 to determine the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e. The video decoding apparatus 100 may determine whether the first coding unit 1500, the second coding unit 1510a, 1510b, or the third coding unit 1520a, 1520b, 1520c, 1520d, and 1520e are divided into an odd number of coding units in the division type based on at least one of the block type information and the division type information. For example, the coding units located on the right side of the second coding units 1510a and 1510b may be divided into an odd number of third coding units 1520c, 1520d, and 1520e. The order in which the coding units included in the first coding unit 1500 are processed may be a predetermined order (e.g., z-scan order (1530)), and the video decoding device 100 may determine whether the third coding units 1520c, 1520d, and 1520e determined by dividing the right second coding unit 1510b into an odd number of coding units satisfy a condition for processing in the predetermined order.

[0486] According to one embodiment, the video decoding device 100 can determine whether the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e included in the first coding unit 1500 satisfy a condition to be processed in a predetermined order, the condition relating to whether at least one of the width and height of the second coding units 1510a, 1510b is divided in half by the boundary of the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e. For example, the third coding units 1520a, 1520b determined by dividing the height of the non-square left second coding unit 1510a in half satisfy the condition, but the boundaries of the third coding units 1520c, 1520d, 1520e determined by dividing the right second coding unit 1510b into three coding units do not allow the width or height of the right second coding unit 1510b to be divided in half, so it is determined that the third coding units 1520c, 1520d, 1520e do not satisfy the condition. If such a condition is not satisfied, the video decoding device 100 may determine that there is a disconnection in the scan order, and based on the determination result, may determine that the right second coding unit 1510b is to be divided into an odd number of coding units. According to one embodiment, when the video decoding device 100 is divided into an odd number of coding units, it may place a certain restriction on a coding unit at a certain position among the divided coding units, and since the content of such a restriction or the certain position has been described through various embodiments, detailed description thereof will be omitted.

[0487] FIG. 16 illustrates a process in which the video decoding apparatus 100 divides the first coding unit 1600 and determines at least one coding unit according to an embodiment. According to an embodiment, the video decoding apparatus 100 may divide the first coding unit 1600 based on at least one of the block shape information and the division shape information acquired via the acquisition unit 110. The square-shaped first coding unit 1600 may be divided into four square-shaped coding units or into a plurality of non-square-shaped coding units. For example, referring to FIG. 16, when the block shape information indicates that the first coding unit 1600 is a square and the division shape information indicates that the first coding unit 1600 is to be divided into non-square coding units, the video decoding apparatus 100 divides the first coding unit 1600 into a plurality of non-square coding units. Specifically, when the division form information indicates that the first coding unit 1600 is to be divided horizontally or vertically to determine an odd number of coding units, the video decoding device 100 can divide the square-shaped first coding unit 1600 into second coding units 1610a, 1610b, and 1610c determined by dividing it vertically, or second coding units 1620a, 1620b, and 1620c determined by dividing it horizontally, as an odd number of coding units.

[0488] According to an embodiment, the video decoding apparatus 100 may determine whether the second coding units 1610a, 1610b, 1610c, 1620a, 1620b, and 1620c included in the first coding unit 1600 satisfy a condition to be processed in a predetermined order, and the condition is related to whether at least one of the width and height of the first coding unit 1600 is divided in half by the boundaries of the second coding units 1610a, 1610b, 1610c, 1620a, 1620b, and 1620c. Referring to FIG 16, the boundaries of the second coding units 1610a, 1610b, and 1610c determined by dividing the square-shaped first coding unit 1600 in the vertical direction cannot divide the width of the first coding unit 1600 in half, so it may be determined that the first coding unit 1600 does not satisfy the condition to be processed in a predetermined order. In addition, since the boundaries of the second coding units 1620a, 1620b, and 1620c determined by dividing the square first coding unit 1600 in the horizontal direction do not divide the width of the first coding unit 1600 in half, it is determined that the first coding unit 1600 does not satisfy the condition of being processed in a predetermined order. If such a condition is not satisfied, the video decoding apparatus 100 may determine that the scan order is broken, and may determine that the first coding unit 1600 is divided into an odd number of coding units based on the determination result. According to an embodiment, when the coding unit is divided into an odd number of coding units, the video decoding apparatus 100 may set a predetermined restriction on a coding unit at a predetermined position among the divided coding units, and the content of such a restriction or the predetermined position has been described through various embodiments, and detailed description thereof will be omitted.

[0489] According to an embodiment, the video decoding apparatus 100 may divide the first coding unit and determine various types of coding units.

[0490] Referring to FIG. 16, the video decoding apparatus 100 may divide a square-shaped first coding unit 1600, a non-square-shaped first coding unit 1630, or a non-square-shaped first coding unit 1650 into various types of coding units.

[0491] FIG. 17 illustrates that, according to one embodiment, when the video decoding device 100 determines that a non-square second coding unit is determined by dividing a first coding unit 1700, the manner in which the second coding unit may be divided is restricted if the non-square second coding unit satisfies certain conditions.

[0492] According to an embodiment, the video decoding apparatus 100 may determine to divide the square-shaped first coding unit 1700 into non-square-shaped second coding units 1710a, 1710b, 1720a, and 1720b based on at least one of the block shape information and the partition shape information acquired via the acquisition unit 105. The second coding units 1710a, 1710b, 1720a, and 1720b may be divided independently. Thus, the video decoding apparatus 100 may determine whether or not to divide the first coding unit 1700 into a plurality of coding units based on at least one of the block shape information and the partition shape information related to each of the second coding units 1710a, 1710b, 1720a, and 1720b. According to an embodiment, the video decoding apparatus 100 may determine the third coding units 1712a and 1712b by dividing the left second coding unit 1710a, which is determined by dividing the first coding unit 1700 in the vertical direction, in the horizontal direction. However, when the left second coding unit 1710a is divided in the horizontal direction, the video decoding apparatus 100 may restrict the right second coding unit 1710b to not be divided in the horizontal direction in the same direction as the left second coding unit 1710a is divided. If the right second coding unit 1710b is divided in the same direction and the third coding units 1714a and 1714b are determined, the left second coding unit 1710a and the right second coding unit 1710b may be divided independently in the horizontal direction to determine the third coding units 1712a, 1712b, 1714a, and 1714b. However, this is the same result as if the video decoding device 100 had divided the first coding unit 1700 into four square-shaped second coding units 1730a, 1730b, 1730c, and 1730d based on at least one of the block shape information and the partition shape information, which is inefficient in terms of video decoding.

[0493] According to an embodiment, the video decoding apparatus 100 may vertically divide the non-square second coding unit 1720a or 1720b, which is determined by dividing the first coding unit 11300 in the horizontal direction, to determine the third coding units 1722a, 1722b, 1724a, and 1724b. However, when the video decoding apparatus 100 divides one of the second coding units (e.g., the top second coding unit 1720a) in the vertical direction, for the above-mentioned reasons, the video decoding apparatus 100 may restrict the other second coding units (e.g., the bottom coding unit 1720b) ​​from being divided vertically in the same direction as the top second coding unit 1720a.

[0494] FIG. 18 illustrates a process in which the video decoding device 100 divides a square-shaped coding unit when the division form information cannot indicate that the coding unit is divided into four square-shaped coding units, according to one embodiment.

[0495] According to an embodiment, the video decoding apparatus 100 may determine second coding units 1810a, 1810b, 1820a, and 1820b by dividing the first coding unit 1800 based on at least one of block shape information and partition shape information. The partition shape information may include information related to various shapes in which the coding unit is divided, but the information related to the various shapes may not include information for dividing the coding unit into four square-shaped coding units. According to such partition shape information, the video decoding apparatus 100 may not divide the square-shaped first coding unit 1800 into four square-shaped second coding units 1830a, 1830b, 1830c, and 1830d. ​​Based on the partition shape information, the video decoding apparatus 100 may determine non-square-shaped second coding units 1810a, 1810b, 1820a, and 1820b.

[0496] According to an embodiment, the video decoding apparatus 100 can independently divide the non-square second coding units 1810a, 1810b, 1820a, and 1820b. Through a recursive method, the second coding units 1810a, 1810b, 1820a, and 1820b are divided in a predetermined order, which is also a division method corresponding to the method in which the first coding unit 1800 is divided based on at least one of the block shape information and the partition shape information.

[0497] For example, the video decoding device 100 may divide the left second coding unit 1810a in the horizontal direction to determine square third coding units 1812a and 1812b, and may divide the right second coding unit 1810b in the horizontal direction to determine square third coding units 1814a and 1814b. Furthermore, the video decoding device 100 may divide both the left second coding unit 1810a and the right second coding unit 1810b in the horizontal direction to determine square third coding units 1816a, 1816b, 1816c, and 1816d. In this case, the coding units may be determined in the same form as when the first coding unit 1800 is divided into four square second coding units 1830a, 1830b, 1830c, and 1830d.

[0498] For another example, the video decoding device 100 may vertically divide the top second coding unit 1820a to determine square third coding units 1822a and 1822b, and may vertically divide the bottom second coding unit 1820b to determine square third coding units 1824a and 1824b. Furthermore, the video decoding device 100 may vertically divide both the top second coding unit 1820a and the bottom second coding unit 1820b to determine square third coding units 1822a, 1822b, 1824a, and 1824b. In this case, the coding units may be determined in the same form as when the first coding unit 1800 is divided into four square second coding units 1830a, 1830b, 1830c, and 1830d.

[0499] FIG. 19 illustrates that a processing order among multiple coding units may vary depending on a division process of the coding units, according to an embodiment.

[0500] According to an embodiment, the video decoding apparatus 100 may divide the first coding unit 1900 based on the block shape information and the partition shape information. When the block shape information indicates a square shape and the partition shape information indicates that the first coding unit 1900 is partitioned in at least one of the horizontal direction and the vertical direction, the video decoding apparatus 100 may divide the first coding unit 1900 to determine, for example, second coding units 1910a, 1910b, 1920a, 1920b, 1930a, 1930b, 1930c, and 1930d. Referring to FIG. 19, the non-square second coding units 1910a, 1910b, 1920a, and 1920b determined by partitioning the first coding unit 1900 only in the horizontal direction or only in the vertical direction may be independently partitioned based on the block shape information and the partition shape information associated therewith. For example, the video decoding apparatus 100 may determine third coding units 1916a, 1916b, 1916c, and 1916d by dividing second coding units 1910a and 1910b, which are generated by dividing the first coding unit 1900 in the vertical direction, horizontally, and determine third coding units 1926a, 1926b, 1926c, and 1926d by dividing second coding units 1920a and 1920b, which are generated by dividing the first coding unit 1900 in the horizontal direction. The division process of the second coding units 1910a, 1910b, 1920a, and 1920b has been described with reference to FIG. 17, and detailed description thereof will be omitted.

[0501] According to an embodiment, the video decoding device 100 may process the coding units in a predetermined order. The characteristics related to the processing of the coding units in a predetermined order have been described with reference to FIG. 14, and therefore detailed description thereof will be omitted. Referring to FIG. 19, the video decoding device 100 may divide the square-shaped first coding unit 1900 to determine four square-shaped third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d. According to an embodiment, the video decoding device 100 may determine a processing order of the third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d according to the manner in which the first coding unit 1900 is divided.

[0502] According to one embodiment, the video decoding device 100 can horizontally divide the second coding units 1910a, 1910b generated by dividing them vertically to determine the third coding units 1916a, 1916b, 1916c, and 1916d, and the video decoding device 100 can process the third coding units 1916a, 1916b, 1916c, and 1916d in an order (1917) of first vertically processing the third coding units 1916a, 1916b included in the left second coding unit 1910a, and then vertically processing the third coding units 1916c, 1916d included in the right second coding unit 1910b.

[0503] According to one embodiment, the video decoding device 100 can vertically divide the second coding units 1920a, 1920b generated by dividing them horizontally to determine the third coding units 1926a, 1926b, 1926c, and 1926d, and the video decoding device 100 can process the third coding units 1926a, 1926b, 1926c, and 1926d in an order (1927) of first horizontally processing the third coding units 1926a, 1926b included in the upper second coding unit 1920a, and then horizontally processing the third coding units 1926c, 1926d included in the lower second coding unit 1920b.

[0504] 19, the second coding units 1910a, 1910b, 1920a, and 1920b are divided, and square-shaped third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d are determined. The second coding units 1910a and 1910b determined by dividing in the vertical direction and the second coding units 1920a and 1920b determined by dividing in the horizontal direction are divided in different shapes, but the third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d determined later ultimately result in the first coding unit 1900 being divided into coding units of the same shape. As a result, the video decoding device 100 recursively divides the coding units through different processes based on at least one of the block type information and the partition type information, so that even if coding units of the same type are determined as a result, multiple coding units determined to be of the same type can be processed in different orders from each other.

[0505] FIG. 20 illustrates a process of determining the depth of a coding unit according to one embodiment, when a coding unit is recursively divided to determine multiple coding units, as the shape and size of the coding unit change.

[0506] According to an embodiment, the video decoding apparatus 100 may determine the depth of the coding unit according to a predetermined criterion. For example, the predetermined criterion may be the length of a long side of the coding unit. The video decoding apparatus 100 may determine that, when the length of the long side of the current coding unit is divided into 2n (n>0) times the length of the long side of the coding unit before division, the depth of the current coding unit is increased by n from the depth of the coding unit before division. Hereinafter, the coding unit with the increased depth will be referred to as a coding unit of a lower depth.

[0507] 20, according to an embodiment, the video decoding apparatus 100 may divide the square-shaped first coding unit 2000 based on block shape information indicating a square shape (e.g., the block shape information may indicate "0:SQUARE"), and determine a second coding unit 2002, a third coding unit 2004, etc., of a lower depth. If the size of the square-shaped first coding unit 2000 is 2Nx2N, the width and height of the first coding unit 2000 may be divided into 1 / 2. 1 The second coding unit 2002 determined by dividing the width and height of the second coding unit 2002 by 1 / 2 may have a size of NxN. Furthermore, the third coding unit 2004 determined by dividing the width and height of the second coding unit 2002 by 1 / 2 may have a size of N / 2xN / 2. In this case, the width and height of the third coding unit 2004 may be 1 / 2 that of the first coding unit 2000. 2 If the depth of the first coding unit 2000 is D, then the width and height of the first coding unit 2000 are halved. 1 The depth of the second coding unit 2002, which is D times the width and height of the first coding unit 2000, is also D+1. 2 The depth of the third coding unit 2004, which is double the depth, is also D+2.

[0508] According to one embodiment, based on block shape information indicating a non-square shape (for example, the block shape information may indicate "1:NS_VER", indicating a non-square shape in which the height is longer than the width, or "2:NS_HOR", indicating a non-square shape in which the width is longer than the height), the video decoding device 100 may divide the first coding unit 2010 or 2020, which is non-square, and determine a second coding unit 2012 or 2022 and a third coding unit 2014 or 2024 of a lower depth.

[0509] The video decoding apparatus 100 may divide at least one of the width and height of the first coding unit 2010 having a size of Nx2N to determine, for example, second coding units 2002, 2012, and 2022. That is, the video decoding apparatus 100 may divide the first coding unit 2010 in the horizontal direction to determine the second coding unit 2002 having a size of NxN or the second coding unit 2022 having a size of NxN / 2, or may divide the first coding unit 2010 in the horizontal and vertical directions to determine the second coding unit 2012 having a size of N / 2xN.

[0510] According to an embodiment, the video decoding apparatus 100 may divide at least one of the width and height of the first coding unit 2020 having a size of 2NxN to determine, for example, the second coding units 2002, 2012, and 2022. That is, the video decoding apparatus 100 may divide the first coding unit 2020 vertically to determine the second coding unit 2002 having a size of NxN or the second coding unit 2012 having a size of N / 2xN, and may divide the first coding unit 2020 horizontally and vertically to determine the second coding unit 2022 having a size of NxN / 2.

[0511] According to an embodiment, the video decoding apparatus 100 may divide at least one of the width and height of the second coding unit 2002 having a size of NxN to determine, for example, the third coding units 2004, 2014, and 2024. That is, the video decoding apparatus 100 may divide the second coding unit 2002 vertically and horizontally to determine the third coding unit 2004 having a size of N / 2xN / 2, or may divide the second coding unit 2002 vertically and horizontally to determine the third coding unit 2004 having a size of N / 2xN / 2. 2 Determine the third coding unit 2014 of size N / 2xN / 2, or 2 The size of the third coding unit 2024 can be determined.

[0512] According to an embodiment, the video decoding apparatus 100 may divide at least one of the width and height of the second coding unit 2012 having a size of N / 2xN to determine, for example, the third coding units 2004, 2014, and 2024. That is, the video decoding apparatus 100 may divide the second coding unit 2012 in the horizontal direction to determine the third coding unit 2004 having a size of N / 2xN / 2 or the third coding unit 2014 having a size of N / 2xN / 2. 2 Determine the third coding unit 2024 of size, or divide it vertically and horizontally and N / 2 2 A third coding unit 2014 of size xN / 2 may be determined.

[0513] According to an embodiment, the video decoding apparatus 100 may divide at least one of the width and height of the second coding unit 2014 having a size of NxN / 2 to determine, for example, the third coding units 2004, 2014, and 2024. That is, the video decoding apparatus 100 may divide the second coding unit 2012 vertically to determine the third coding unit 2004 having a size of N / 2xN / 2 or the third coding unit 2014 having a size of N / 2xN / 2. 2 2. Determine the third coding unit 2014 of size N / 2xN / 2, or divide it vertically and horizontally to N / 2xN / 2 2 The size of the third coding unit 2024 can be determined.

[0514] According to an embodiment, the video decoding device 100 may divide, for example, the square-shaped coding units 2000, 2002, and 2004 in the horizontal or vertical direction. For example, the first coding unit 2000 having a size of 2Nx2N may be divided vertically to determine the first coding unit 2010 having a size of Nx2N, or may be divided horizontally to determine the first coding unit 2020 having a size of 2NxN. According to an embodiment, when the depth is determined based on the length of the longest side of the coding unit, the depth of the coding unit determined by dividing the first coding unit 2000, 2002, or 2004 having a size of 2Nx2N in the horizontal or vertical direction is also the same as the depth of the first coding unit 2000, 2002, or 2004.

[0515] According to one embodiment, the width and height of the third coding unit 2014 or 2024 is half that of the first coding unit 2010 or 2020. 2 When the depth of the first coding unit 2010 or 2020 is D, the depth of the second coding unit 2012 or 2014, which is 1 / 2 the width and height of the first coding unit 2010 or 2020, is also D+1. 2 The depth of the third coding unit 2014 or 2024, which is double the depth, is also D+2.

[0516] FIG. 21 illustrates a depth determined according to the type and size of a coding unit and an index (PID: part index) for dividing the coding unit, according to an embodiment.

[0517] According to an embodiment, the video decoding apparatus 100 may divide the square-shaped first coding unit 2100 to determine various types of second coding units. Referring to FIG. 21, the video decoding apparatus 100 may divide the first coding unit 2100 in at least one of the vertical direction and the horizontal direction according to the division type information to determine the second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d. That is, the video decoding apparatus 100 may determine the second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d based on the division type information related to the first coding unit 2100.

[0518] According to an embodiment, the depths of the second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d determined by the division form information related to the square-shaped first coding unit 2100 may be determined based on the length of the long side. For example, since the length of one side of the square-shaped first coding unit 2100 is the same as the length of the long side of the non-square-shaped second coding units 2102a, 2102b, 2104a, and 2104b, the depths of the first coding unit 2100 and the non-square-shaped second coding units 2102a, 2102b, 2104a, and 2104b can be regarded as being the same as D. In contrast, when the video decoding device 100 divides the first coding unit 2100 into four square-shaped second coding units 2106a, 2106b, 2106c, and 2106d based on the division form information, the length of one side of the square-shaped second coding units 2106a, 2106b, 2106c, and 2106d is 1 / 2 times the length of one side of the first coding unit 2100, so the depth of the second coding units 2106a, 2106b, 2106c, and 2106d is also a depth of D+1, which is one depth lower than the depth D of the first coding unit 2100.

[0519] According to an embodiment, the video decoding apparatus 100 may divide a first coding unit 2110 having a height greater than its width in a horizontal direction according to the division format information, and divide the first coding unit 2110 into a plurality of second coding units 2112a, 2112b, 2114a, 2114b, and 2114c. According to an embodiment, the video decoding apparatus 100 may divide a first coding unit 2120 having a width greater than its height in a vertical direction according to the division format information, and divide the first coding unit 2120 into a plurality of second coding units 2122a, 2122b, 2124a, 2124b, and 2124c.

[0520] According to an embodiment, the depths of the second coding units 2112a, 2112b, 2114a, 2114b, 2116a, 2116b, 2116c, and 2116d determined by the division type information related to the non-square first coding unit 2110 or 2120 may be determined based on the length of the long side. For example, the length of one side of the square second coding units 2112a and 2112b is ½ times the length of one side of the non-square first coding unit 2110 whose height is longer than its width, so the depths of the square second coding units 2102a, 2102b, 2104a, and 2104b are D+1, which is one depth lower than the depth D of the non-square first coding unit 2110.

[0521] Furthermore, the video decoding device 100 can divide the non-square first coding unit 2110 into an odd number of second coding units 2114a, 2114b, and 2114c based on the division form information. The odd number of second coding units 2114a, 2114b, and 2114c may include the non-square second coding units 2114a and 2114c and the square second coding unit 2114b. In this case, the length of the long side of the non-square second coding units 2114a and 2114c and the length of one side of the square second coding unit 2114b are 1 / 2 times the length of one side of the first coding unit 2110, so the depths of the second coding units 2114a, 2114b, and 2114c are also a depth of D+1, which is one depth lower than D, which is the depth of the first coding unit 2110. The video decoding device 100 may determine the depth of a coding unit associated with a non-square first coding unit 2120 whose width is greater than its height in a manner corresponding to the above-mentioned manner for determining the depth of a coding unit associated with the first coding unit 2110.

[0522] According to an embodiment, the video decoding apparatus 100 may determine an index (PID) for partitioning the divided coding units based on a ratio of sizes between the coding units when the coding units divided into an odd number of coding units are not the same size. Referring to FIG. 21, the coding unit 2114b located in the middle of the coding units 2114a, 2114b, and 2114c divided into an odd number of coding units has the same width as the other coding units 2114a and 2114c, but is twice as tall as the other coding units 2114a and 2114c, which have different heights. That is, the coding unit 2114b located in the middle in this case may include the other coding units 2114a and 2114c. Therefore, if the index (PID) of the coding unit 2114b located in the middle according to the scanning order is 1, the coding unit 2114c located next in the order is 3, which is an index increased by 2. That is, there is a discontinuity in the index value. According to one embodiment, the video decoding device 100 can determine whether coding units divided into an odd number of parts are not the same size as each other based on the presence or absence of discontinuities in the indexes for partitions between such divided coding units.

[0523] According to an embodiment, the video decoding apparatus 100 may determine whether the current coding unit is divided into a specific division type based on the value of an index for distinguishing the plurality of coding units determined by dividing the current coding unit. Referring to FIG. 21, the video decoding apparatus 100 may divide a rectangular first coding unit 2110 having a height longer than its width to determine an even number of coding units 2112a, 2112b or an odd number of coding units 2114a, 2114b, 2114c. The video decoding apparatus 100 may use an index (PID) indicating each coding unit to distinguish each of the plurality of coding units. According to an embodiment, the PID is also obtained from a sample at a predetermined position (e.g., the upper left sample) of each coding unit.

[0524] According to an embodiment, the video decoding apparatus 100 may determine a coding unit at a predetermined position among the coding units determined by dividing the coding units using an index for dividing the coding units. According to an embodiment, when division form information related to a rectangular first coding unit 2110 having a height greater than a width indicates that the first coding unit 2110 is divided into three coding units, the video decoding apparatus 100 may divide the first coding unit 2110 into three coding units 2114a, 2114b, and 2114c. The video decoding apparatus 100 may assign an index related to each of the three coding units 2114a, 2114b, and 2114c. The video decoding apparatus 100 may compare the indexes related to each coding unit in order to determine a middle coding unit among the odd number of divided coding units. The video decoding apparatus 100 may determine the coding unit 2114b having an index corresponding to a middle value among the indexes as a coding unit at a middle position among the coding units determined by dividing the first coding unit 2110, based on the index of the coding unit. According to an embodiment, the video decoding apparatus 100 may determine an index based on a size ratio between the coding units when the coding units are not the same size in determining an index for partitioning the divided coding units. Referring to FIG. 21, a coding unit 2114b generated by dividing a first coding unit 2110 has the same width as other coding units 2114a and 2114c, but is twice as high as other coding units 2114a and 2114c, which have different heights. In this case, if the index (PID) of the coding unit 2114b located in the middle is 1, the next coding unit 2114c is 3, which is an index increased by 2. In such a case, when the indexes increase uniformly but the increase widths are different, the video decoding apparatus 100 may determine that the coding unit is divided into a plurality of coding units including a coding unit having a different size from other coding units.According to an embodiment, when the division type information indicates division into an odd number of coding units, the video decoding apparatus 100 may divide the current coding unit into a type in which a coding unit at a predetermined position (e.g., a middle coding unit) among the odd number of coding units has a different size from other coding units. In this case, the video decoding apparatus 100 may determine a middle coding unit having a different size using an index (PID) related to the coding units. However, the above index and the size or position of the coding unit at the predetermined position to be determined are specified for describing an embodiment, and should not be construed as being limited thereto, and various indexes, positions and sizes of coding units may be used.

[0525] According to one embodiment, video decoding device 100 can utilize a predetermined data unit from which the recursive division of the coding unit begins.

[0526] FIG. 22 illustrates a number of coding units being determined according to a number of predefined data units included in a picture, according to one embodiment.

[0527] According to an embodiment, the predetermined data unit is also defined as a data unit into which the coding unit starts to be recursively divided using at least one of block configuration information and partition configuration information. That is, the predetermined data unit corresponds to a coding unit of the highest depth used in the process of determining a plurality of coding units into which the current picture is divided. Hereinafter, for convenience of explanation, such a predetermined data unit is referred to as a reference data unit.

[0528] According to an embodiment, the reference data unit may have a predetermined size and shape. According to an embodiment, the reference coding unit may include MxN samples, where M and N may be the same as each other or may be integers expressed by a power of 2. That is, the reference data unit may have a square or non-square shape and may then be divided into an integer number of coding units.

[0529] According to an embodiment, the video decoding apparatus 100 may divide the current picture into a plurality of reference data units. According to an embodiment, the video decoding apparatus 100 may divide the plurality of reference data units into which the current picture is divided, using partition information related to each of the reference data units. Such a process of dividing the reference data units corresponds to a process of dividing using a quad-tree structure.

[0530] According to an embodiment, the video decoding apparatus 100 may determine in advance a minimum size that a reference data unit included in a current picture may have, and may determine reference data units of various sizes that are equal to or larger than the minimum size, and may determine at least one coding unit based on the determined reference data unit using block configuration information and partition configuration information.

[0531] 22, the video decoding apparatus 100 may use a square-shaped reference coding unit 2200 or a non-square-shaped reference coding unit 2202. According to an embodiment, the shape and size of the reference coding unit may be dete...

Claims

1. 1. A method for decoding a video, comprising: obtaining a first extended reference block in the first reference picture and a second extended reference block in the second reference picture; The first extended reference block includes a first reference block and a first portion extended from the first reference block; the second extended reference block includes a second reference block and a second portion extended from the second reference block; determining a displacement vector of a pixel group including at least one pixel adjacent to an inner boundary of a current block using a gradient value of at least one reference pixel in the first extended reference block and a gradient value of at least one reference pixel in the second extended reference block; the first portion of the first extended reference block is used to calculate a gradient value of at least one reference pixel in the first reference block; the second portion of the second extended reference block is used to calculate a gradient value of at least one reference pixel in the second reference block; performing optical flow based compensation on the current block using gradient values ​​of the at least one reference pixel in the first reference block, gradient values ​​of the at least one reference pixel in the second reference block, and a displacement vector of the pixel group to obtain a predicted pixel value of the current block; reconstructing the current block based on the predicted pixel values; The size of the pixel group is 4×4.

13. A video decoding method comprising:

2. 1. A method for encoding video, comprising the steps of: obtaining a predicted pixel value of the current block by performing optical flow based compensation on the current block using a gradient value of at least one reference pixel in a first reference block, a gradient value of at least one reference pixel in a second reference block, and a displacement vector of a pixel group of the current block; the pixel group includes at least one pixel adjacent to an inside boundary of the current block; encoding the current block based on the predicted pixel values; A first extended reference block in the first reference picture and a second extended reference block in the second reference picture are obtained; the first extended reference block includes the first reference block and a first portion extended from the first reference block; the second extended reference block includes the second reference block and a second portion extended from the second reference block; a displacement vector of the pixel group is determined using a gradient value of at least one reference pixel in the first extended reference block and a gradient value of at least one reference pixel in the second extended reference block; the first portion of the first extended reference block is used to calculate a gradient value of the at least one reference pixel in the first reference block; the second portion of the second extended reference block is used to calculate a gradient value of the at least one reference pixel in the second reference block; The size of the pixel group is 4×4.

13. A video encoding method comprising:

3. A computer-readable storage medium having a program recorded thereon for executing a method for generating a bitstream, comprising: the bitstream includes an encoding result of a current block, The method comprises: obtaining a predicted pixel value of the current block by performing optical flow based compensation on the current block using a gradient value of at least one reference pixel in a first reference block, a gradient value of at least one reference pixel in a second reference block, and a displacement vector of a pixel group of the current block; encoding the current block based on the predicted pixel values; the pixel group includes at least one pixel adjacent to an inside boundary of the current block; A first extended reference block in the first reference picture and a second extended reference block in the second reference picture are obtained; the first extended reference block includes the first reference block and a first portion extended from the first reference block; the second extended reference block includes the second reference block and a second portion extended from the second reference block; a displacement vector of the pixel group is determined using a gradient value of at least one reference pixel in the first extended reference block and a gradient value of at least one reference pixel in the second extended reference block; the first portion of the first extended reference block is used to calculate a gradient value of the at least one reference pixel in the first reference block; the second portion of the second extended reference block is used to calculate a gradient value of the at least one reference pixel in the second reference block; The size of the pixel group is 4×4. A computer-readable storage medium comprising:

Images