Video decoding method and apparatus thereof, and video encoding method and apparatus thereof

Abstract

Provided is a video decoding method for obtaining a prediction block of a current block.SOLUTION: Determining a displacement vector per unit time in a horizontal direction or a vertical direction of a current block including a pixel adjacent to an inside of a boundary of the current block by using values of reference pixels included in the first and second reference blocks without using values stored for pixels located outside a boundary of the first and second reference blocks; Performing block-based motion compensation and pixel-group-based motion compensation on the current block based on a gradient value in a horizontal direction or a vertical direction of a corresponding reference pixel in the first reference block or the second reference block corresponding to a current pixel included in a current pixel group in the current block, a pixel value of the corresponding reference pixel, and a displacement vector per unit time in the horizontal direction or the vertical direction of the current pixel to obtain a prediction block of the current block. The pixel group includes at least one pixel.SELECTED DRAWING: FIG. 1B

Description

Technical Field

[0001] The present invention relates to a video decoding method and video encoding, and more particularly, to video decoding and video encoding that perform inter prediction in a bi-directional motion prediction mode.

Background Art

[0002] With the development and popularization of hardware capable of playing and storing high-resolution or high-quality video content, the need for video coders that can effectively encode and decode high-resolution or high-quality video content has been increasing. According to existing video coders, video is encoded by a restricted encoding method based on a tree-structured encoding unit.

[0003] Using frequency conversion, video data in the spatial domain is converted into coefficients in the frequency domain. For the rapid operation of frequency conversion, a video coder divides a video into blocks of a predetermined size, performs DCT conversion for each block, and encodes the frequency coefficients in block units. The coefficients in the frequency domain have a form that is more easily compressed than the video data in the spatial domain. In particular, through inter prediction or intra prediction of a video coder, video pixel values in the spatial domain are expressed as prediction errors. Therefore, if frequency conversion is performed on the prediction errors, many data are also converted to 0. A video coder reduces the amount of data by replacing continuously repeating data with small-size data.

Summary of the Invention

Problems to be Solved by the Invention

[0004] In various embodiments, in the bidirectional motion prediction mode, not only are the pixel values ​​of the first reference block of the first reference picture and the pixel values ​​of the second reference block of the second reference picture used, but the first gradient value of the first reference block and the second gradient value of the second reference block are also used to generate the predicted pixel value of the current block. Therefore, since a predicted block similar to the original block is generated, the encoding / decoding efficiency can be increased.

[0005] The pixel values ​​of the first reference block, the pixel values ​​of the second reference block, the first gradient value of the first reference block, and the second gradient value of the second reference block are used to determine the horizontal or vertical displacement vector of the current block when performing motion compensation on a pixel group basis. In particular, to determine the horizontal or vertical displacement vector of the current pixel within the current block, not only are the pixel value and gradient value of the first reference pixel in the first reference block corresponding to the current pixel and the pixel value and gradient value of the second reference pixel in the second reference block used, but the pixel values ​​and gradient values ​​of surrounding pixels contained within a predetermined-size window centered on the first and second reference pixels are also used. Therefore, when the current pixel is located at a boundary, the surrounding pixels of the reference pixel corresponding to the current pixel are located outside the reference block, and since the pixel values ​​and gradient values ​​of pixels located outside the reference block must be referenced, there was a problem in that additional memory access was required.

[0006] According to various embodiments, the number of memory accesses can be minimized by determining the current horizontal or vertical displacement vector of the block by referring only to the pixel values ​​and gradient values ​​of pixels located inside the reference block, and not to the stored pixel values ​​and gradient values ​​of pixels located outside the reference block.

[0007] According to various embodiments, instead of using integer pixel values ​​as input and employing horizontal and vertical gradient filters and interpolation filters to determine the horizontal or vertical gradient value of a reference pixel, the process can be streamlined by applying an interpolation filter to the integer pixel values ​​to determine the pixel values ​​of pixels located at decimal pixel units, applying a horizontal or vertical gradient filter with a relatively short filter length to the pixel values ​​of pixels located at decimal pixel units, and then determining the horizontal or vertical gradient value of the reference pixel.

[0008] According to various embodiments, performing motion compensation on a pixel group basis minimizes the need for more complex multiplication operations compared to performing motion compensation on a pixel-by-pixel basis.

[0009] The method may include a computer-readable recording medium on which a program for implementing the method according to various embodiments is recorded.

[0010] Herein, the technical challenges of the various embodiments are not limited to the features mentioned above, and other technical challenges not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0011] The technical problems of the present invention are not limited to the features mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

[0012] A video decoding method according to various embodiments includes the step of obtaining motion prediction mode information relating to the current block in the current picture from a bitstream, If the acquired motion prediction mode information indicates a bi-direction motion prediction mode, the steps include acquiring from the bitstream information about a first motion vector indicating the first reference block of the current block within the first reference picture, and information about a second motion vector indicating the second reference block of the current block within the second reference picture. Without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks, the steps include determining the horizontal or vertical displacement vector per unit time of pixels in the current block, including pixels adjacent to the inside of the current block's boundaries, using values ​​relating to reference pixels included in the first and second reference blocks, Based on the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel, a step is to perform block-level motion compensation and pixel group-level motion compensation for the current block to obtain a predicted block for the current block. The process involves obtaining information about the current block's residual block from the aforementioned bitstream, The step includes restoring the current block based on the predicted block and the residual block, The aforementioned pixel group may include at least one pixel.

[0013] In video decoding methods according to various embodiments, the step of obtaining the predicted block of the current block is: The process further includes the step of applying a horizontal or vertical gradient filter to the pixel value of a decimal-position pixel within the first or second reference block to calculate the horizontal or vertical gradient value of the first corresponding reference pixel or the second corresponding reference pixel. The gradient filter is a 5-tap filter, The aforementioned decimal position pixel is also a pixel in which at least one component of the horizontal or vertical component of the coordinates indicating the pixel's position has a decimal value.

[0014] In various embodiments of video decoding methods, the pixel values ​​of decimal-position pixels within the first or second reference block can also be calculated by applying a horizontal or vertical interpolation filter to the pixel values ​​of integer-position pixels.

[0015] In video decoding methods according to various embodiments, the size of the pixel group is also determined based on the minimum value of the height and width of the current block.

[0016] In a video decoding method according to various embodiments, the displacement vector per unit time in the horizontal or vertical direction for the current pixel group is also a displacement vector per unit time determined using a value determined based on the pixel value gradient value between the first corresponding reference pixel and its surrounding pixels included in the first corresponding reference pixel group in the first reference picture corresponding to the current pixel group, and the second corresponding reference pixel and its surrounding pixels included in the second corresponding reference pixel group in the second reference picture, the first POC (picture order count) difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture.

[0017] In a video decoding method according to various embodiments, the step of determining the horizontal or vertical displacement vector per unit time of the pixels of the current block is: If the first corresponding reference pixel or the second corresponding reference pixel is a boundary-peripheral pixel adjacent to the inside boundary of the first reference block or the second reference block, The steps include: using the pixel values ​​of the pixels surrounding the boundary to derive the pixel values ​​and gradient values ​​of pixels located outside the boundary of the first or second reference block; The process may include the step of determining a horizontal or vertical displacement vector per unit time of the current pixel based on the pixel values ​​and gradient values ​​of the boundary-peripheral pixels, and the pixel values ​​and gradient values ​​of pixels located outside the boundary of the current block, which are derived using the pixel values ​​of the boundary-peripheral pixels.

[0018] In a video decoding method according to various embodiments, the step of determining the horizontal or vertical displacement vector per unit time of the pixels of the current block is: A step of calculating a value related to the current pixel using the pixel value of the first corresponding reference pixel contained in the first reference block, the pixel value of the second corresponding reference pixel contained in the second reference block, the gradient value of the first corresponding reference pixel, the gradient value of the second corresponding reference pixel, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture. A step of calculating a value related to the surrounding pixels calculated using the pixel value of the first corresponding surrounding pixel of the first corresponding reference pixel, the pixel value of the second corresponding surrounding pixel of the second corresponding reference pixel, the gradient value of the first corresponding surrounding pixel, the gradient value of the second corresponding surrounding pixel, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture. A step of calculating a weighted average value for the current pixel, which is necessary to calculate a displacement vector per unit time in the horizontal or vertical direction, using the value relating to the current pixel, the value relating to the surrounding pixels, and the weight value. The process may also include the step of determining the horizontal or vertical displacement vector per unit time of the current pixel using the weighted average value of the current pixel calculated above.

[0019] In the video decoding method according to various embodiments, the weighted average value for the current pixel is also a value calculated by applying an exponential smoothing technique in the vertical, horizontal, and vertical directions to the values ​​relating to the pixels included in the first and second reference blocks.

[0020] A video decoding device in various embodiments includes the step of obtaining motion prediction mode information relating to the current block in the current picture from a bitstream, If the acquired motion prediction mode information indicates a bidirectional motion prediction mode, the acquisition unit acquires from the bitstream information about a first motion vector indicating the first reference block of the current block in the first reference picture, and information about a second motion vector indicating the second reference block of the current block in the second reference picture, and acquires from the bitstream information about the current block's residual block. Without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks, an interpretation unit determines the horizontal or vertical displacement vector per unit time of pixels in the current block, including pixels adjacent to the inside of the current block's boundary, using values ​​related to reference pixels included in the first and second reference blocks, and performs block-level motion compensation and pixel group-level motion compensation related to the current block, thereby obtaining a predicted block of the current block, based on the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel. A decoding unit that restores the current block based on the predicted block and the residual block, The aforementioned pixel group may include at least one pixel.

[0021] In video decoding devices according to various embodiments, the interpretation unit is: A horizontal or vertical gradient filter is applied to the pixel value of a decimal-position pixel within the first or second reference block to calculate the horizontal or vertical gradient value of the first or second corresponding reference pixel. The gradient filter is a 5-tap filter, The aforementioned decimal position pixel is also a pixel in which at least one component of the horizontal or vertical component of the coordinates indicating the pixel's position has a decimal value.

[0022] In video decoding devices according to various embodiments, the interpretation unit is: The horizontal or vertical displacement vector per unit time for the current pixel group is also a displacement vector per unit time determined using values ​​determined based on the pixel value gradient value between the first corresponding reference pixel and its surrounding pixels included in the first corresponding reference pixel group in the first reference picture corresponding to the current pixel group, the second corresponding reference pixel and its surrounding pixels included in the second corresponding reference pixel group in the second reference picture, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture.

[0023] In video decoding devices according to various embodiments, the interpretation unit is: Using the pixel value of the first corresponding reference pixel contained in the first reference block, the pixel value of the second corresponding reference pixel contained in the second reference block, the gradient value of the first corresponding reference pixel, the gradient value of the second corresponding reference pixel, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture, the value related to the current pixel is calculated. Values ​​related to surrounding pixels are calculated using the pixel values ​​of the first corresponding surrounding pixels of the first corresponding reference pixel, the pixel values ​​of the second corresponding surrounding pixels of the second corresponding reference pixel, the gradient value of the first corresponding surrounding pixels, the gradient value of the second corresponding surrounding pixels, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture. Using the value related to the current pixel, the value related to the surrounding pixels, and the weighted value, a weighted average value for the current pixel is calculated, which is necessary to calculate the displacement vector per unit time in the horizontal or vertical direction. Using the weighted average value calculated for the current pixel, the horizontal or vertical displacement vector per unit time of the current pixel can be determined.

[0024] A video encoding method according to various embodiments includes the steps of performing block-level motion compensation and pixel group-level motion compensation on the current block to obtain a predicted block, a first motion vector, and a second motion vector for the current block, The process includes the step of generating a bitstream that includes information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether or not the motion prediction mode relating to the current block is a bidirectional motion prediction mode, The aforementioned pixel group includes at least one pixel, The first motion vector is a motion vector that points from the current block to the first reference block of the first reference picture corresponding to the current block in the current picture, The second motion vector is a motion vector that points from the current block to the second reference block of the second reference picture corresponding to the current block in the current picture, Based on the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel, block-level motion compensation and pixel group-level motion compensation related to the current block are performed, and a predicted block of the current block is obtained. The horizontal or vertical displacement vectors per unit time of pixels in the current block, including pixels adjacent to the boundary of the current block, can be determined using values ​​relating to reference pixels contained within the first and second reference blocks, without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks.

[0025] A video encoding apparatus according to various embodiments includes an interpretation unit that performs block-level motion compensation and pixel group-level motion compensation on the current block, and obtains a predicted block, a first motion vector, and a second motion vector for the current block. The system includes a bitstream generation unit that generates a bitstream containing information about the first motion vector and the second motion vector, as well as motion prediction mode information indicating whether or not the motion prediction mode for the current block is a bidirectional motion prediction mode, The aforementioned pixel group includes at least one pixel, The first motion vector is a motion vector that points from the current block to the first reference block of the first reference picture corresponding to the current block in the current picture, and the second motion vector is a motion vector that points from the current block to the second reference block of the second reference picture corresponding to the current block in the current picture. Based on the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel, block-level motion compensation and pixel group-level motion compensation related to the current block are performed, and a predicted block of the current block is obtained. The horizontal or vertical displacement vectors per unit time of pixels in the current block, including pixels adjacent to the boundary of the current block, can be determined using values ​​relating to reference pixels contained within the first and second reference blocks, without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks. The method may include a computer-readable recording medium on which a program for implementing the method according to various embodiments is recorded. [Effects of the Invention]

[0026] Through various embodiments, in the bidirectional motion prediction mode, the efficiency of encoding / decoding can be improved by utilizing the gradient value of the reference block of the reference picture, performing interpretation related to the current block, and predicting a value similar to the original block value of the current block. [Brief explanation of the drawing]

[0027] [Figure 1A] This is a block diagram of a video decoding device according to various embodiments. [Figure 1B] This is a flowchart of video decoding methods according to various embodiments. [Figure 1C] This is a block diagram of a video encoding device according to various embodiments. [Figure 1D] This is a flowchart of video encoding methods according to various embodiments. [Figure 1E] This is a block diagram of a video decoding unit in various embodiments. [Figure 1F] This is a block diagram of a video decoding unit in various embodiments. [Figure 2] This is a reference diagram illustrating the process of predicting and compensating for bidirectional motion of a block substrate according to one embodiment. [Figure 3A] This is a reference diagram illustrating the process of performing pixel-level motion compensation according to one embodiment. [Figure 3B] This is a reference diagram illustrating the process of performing pixel-level motion compensation according to one embodiment. [Figure 3C] This is a reference diagram illustrating the process of performing pixel-level motion compensation according to one embodiment. [Figure 4] This is a reference diagram illustrating the process of calculating horizontal and vertical gradient values ​​according to one embodiment. [Figure 5] Another embodiment is shown as a reference diagram illustrating the process of calculating horizontal and vertical gradient values. [Figure 6A] This diagram illustrates the process of determining horizontal and vertical gradient values ​​using a one-dimensional filter, according to one embodiment. [Figure 6B] This diagram illustrates the process of determining horizontal and vertical gradient values ​​using a one-dimensional filter, according to one embodiment. [Figure 7A] This table shows the filter coefficients of a filter used to determine the pixel value at decimal pixel positions in decimal pixel units, as well as the horizontal and vertical gradient values, according to one embodiment. [Figure 7B] This table shows the filter coefficients of a filter used to determine the pixel value at decimal pixel positions in decimal pixel units, as well as the horizontal and vertical gradient values, according to one embodiment. [Figure 7C] This table shows the filter coefficients of a filter used to determine the pixel value at decimal pixel positions in decimal pixel units, as well as the horizontal and vertical gradient values, according to one embodiment. [Figure 7D] This table shows the filter coefficients of a filter used to determine the pixel value at decimal pixel positions in decimal pixel units, as well as the horizontal and vertical gradient values, according to one embodiment. [Figure 7E] This table shows the filter coefficients of a filter used to determine the pixel value at decimal pixel positions in decimal pixel units, as well as the horizontal and vertical gradient values, according to one embodiment. [Figure 8A] This is a reference diagram illustrating the process of determining the horizontal and vertical displacement vectors for a pixel according to one embodiment. [Figure 8B] This is a reference diagram illustrating the process of determining the horizontal and vertical displacement vectors related to a pixel group according to one embodiment. [Figure 8C]This is a reference diagram illustrating the process of determining the horizontal and vertical displacement vectors related to a pixel according to one embodiment. [Figure 8D] This is a reference diagram illustrating the process of determining the horizontal and vertical displacement vectors related to a pixel group according to one embodiment. [Figure 9A] This diagram illustrates a process for determining horizontal or vertical gradient values ​​by applying an offset and performing inverse scaling after filtering, according to one embodiment. [Figure 9B] This diagram illustrates a process for determining horizontal or vertical gradient values ​​by adding an offset and performing inverse scaling after filtering, according to another embodiment. [Figure 9C] This diagram illustrates the input range required to determine the horizontal and vertical displacement vectors during the process of performing pixel-level motion compensation on a block. [Figure 9D] This diagram illustrates the input range of the region used in the process of performing pixel-level motion compensation using various embodiments. [Figure 9E] This diagram illustrates the input range of the region used in the process of performing pixel-level motion compensation using various embodiments. [Figure 9F] This diagram illustrates the process for determining the horizontal and vertical displacement vectors without extending the reference block. [Figure 9G] This is a diagram illustrating a process for determining the horizontal and vertical displacement vectors without extending the reference block, according to another embodiment. [Figure 9H] This is a diagram illustrating a process for determining the horizontal and vertical displacement vectors without extending the reference block, according to another embodiment. [Figure 9I]This is a diagram illustrating a process for determining the horizontal and vertical displacement vectors without extending the reference block, according to another embodiment. [Figure 9J] This diagram illustrates a process for determining pixel-specific horizontal and vertical displacement vectors by referencing the pixel and gradient values ​​of a reference block and applying an exponential smoothing technique to the up, down, left, and right directions, without expanding the block, according to one embodiment. [Figure 9K] This diagram illustrates a process, according to one embodiment, for determining the pixel values ​​of reference pixels within a reference block, as well as the horizontal and vertical gradient values, by using a filter to perform motion compensation related to the current block. [Figure 9L] This diagram illustrates a process for determining the pixel values ​​of reference pixels within a reference block, as well as the horizontal and vertical gradient values, using a filter to perform motion compensation related to the current block, according to another embodiment. [Figure 10] This diagram illustrates the process by which an encoding unit is currently divided and at least one encoding unit is determined, according to one embodiment. [Figure 11] This diagram illustrates the process by which a non-square coding unit is divided and at least one coding unit is determined, according to one embodiment. [Figure 12] This diagram illustrates the process by which an encoded unit is divided based on at least one of block shape information and division shape information, according to one embodiment. [Figure 13] This diagram illustrates a method for determining a predetermined coding unit from an odd number of coding units, according to one embodiment. [Figure 14] This diagram illustrates the order in which multiple coding units are processed when, according to one embodiment, a coding unit is currently divided and multiple coding units are determined. [Figure 15]This diagram illustrates, in one embodiment, the process by which, when an encoding unit cannot be processed in a predetermined order, the current encoding unit is determined to be divided into an odd number of encoding units. [Figure 16] This diagram illustrates the process by which a first coding unit is divided and at least one coding unit is determined, according to one embodiment. [Figure 17] This diagram illustrates, in one embodiment, that when a non-square second coding unit determined by dividing a first coding unit satisfies predetermined conditions, the possible forms in which the second coding unit can be divided are limited. [Figure 18] In cases where one embodiment cannot show that the division information is divided into four square-shaped coding units, this diagram illustrates the process by which the square-shaped coding units are divided. [Figure 19] This diagram illustrates, in one embodiment, that the processing order between multiple coding units may differ depending on the coding unit division process. [Figure 20] This diagram illustrates the process by which the depth of a coding unit is determined when, according to one embodiment, a coding unit is recursively divided to determine multiple coding units, and the shape and size of the coding unit change. [Figure 21] This diagram illustrates, according to one embodiment, the depth determined by the shape and size of the coding unit, and an index (PID: part index) for classifying coding units. [Figure 22] This diagram illustrates, in one embodiment, that multiple encoding units are determined by multiple predetermined data units contained in a picture. [Figure 23] This diagram illustrates a processing block that serves as a criterion for determining the order in which reference coding units included in a picture are determined, according to one embodiment. [Modes for carrying out the invention]

[0028] A video decoding method according to various embodiments includes the step of obtaining motion prediction mode information relating to the current block in the current picture from a bitstream, If the acquired motion prediction mode information indicates a bi-direction motion prediction mode, the steps include acquiring from the bitstream information about a first motion vector indicating the first reference block of the current block within the first reference picture, and information about a second motion vector indicating the second reference block of the current block within the second reference picture. Without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks, the steps include determining the horizontal or vertical displacement vector per unit time of pixels in the current block, including pixels adjacent to the inside of the current block's boundaries, using values ​​relating to reference pixels included in the first and second reference blocks, Based on the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel, a step is to perform block-level motion compensation and pixel group-level motion compensation for the current block to obtain a predicted block for the current block. The process involves obtaining information about the current block's residual block from the aforementioned bitstream, The step includes restoring the current block based on the predicted block and the residual block, The aforementioned pixel group may include at least one pixel.

[0029] A video decoding device in various embodiments includes the step of obtaining motion prediction mode information relating to the current block in the current picture from a bitstream, If the acquired motion prediction mode information indicates a bidirectional motion prediction mode, the acquisition unit acquires from the bitstream information about a first motion vector indicating the first reference block of the current block in the first reference picture, and information about a second motion vector indicating the second reference block of the current block in the second reference picture, and acquires from the bitstream information about the current block's residual block. Without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks, an interpretation unit determines the horizontal or vertical displacement vector per unit time of pixels in the current block, including pixels adjacent to the inside of the current block's boundary, using values ​​related to reference pixels included in the first and second reference blocks, and performs block-level motion compensation and pixel group-level motion compensation related to the current block, thereby obtaining a predicted block of the current block, based on the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel. A decoding unit that restores the current block based on the predicted block and the residual block, The aforementioned pixel group may include at least one pixel.

[0030] A video encoding method according to various embodiments includes the steps of performing block-level motion compensation and pixel group-level motion compensation on the current block to obtain a predicted block, a first motion vector, and a second motion vector for the current block, The process includes the step of generating a bitstream that includes information about the first motion vector and the second motion vector, and motion prediction mode information indicating whether or not the motion prediction mode relating to the current block is a bidirectional motion prediction mode, The aforementioned pixel group includes at least one pixel, The first motion vector is a motion vector that points from the current block to the first reference block of the first reference picture corresponding to the current block in the current picture, The second motion vector is a motion vector that points from the current block to the second reference block of the second reference picture corresponding to the current block in the current picture, Based on the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel, block-level motion compensation and pixel group-level motion compensation related to the current block are performed, and a predicted block of the current block is obtained. The horizontal or vertical displacement vectors per unit time of pixels in the current block, including pixels adjacent to the boundary of the current block, can be determined using values ​​relating to reference pixels contained within the first and second reference blocks, without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks.

[0031] A video encoding apparatus according to various embodiments includes an interpretation unit that performs block-level motion compensation and pixel group-level motion compensation on the current block, and obtains a predicted block, a first motion vector, and a second motion vector for the current block. The system includes a bitstream generation unit that generates a bitstream containing information about the first motion vector and the second motion vector, as well as motion prediction mode information indicating whether or not the motion prediction mode for the current block is a bidirectional motion prediction mode, The aforementioned pixel group includes at least one pixel, The first motion vector is a motion vector that points from the current block to the first reference block of the first reference picture corresponding to the current block in the current picture, and the second motion vector is a motion vector that points from the current block to the second reference block of the second reference picture corresponding to the current block in the current picture. Based on the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel, block-level motion compensation and pixel group-level motion compensation related to the current block are performed, and a predicted block of the current block is obtained. The horizontal or vertical displacement vectors per unit time of pixels in the current block, including pixels adjacent to the boundary of the current block, can be determined using values ​​relating to reference pixels contained within the first and second reference blocks, without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks.

[0032] The method may include a computer-readable recording medium on which a program for implementing the method according to various embodiments is recorded.

[0033] Hereafter, "video" can refer to still images or moving images from a video, that is, the video itself.

[0034] Hereafter, "sample" refers to data assigned to the sampling position of the video, and thus data that will be processed. For example, in a video in the spatial domain, a pixel is also a sample.

[0035] Hereafter, "current block" refers to the block of video that is being encoded or decoded.

[0036] Figure 1A illustrates block diagrams of video decoding devices according to various embodiments.

[0037] The video decoding device 100, according to various embodiments, includes an acquisition unit 105, an interpretation unit 110, and a reconstruction unit 125.

[0038] The acquisition unit 105 receives a bitstream containing information about the current block's prediction mode, information indicating the current block's motion prediction mode, and information about the motion vector.

[0039] The acquisition unit 105 can acquire information about the prediction mode of the current block, information indicating the motion prediction mode of the current block, and information about the motion vector from the received bitstream. The acquisition unit 105 can also acquire a reference picture index from the received bitstream that indicates a reference picture among previously decoded pictures.

[0040] The inter-prediction unit 110 performs inter-prediction for the current block if the prediction mode of the current block is inter-prediction mode. That is, the inter-prediction unit 110 can generate predicted pixel values ​​for the current block using at least one of the pictures decoded prior to the current picture containing the current block. For example, if the motion prediction mode of the current block is bidirectional motion prediction mode, the inter-prediction unit 110 can generate predicted pixel values ​​for the current block using two pictures decoded prior to the current picture. That is, if the information about the motion prediction mode obtained from the bitstream indicates bidirectional motion prediction mode, the inter-prediction unit 110 can generate predicted pixel values ​​for the current block using two pictures decoded prior to the current picture.

[0041] The interpretation unit 110 may include a block-level motion compensation unit 115 and a pixel group-level motion compensation unit 120.

[0042] The block-level motion compensation unit 115 can currently perform block-level motion compensation related to the block.

[0043] The block-level motion compensation unit 115 can determine at least one reference picture from previously decoded pictures using the reference picture index obtained from the bitstream. In this case, the reference picture index refers to the reference picture index for each prediction direction, including the L0 direction and the L1 direction. Here, the reference picture index for the L0 direction refers to the index that indicates the reference picture among the pictures included in the L0 reference picture list, and the reference picture index for the L1 direction refers to the index that indicates the reference picture among the pictures included in the L1 reference picture list.

[0044] The block-level motion compensation unit 115 can use information about motion vectors received from the bitstream to determine the reference block of the current block located within at least one reference picture. Here, the corresponding block in the reference picture that corresponds to the current block in the current picture is also the reference block. That is, the block-level motion compensation unit 115 can use the motion vector indicating the reference block from the current block to determine the reference block of the current block. Here, the motion vector means a vector indicating the displacement between the reference coordinates of the current block in the current picture and the reference coordinates of the reference block in the reference picture. For example, if the upper-left coordinate of the current block is (1,1) and the upper-left coordinate of the reference block in the reference picture is (3,3), then the motion vector is also (2,2).

[0045] Here, the information about the motion vector may include the difference value of the motion vector. The block-level motion compensation unit 115 uses the motion vector predictor and the difference value of the motion vector obtained from the bitstream to reconstruct the motion vector, and uses the reconstructed motion vector to determine the reference block of the current block located in at least one reference picture. At this time, the difference value of the motion vector means the difference value of the motion vector related to the reference picture related to each of the prediction directions, including the L0 direction and the L1 direction. Here, the difference value of the motion vector related to the L0 direction means the difference value of the motion vector indicating the reference block in the reference picture included in the L0 reference picture list, and the difference value of the motion vector related to the L1 direction means the difference value of the motion vector indicating the reference block in the reference picture included in the L1 reference picture list.

[0046] The block-level motion compensation unit 115 can perform block-level motion compensation related to the current block by utilizing the pixel value of the reference block. The block-level motion compensation unit 115 can perform block-level motion compensation related to the current block by utilizing the reference pixel value in the reference block that corresponds to the current pixel in the current block. Here, the reference pixel is a pixel included in the reference block, and the corresponding pixel that corresponds to the current pixel in the current block is also the reference pixel.

[0047] The block-level motion compensation unit 115 can perform block-level motion compensation for the current block by utilizing the multiple reference blocks contained in each of the multiple reference pictures. For example, if the motion prediction mode of the current block is the bidirectional motion prediction mode, the block-level motion compensation unit 115 can determine two reference pictures from among the previously encoded pictures and determine two reference blocks contained in the two pictures.

[0048] The block-level motion compensation unit 115 can perform block-level motion compensation for the current block by utilizing the pixel values ​​of two reference pixels within two reference blocks. The block-level motion compensation unit 115 can perform block-level motion compensation for the current block by utilizing the average value or weighted sum of the pixel values ​​of the two reference pixels, and generate a block-level motion compensation value.

[0049] The reference position of a reference block is, but not limited to, an integer pixel position, and can also be a fractional pixel position. Here, an integer pixel is a pixel whose positional component is an integer, meaning a pixel located at an integer pixel position. A fractional pixel is a pixel whose positional component is a decimal, meaning a pixel located at a fractional pixel position.

[0050] For example, if the current top-left coordinate of a block is (1,1) and its motion vector is (2.5,2.5), then the top-left coordinate of the referenced block within the referenced picture is also (3.5,3.5). In this case, the position of a fractional pixel can also be determined in units of 1 / 4 pel (pixel element) or 1 / 16 pel. However, it is not limited to these units, and the position of a fractional pixel can also be determined by a variety of fractional pel units.

[0051] If the reference position of the reference block is a fractional pixel position, the block-level motion compensation unit 115 can apply an interpolation filter to the first peripheral region including the first pixel among the pixels of the first reference block indicated by the first motion vector, and to the second peripheral region including the second pixel among the pixels of the second reference block indicated by the second motion vector, thereby generating the pixel value of the first pixel and the pixel value of the second pixel.

[0052] In other words, the reference pixel value within the reference block can also be determined using the pixel values ​​of surrounding pixels whose predetermined directional component is an integer. In this case, the predetermined direction can be either horizontal or vertical.

[0053] For example, the block-unit motion compensation unit 115 can filter the pixel values ​​of pixels whose predetermined directional components are integers using an interpolation filter, determine a reference pixel value with the resulting value, and use this reference pixel value to determine the block-unit motion compensation value for the current block. The block-unit motion compensation value can also be determined using the average value or weighted sum of the reference pixels. In this case, the interpolation filter can be an M-tap interpolation filter based on DCT (discrete cosine transformation). The coefficients of the M-tap interpolation filter based on DCT are also derived from DCT and IDCT (inverse discrete cosine transform). In this case, the coefficients of the interpolation filter are also filter coefficients scaled to integer coefficients in order to reduce real number calculations during filtering. In this case, the interpolation filter is also a one-dimensional interpolation filter in the horizontal or vertical direction. For example, when representing the position of a pixel using x,y orthogonal coordinate components, the horizontal direction means the direction parallel to the x-axis. The vertical direction means the direction parallel to the y-axis.

[0054] The block-based motion compensation unit 115 first uses a vertical one-dimensional interpolation filter to filter pixel values ​​at integer positions, and then uses a horizontal one-dimensional interpolation filter to filter the resulting values ​​to determine reference pixel values ​​at decimal pixel positions.

[0055] On the other hand, when using scaled filter coefficients, the values ​​generated after filtering are larger than those when using unscaled filter coefficients. Therefore, the block-level motion compensation unit 115 can de-scale the values ​​generated after filtering.

[0056] The block-based motion compensation unit 115 can perform inverse scaling after filtering pixel values ​​at integer positions using a vertical one-dimensional interpolation filter. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits can also be determined based on the bit depth of the input video samples. For example, the number of inverse scaling bits is also the bit depth of the samples minus 8.

[0057] Furthermore, the block-level motion compensation unit 115 can use a vertical one-dimensional interpolation filter to filter pixel values ​​at integer positions, and then use a horizontal one-dimensional interpolation filter to filter the resulting values ​​before performing inverse scaling. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits can also be determined based on the scaling bits of the vertical one-dimensional interpolation filter, the scaling bits of the horizontal one-dimensional interpolation filter, and the bit depth of the sample. For example, if the scaling bits p of the vertical one-dimensional interpolation filter are 6, the scaling bits q of the horizontal one-dimensional interpolation filter are 6, and the bit depth of the sample is b, then the number of inverse scaling bits is p + q + 8 - b, which is also 20 - b.

[0058] If the block-level motion compensation unit 115 uses a one-dimensional interpolation filter to filter pixels whose predetermined directional components are integers, and then simply performs bit shifting to the right by the number of inverse scaling bits, a rounding error may occur. Therefore, after using a one-dimensional interpolation filter to filter pixels whose predetermined directional components are integers, an offset is added, and then inverse scaling is performed. In this case, the offset is also 2^(number of inverse scaling bits - 1).

[0059] The pixel group motion compensation unit 120 can perform pixel group motion compensation for the current block and generate a pixel group motion compensation value. When the motion prediction mode of the current block is the bidirectional motion prediction mode, the pixel group motion compensation unit 120 can perform pixel group motion compensation for the current block and generate a pixel group motion compensation value.

[0060] The pixel group motion compensation unit 120 can perform pixel group motion compensation for the current block based on the optical flow of the pixel groups of the first and second reference pictures, and generate a pixel group motion compensation value. The optical flow will be described later in the explanation related to Figure 3A.

[0061] The pixel group motion compensation unit 120 can perform pixel group motion compensation on a pixel group basis for pixel groups contained in the reference block of the current block, and generate a pixel group motion compensation value. The pixel group may contain at least one pixel. For example, the pixel group may be a single pixel. Alternatively, the pixel group may be multiple pixels containing two or more pixels. The pixel group may also be multiple pixels contained in a block of size KxK (where K is an integer).

[0062] The pixel group motion compensation unit 120 can determine a pixel group and, based on the determined pixel group, perform motion compensation on a pixel group basis for the current block.

[0063] The pixel group motion compensation unit 120 can determine the size of a pixel group based on the current size of a block. For example, the pixel group motion compensation unit 120 can determine the height and width of a pixel group as the maximum value of 2 obtained by dividing the minimum value of the current block's height and width by 8.

[0064] The pixel group motion compensation unit 120 performs motion compensation in units of pixel groups containing multiple pixels, thereby reducing the complexity of encoding / decoding compared to performing motion compensation on a pixel-by-pixel basis from a high video resolution.

[0065] The acquisition unit 105 can acquire information about the size of the pixel groups contained in the bitstream. This information about the size of the pixel groups also indicates the height or width K if the size of the pixel groups is KxK. This information about the size of the pixel groups is also included in the high-level syntax carrier.

[0066] The pixel group motion compensation unit 120 can determine at least one pixel group partition containing pixels with similar pixel values ​​from among multiple pixels included in a pixel group, and perform motion compensation on the pixel group partition. In this case, since pixel group partitions containing pixels with similar pixel values ​​are highly likely to be the same object and therefore likely to have similar movements, the pixel group motion compensation unit 120 can perform even more detailed pixel group motion compensation.

[0067] On the other hand, motion compensation at the pixel group level is performed when the motion prediction mode information indicates a bidirectional motion prediction mode, but even in that case, it is not always performed, and can also be performed selectively.

[0068] The pixel group-based motion compensation unit 120 can determine the reference pixel group in the reference block corresponding to the current pixel group of the current block, and determine the gradient value of the reference pixel group. For example, the pixel group-based compensation unit 120 can determine the gradient value of the reference pixel group by utilizing the gradient value of at least one pixel value included in the reference pixel group.

[0069] The pixel group motion compensation unit 120 can use the gradient value of a reference pixel group to perform motion compensation on a pixel group basis related to the current block and generate a pixel group motion compensation value.

[0070] The pixel group-based motion compensation unit 120 can apply a filter to the first peripheral region of the first pixel group, which includes the first pixel group, among the pixel groups of the first reference block indicated by the first motion vector, and to the second peripheral region of the second pixel group, which includes the second pixel group, among the pixel groups of the second reference block indicated by the second motion vector, thereby generating gradient values ​​for the first pixel group and the second pixel group.

[0071] The pixel group motion compensation unit 120 can determine the pixel values ​​and pixel gradient values ​​of pixels within a predetermined size first window including the first pixel group, centered on the first pixel group in the first reference picture, and determine the pixel values ​​and pixel gradient values ​​of pixels within a predetermined size second window including the second reference pixels, centered on the second reference pixel group in the second reference picture.

[0072] The pixel group motion compensation unit 120 can determine the current displacement vector per unit time for the pixel group by utilizing the pixel value and gradient value of the pixels in the first window and the pixel value and gradient value of the pixels in the second window. At this time, the value of the current displacement vector per unit time for the pixel group is also adjusted by a regularization parameter. The regularization parameter is a parameter introduced to prevent errors from occurring when an ill-posed displacement vector per unit time for the current pixel group is determined in order to perform motion compensation for the pixel group. The pixel group motion compensation unit 120 can perform motion compensation for the current block at the pixel group level based on the regularization parameter for the horizontal or vertical displacement vector per unit time. The regularization parameter will be described later in the explanation of Figure 8A.

[0073] The pixel group motion compensation unit 120 can perform motion compensation for the current block on a pixel group basis by utilizing the displacement vector per unit time related to the current pixel group and the gradient value of the reference pixel.

[0074] The reference position of a reference block is, but not limited to, an integer pixel position; it can also be a decimal pixel position.

[0075] If the reference position of a reference block is a decimal pixel position, the gradient value of the reference pixels within the reference block can also be determined using the pixel values ​​of surrounding pixels whose given directional component is an integer.

[0076] For example, the motion compensation unit 120 for each pixel group can filter the pixel values ​​of surrounding pixels whose predetermined directional components are integers using a gradient filter, and determine the gradient value of the reference pixel using the resulting value. In this case, the filter coefficients of the gradient filter can also be determined using coefficients predetermined for the interpolation filter of the DCT base. These gradient filter coefficients are also scaled to integer coefficients in order to reduce real number calculations during filtering.

[0077] In this case, the gradient filter is also a one-dimensional gradient filter in the horizontal or vertical direction.

[0078] The pixel group motion compensation unit 120 can use a horizontal or vertical one-dimensional gradient filter to determine the horizontal or vertical gradient value related to the reference pixel, and can filter surrounding pixels whose corresponding directional components are integers.

[0079] For example, the pixel group motion compensation unit 120 can use a horizontal one-dimensional gradient filter to filter pixels located around the reference pixel, specifically those whose horizontal component is an integer, and then determine the horizontal gradient value related to the reference pixel.

[0080] If the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are decimals), the pixel group motion compensation unit 120 can filter the pixel at position (x,y) and the pixels located vertically from the pixel at position (x,y) whose vertical component is an integer using a vertical one-dimensional interpolation filter, and determine the pixel value of (x,y+β) using the resulting value.

[0081] The pixel group motion compensation unit 120 filters the pixel value at position (x, y+β) and the pixels located horizontally from position (x, y+β) whose horizontal component is an integer, using a horizontal gradient filter, and determines the horizontal gradient value at position (x+α, y+β) based on the resulting value.

[0082] The order in which the one-dimensional gradient filter and the one-dimensional interpolation filter are used is not restricted. While the above describes a method where a vertical interpolation filter is first used to filter pixels at integer positions to generate vertical interpolation filtering values, and then a one-dimensional horizontal gradient filter is used to filter these vertical interpolation filtering values, it is also possible to first use a one-dimensional horizontal gradient filter to filter pixels at integer positions to generate horizontal interpolation filtering values, and then use a one-dimensional vertical interpolation filter to filter these horizontal interpolation filtering values.

[0083] The above has provided a detailed explanation of how the pixel group motion compensation unit 120 determines the horizontal gradient value at position (x+α, y+β). The pixel group motion compensation unit 120 also determines the vertical gradient value at position (x+α, y+β) in a similar manner to how the horizontal gradient value is determined, so a detailed explanation is omitted.

[0084] The above has described in detail how the motion compensation unit (230) for each pixel group uses a one-dimensional gradient filter and a one-dimensional interpolation filter to determine the gradient value at decimal pixel positions. However, it is not limited to these, and the gradient filter and interpolation filter can also be used to determine the gradient value at integer pixel positions. In the case of integer pixels, the pixel value can be determined even without using an interpolation filter, but in order to maintain consistent processing with the processing at decimal pixels, the interpolation filter can be used to filter integer pixels and surrounding pixels whose predetermined directional components are integers to determine the integer pixel value. For example, the interpolation filter coefficient for an integer pixel is {0,0,64,0,0}, and since the interpolation filter coefficient related to the surrounding integer pixels is 0, filtering is performed using only the pixel value of the current integer pixel. As a result, the interpolation filter is used to filter the current integer pixel and surrounding integer pixels, and the pixel value of the current integer pixel is determined.

[0085] The pixel group motion compensation unit 120 can perform inverse scaling after filtering pixels at integer positions using a vertical one-dimensional interpolation filter. In this case, the inverse scaling may include bit shifting to the right by a number of inverse scaling bits. The number of inverse scaling bits can also be determined based on the bit depth of the sample. Alternatively, the number of inverse scaling bits can also be determined based on the specific input data within the block.

[0086] For example, the bit shifting number is also the sample bit depth minus 8.

[0087] The pixel group motion compensation unit 120 can perform inverse scaling after filtering the values ​​generated by the inverse scaling using a horizontal gradient filter. Similarly, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits can also be determined based on the scaled number of bits of the vertical one-dimensional interpolation filter, the scaled number of bits of the horizontal one-dimensional gradient filter, and the bit depth of the sample. For example, if the scaling number of bits p of the one-dimensional interpolation filter is 6, the scaling number of bits q of the one-dimensional gradient filter is 4, and the bit depth of the sample is b, then the number of inverse scaling bits is p + q + 8 - b, and also 18 - b.

[0088] If the pixel group motion compensation unit 120 performs bit shifting to the right by the number of inverse scaling bits on the filtered value after filtering, a rounding error may occur. Therefore, an offset is added to the filtered value, and then inverse scaling is performed. In this case, the offset is also 2^(number of inverse scaling bits - 1).

[0089] The interpretation unit 110 can generate predicted pixel values ​​for the current block by utilizing block-level motion compensation values ​​and pixel group-level motion compensation values ​​related to the current block. For example, the interpretation unit 110 can generate predicted pixel values ​​for the current block by combining block-level motion compensation values ​​and pixel group-level motion compensation values ​​related to the current block. Here, the block-level motion compensation value means the value generated by performing block-level motion compensation, and the pixel group-level motion compensation value is the value generated by performing pixel group-level motion compensation. The block-level motion compensation value is also the average value or weighted sum of the reference pixels, and the pixel group-level motion compensation value is also a value determined based on the displacement vector per unit time for the current pixel and the gradient value of the reference pixels.

[0090] The pixel group motion compensation unit 120 can obtain a shift value for descaling after interpolation or gradient calculation, based on at least one of the sample bit depth, the input range of the filter used for interpolation or gradient calculation, and the coefficients of the filter. The pixel group motion compensation unit 120 can use the shift value for descaling to perform descaling after interpolation or gradient calculation on the pixels included in the first and second reference blocks.

[0091] When the interpretation unit 110 performs block-level motion compensation, it can utilize and store motion vectors. In this case, the motion vector unit is also a 4x4 block. On the other hand, when storing motion vectors after block-level motion compensation, the motion vector storage unit can also be a block of various sizes other than 4x4 (for example, an RxR block; R is an integer). In this case, the motion vector storage unit can also be a block larger than 4x4. For example, it can be a 16x16 block.

[0092] On the other hand, when performing motion compensation at the pixel group level, the size of the target block for motion compensation is expanded based on the window size and interpolation filter length, along with the current block size. The reason the target block size is expanded beyond the current block size based on the window size is that, for pixels located at the edges of the current block, the window is used to perform motion compensation at the pixel group level related to the current block, based on the pixels located at the edges and surrounding pixels.

[0093] Therefore, in order to reduce the number of memory accesses and multiplication operations, the pixel group motion compensation unit 120, in the process of performing motion compensation on a pixel group basis using a window, adjusts the position of a pixel that is currently outside the block to the position of a pixel adjacent to the current block, and determines the pixel value and gradient value at the adjusted pixel position, thereby reducing the number of memory accesses and multiplication operations.

[0094] The pixel group motion compensation unit 120 does not use the pixel values ​​of integer-position pixels to determine the gradient value of the reference pixel, which is the value necessary for motion compensation at the pixel group level. That is, the pixel group compensation unit 120 can apply a horizontal or vertical gradient filter to the pixel value of a decimal-position pixel to calculate the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block or the second corresponding reference pixel in the second reference block. In this case, the gradient filter length is also 5. In this case, the filter coefficients can have coefficients that are symmetrical with respect to the filter's central coefficient. A decimal-position pixel is also a pixel in which at least one of the horizontal and vertical components indicating the pixel's position has a decimal value.

[0095] The pixel values ​​of decimal-position pixels within the first or second reference block are also calculated by applying a horizontal or vertical interpolation filter to the pixel values ​​of integer-position pixels.

[0096] The current pixel group's horizontal or vertical displacement vector per unit time is also a displacement vector per unit time determined using values ​​determined based on the pixel values ​​and gradient values ​​of the surrounding pixels, the first corresponding reference pixels included in the first corresponding reference pixel group within the first reference picture corresponding to the current pixel group, the second corresponding reference pixels included in the second corresponding reference pixel group within the second reference picture, and the first POC (picture order count) difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture.

[0097] The pixel group unit compensation unit 120 can use the pixel value of the boundary-peripheral pixel to derive the pixel value of a pixel located outside the boundary of the first or second reference block if the first corresponding reference pixel or the second corresponding reference pixel is a boundary-peripheral pixel adjacent to the boundary inside the boundary of the first or second reference block.

[0098] The pixel group unit compensation unit 120 can determine the horizontal or vertical displacement vector per unit time of the current block based on the pixel values ​​of the boundary-peripheral pixels and the pixel values ​​of pixels located outside the boundary of the current block, which are derived using the pixel values ​​of the boundary-peripheral pixels. That is, if there are pixels located outside the boundary within a window centered on the boundary-peripheral pixels, then the pixel values ​​and gradient values ​​of the pixels located outside the boundary are also the pixel values ​​and gradient values ​​of pixels derived from boundary-peripheral pixels, which are not values ​​stored in memory.

[0099] The pixel group unit compensation unit 120 can calculate a value related to the current pixel using the pixel value of the first corresponding reference pixel included in the first reference block, the pixel value of the second corresponding reference pixel included in the second reference block, the gradient value of the first corresponding reference pixel, the gradient value of the second corresponding reference pixel, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture. In other words, the value related to the current pixel is also the result value of a function based on the pixel value and gradient value of the corresponding reference pixel of each reference picture, and the POC difference between each reference picture and the current picture.

[0100] The pixel group unit compensation unit 120 can calculate values ​​related to surrounding pixels using the pixel value of the first corresponding surrounding pixel of the first corresponding reference pixel, the gradient value of the first corresponding surrounding pixel, the pixel value of the second corresponding surrounding pixel of the second corresponding reference pixel, the gradient value of the second corresponding surrounding pixel, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture. In other words, the values ​​related to surrounding pixels are also the result of a function based on the pixel value and gradient value of the corresponding reference pixel of each reference picture, and the POC difference between each reference picture and the current picture.

[0101] The pixel group unit compensation unit 120 can use the value related to the current pixel, the value related to the surrounding pixels, and the weight value to calculate the weighted average value for the current pixel necessary to calculate the horizontal displacement vector per unit time. At this time, the weight value is also determined based on the distance between the current pixel and surrounding pixels, the distance between the pixel and the block boundary, the number of pixels located outside the boundary, or whether the pixel is located inside or outside the boundary.

[0102] The weighted average value for the current pixel is also calculated by applying exponential smoothing in the vertical, horizontal, and vertical directions to the values ​​related to pixels contained in the first and second reference blocks. By applying exponential smoothing in the vertical, horizontal, and vertical directions to the values ​​related to pixels, the value calculated for the current pixel has the largest weight related to the current pixel, and the weight related to the values ​​of surrounding pixels decreases exponentially with distance from the current pixel.

[0103] The pixel group unit compensation unit 120 can determine the displacement vector per unit time in the horizontal or vertical direction of the current pixel by utilizing the weighted average value for the current pixel.

[0104] The restoration unit 125 can obtain the current block's residual block from the bitstream and restore the current block using the residual block and the predicted block of the current block. For example, the restoration unit 125 can generate the pixel value of the restored block by combining the pixel value of the current block's residual block and the pixel value of the predicted block of the current block from the bitstream.

[0105] The video decoding device 100 may include a video decoding unit (not shown), which may include an acquisition unit 105, an interpretation unit 110, and a reconstruction unit 125. The video decoding unit will be described with reference to Figure 1E.

[0106] Figure 1B illustrates flowcharts of video decoding methods according to various embodiments.

[0107] In step S105, the video decoder 100 can obtain motion prediction mode information relating to the current block in the current picture from the bitstream. The video decoder 100 receives a bitstream containing motion prediction mode information relating to the current block in the current picture, and can obtain motion prediction mode information relating to the current block from the received bitstream. The video decoder 100 obtains information about the prediction mode of the current block from the bitstream, and can determine the prediction mode of the current block based on the information about the prediction mode of the current block. At this time, if the prediction mode of the current block is inter-prediction mode, the video decoder 100 can obtain motion prediction mode information relating to the current block.

[0108] For example, the video decoder 100 can determine the prediction mode of the current block to be inter-prediction mode based on information about the prediction mode of the current block. If the prediction mode of the current block is inter-prediction mode, the video decoder 100 can obtain motion prediction mode information related to the current block from the bitstream.

[0109] In step S110, if the motion prediction mode information indicates a bidirectional motion prediction mode, the video decoder 100 can obtain from the bitstream a first motion vector in the first reference picture that indicates the first reference block of the current block, and a second motion vector in the second reference picture that indicates the second reference block of the current block.

[0110] In other words, the video decoder 100 receives a bitstream containing information about a first motion vector and a second motion vector, and can obtain the first motion vector and the second motion vector from the received bitstream. The video decoder 100 obtains a reference picture index from the bitstream, and based on the reference picture index, can determine the first reference picture and the second reference picture from previously decoded pictures.

[0111] In step S115, the video decoder 100 can determine the horizontal or vertical displacement vector of pixels in the current block, including pixels adjacent to the current block's boundary, by using values ​​related to reference pixels contained within the first and second reference blocks, without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks. At this time, the values ​​stored for pixels located outside the boundaries of the first and second reference blocks, as well as the values ​​related to reference pixels contained within the first and second reference blocks, are also the pixel values ​​of the related pixels, or the horizontal gradient values ​​or vertical gradient values ​​of the related pixels. Alternatively, the values ​​stored for pixels located outside the boundaries of the first and second reference blocks, as well as the values ​​related to reference pixels contained within the first and second reference blocks, are also values ​​determined using the pixel values ​​or gradient values ​​of the related pixels.

[0112] In step S120, the video decoder 100 uses the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector of the current pixel to perform block-level motion compensation and pixel group-level motion compensation for the current block, thereby obtaining a predicted block for the current block.

[0113] In other words, the video decoder 100 can generate a predicted block for the current block by performing block-level motion compensation and pixel group-level motion compensation for the current block based on the first motion vector and the second motion vector. The video decoder 100 can perform block-level motion compensation for the current block using the pixel value of the first reference block indicated by the first motion vector and the pixel value of the second reference block indicated by the second motion vector. Furthermore, the video decoder 100 can perform pixel group-level motion compensation for the current pixel group using the horizontal or vertical gradient value of at least one first corresponding reference pixel in the first reference block corresponding to at least one pixel included in the current pixel group, the horizontal or vertical gradient value of at least one second corresponding reference pixel in the second reference block, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector of the current pixel.

[0114] The video decoding device 100 can obtain a predicted block for the current block by using the block-level motion compensation value generated by performing block-level motion compensation on the current block, and the pixel group-level motion compensation value generated by performing pixel group-level motion compensation on the current pixel group.

[0115] In step S125, the video decoder 100 can obtain information about the current block's residual block from the bitstream.

[0116] In step S130, the video decoder 100 can reconstruct the current block based on the predicted block and the residual block. That is, the video decoder 100 can combine the pixel value of the residual block indicated by the residual block related to the current block with the predicted pixel value of the predicted block to generate the pixel value of the reconstructed block of the current block.

[0117] Figure 1C illustrates block diagrams of video encoding devices according to various embodiments.

[0118] The video encoding apparatus 150, according to various embodiments, includes an interpretation unit 155 and a bitstream generation unit 170.

[0119] The interpretation unit 155 performs interpretation on the current block by referencing various blocks based on the rate and distortion cost. That is, the interpretation unit 155 can generate predicted pixel values ​​for the current block by utilizing at least one picture encoded prior to the current picture in which the current block is contained.

[0120] The interpretation unit 155 may include a block-level motion compensation unit 160 and a pixel group-level motion compensation unit 165.

[0121] The block-level motion compensation unit 160 can perform block-level motion compensation on the current block and generate block-level motion compensation values.

[0122] The block-level motion compensation unit 160 can determine at least one reference picture from previously decoded pictures and determine the reference block of the current block located within at least one reference picture.

[0123] The block-level motion compensation unit 160 can use the pixel values ​​of the reference block to perform block-level motion compensation related to the current block and generate a block-level motion compensation value. The block-level motion compensation unit 160 can use the reference pixel values ​​of the reference block corresponding to the current pixel of the current block to perform block-level motion compensation related to the current block and generate a block-level motion compensation value.

[0124] The block-level motion compensation unit 160 can use multiple reference blocks contained in each of the multiple reference pictures to perform block-level motion compensation related to the current block and generate a block-level motion compensation value. For example, if the motion prediction mode of the current block is bidirectional prediction mode, the block-level motion compensation unit 160 can determine two reference pictures from among previously encoded pictures and determine two reference blocks contained in the two pictures. Here, bidirectional prediction is not limited to meaning that interpretation is performed using a picture whose display order is earlier than the current picture and a picture whose display order is later than the current picture, but rather means that interpretation is performed using two pictures encoded earlier than the current picture, regardless of the display order.

[0125] The block-level motion compensation unit 160 can perform block-level motion compensation related to the current block using two reference pixel values ​​within two reference blocks and generate a block-level motion compensation value. The block-level motion compensation unit 160 can perform block-level motion compensation related to the current block using the average pixel value or weighted sum of two reference pixels and generate a block-level motion compensation value.

[0126] The block-level motion compensation unit 160 can output a reference picture index that indicates a reference picture for motion compensation of the current block from among previously encoded pictures.

[0127] The block-level motion compensation unit 160 can determine a motion vector that starts at the current block and ends at the reference block of the current block, and output the motion vector. This motion vector represents a vector indicating the displacement between the reference coordinates of the current block in the current picture and the reference coordinates of the reference block in the reference picture. For example, if the coordinates of the upper left corner of the current block are (1,1) and the upper left coordinates of the reference block in the reference picture are (3,3), then the motion vector is also (2,2).

[0128] The reference position of a reference block is, but not limited to, an integer pixel position; it can also be a decimal pixel position. In this case, the decimal pixel position may be determined in units of 1 / 4 or 1 / 16 of a per unit, but not limited to these; it can also be determined by a variety of decimal units.

[0129] For example, if the reference position of a reference block is (1.5, 1.5) and the coordinates of the current upper-left corner of the block are (1, 1), then the motion vector is also (0.5, 0.5). If the motion vector is determined to be in 1 / 4 or 1 / 16 per unit to represent the reference position of a reference block which is a decimal pixel position, the motion vector can be scaled to determine an integer motion vector, and the reference position of the reference block can be determined using the upscaled motion vector. If the reference position of the reference block is a decimal pixel position, then the position of the reference pixel of the reference block is also a decimal pixel position. Therefore, in the reference block, the pixel value of a decimal pixel position can also be determined using the pixel value of a surrounding pixel whose predetermined directional component is an integer.

[0130] For example, the block-unit motion compensation unit 160 can use an interpolation filter to filter the pixel values ​​of surrounding pixels whose predetermined directional components are integers, determine a reference pixel value at a decimal pixel position using the resulting value, and use the pixel value of the reference pixel to determine the block-unit motion compensation value for the current block. In this case, the interpolation filter can be the M-tap interpolation filter of the DCT base. The coefficients of the M-tap interpolation filter of the DCT base can also be derived from the DCT and IDCT. In this case, the coefficients of the interpolation filter are also filter coefficients scaled to integer coefficients in order to reduce real number calculations during filtering.

[0131] In this case, the interpolation filter is also a one-dimensional interpolation filter in the horizontal or vertical direction.

[0132] The block-level motion compensation unit 160 can first filter surrounding integer pixels using a vertical one-dimensional interpolation filter, and then filter the filtered values ​​using a horizontal one-dimensional interpolation filter to determine the reference pixel value at decimal pixel positions. When using scaled filter coefficients, the block-level motion compensation unit 160 can filter pixels at integer positions using a vertical one-dimensional interpolation filter, and then perform inverse scaling on the filtered values. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits can also be determined based on the sample bit depth. For example, the number of bit shifts is also the sample bit depth minus 8.

[0133] Furthermore, the block-level motion compensation unit 160 may include using a horizontal one-dimensional interpolation filter to filter pixels whose horizontal components are integers, and then bit-shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits can also be determined based on the number of bits scaled with respect to the vertical one-dimensional interpolation filter coefficients, the number of bits scaled with respect to the horizontal one-dimensional interpolation filter coefficients, and the bit depth of the sample.

[0134] If the block-level motion compensation unit 160 only performs bit shifting to the right by the number of inverse scaling bits, a rounding error may occur. Therefore, a one-dimensional interpolation filter in a predetermined direction is used to filter pixels whose components in the predetermined direction are integers, an offset is added to the filtered value, and inverse scaling is performed on the value to which the offset has been added. In this case, the offset is also 2^(number of inverse scaling bits - 1).

[0135] Earlier, we explained that after filtering using a vertical one-dimensional interpolation filter, the number of inverse scaling bits is determined based on the sample's bit depth. However, it is not limited to this; it can also be determined by considering the number of bits scaled to the interpolation filter coefficients, not just the sample's bit depth. That is, during filtering, the size of the registers used and the size of the buffer that stores the values ​​generated during the filtering process are considered, and the number of inverse scaling bits is determined based on the sample's bit depth and the number of bits scaled to the interpolation filter coefficients, within a range that does not cause overflow.

[0136] The pixel group motion compensation unit 165 can perform pixel group motion compensation for the current block and generate a pixel group motion compensation value. For example, if the motion prediction mode is a bidirectional motion prediction mode, the pixel group motion compensation unit 165 can perform pixel group motion compensation for the current block and generate a pixel group motion compensation value.

[0137] The pixel group motion compensation unit 165 can use the gradient values ​​of the pixels included in the reference block of the current block to perform pixel group motion compensation related to the current block and generate a pixel group motion compensation value.

[0138] The pixel group motion compensation unit 165 can apply a filter to the first peripheral region of the first pixel among the pixels of the first reference block in the first reference picture, and to the second peripheral region of the second pixel among the pixels of the second reference block in the second reference picture, thereby generating a gradient value for the first pixel and a gradient value for the second pixel.

[0139] The pixel group motion compensation unit 165 can determine the pixel value and pixel gradient value of pixels in a predetermined size first window containing the first reference pixel, centered on the first reference pixel in the first reference picture, and determine the pixel value and pixel gradient value of pixels in a predetermined size second window containing the second reference pixel, centered on the second reference pixel in the second reference picture. The pixel group motion compensation unit 165 can use the pixel value and pixel gradient value of the pixels in the first window and the pixel value and pixel gradient value of the pixels in the second window to determine the displacement vector per unit time related to the current pixel.

[0140] The pixel group motion compensation unit 165 can use the displacement vector per unit time and the gradient value of the reference pixel to perform motion compensation on a pixel group basis related to the current block and generate a pixel group motion compensation value.

[0141] The position of a reference pixel is, but not limited to, an integer pixel position; it can also be a decimal pixel position.

[0142] If the reference position of a reference block is a decimal pixel position, the gradient value of the reference pixels within the reference block can also be determined using the pixel values ​​of surrounding pixels whose given directional component is an integer.

[0143] For example, the motion compensation unit 165 for each pixel group can filter the pixel values ​​of surrounding pixels whose predetermined directional components are integers using a gradient filter, and determine the gradient value of the reference pixel using the resulting value. In this case, the filter coefficients of the gradient filter can also be determined using coefficients predetermined for the interpolation filter of the DCT base.

[0144] The coefficients of a gradient filter are also the filter coefficients scaled to integer coefficients in order to reduce real number calculations during filtering. In this case, the gradient filter used is also a one-dimensional gradient filter in the horizontal or vertical direction.

[0145] The pixel group motion compensation unit 165 can use a horizontal or vertical one-dimensional gradient filter to determine the horizontal or vertical gradient value related to the reference pixel, and can filter surrounding pixels whose corresponding directional components are integers.

[0146] For example, the pixel group motion compensation unit 165 can use a vertical one-dimensional interpolation filter to filter out pixels whose vertical component is an integer from integer pixels tangent to the reference pixel, and determine the pixel value of pixels whose vertical component is a decimal.

[0147] The pixel group motion compensation unit 165 can also use a vertical one-dimensional interpolation filter to filter surrounding integer pixels in the vertical direction, even for pixels located in other columns adjacent to the reference pixel and the integer pixels adjacent to it, thereby determining the pixel value of a decimal pixel located in the other column. Here, the position of a pixel located in the other column is a decimal pixel position in the vertical direction and an integer pixel position in the horizontal direction.

[0148] In other words, if the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are decimals), the pixel group motion compensation unit 165 can determine the pixel value at position (x, y+β) by filtering the surrounding integer pixels in the vertical direction from position (x, y) using a vertical interpolation filter.

[0149] The pixel group motion compensation unit 165 can determine the horizontal gradient value at position (x+α,y+β) by filtering the pixel values ​​of pixels located horizontally whose horizontal component is an integer, using a horizontal gradient filter, based on the pixel value at position (x,y+β) and the pixel value at position (x,y+β).

[0150] The order in which the one-dimensional gradient filter and the one-dimensional interpolation filter are used is not restricted. As mentioned above, one can first use the vertical interpolation filter to filter pixels at integer positions to generate vertical interpolation filtering values, then use the one-dimensional horizontal gradient filter to filter the vertical interpolation filtering values, but this is not the only way. One can first use the one-dimensional horizontal gradient filter to filter pixels at integer positions to generate horizontal gradient filtering values, then use the one-dimensional vertical interpolation filter to filter the horizontal gradient filtering values.

[0151] The above has provided a detailed explanation of how the motion compensation unit 165 for each pixel group determines the horizontal gradient value at the (x+α, y+β) position.

[0152] The motion compensation unit 165 for each pixel group can determine the vertical gradient value at the (x+α, y+β) position in a manner similar to how the horizontal gradient value is determined.

[0153] The pixel group motion compensation unit 165 can determine the vertical gradient value related to the reference pixel by using a vertical one-dimensional gradient filter to filter from integer pixels around the reference pixel to the vertically surrounding integer pixels. The pixel group motion compensation unit 165 can also determine the vertical gradient value related to pixels located in other columns adjacent to the reference pixel by using a vertical one-dimensional gradient filter to filter from the vertically surrounding integer pixels, while being adjacent to the reference pixel and located in other columns. Here, the position of the pixel is a decimal pixel position in the vertical direction and an integer pixel position in the horizontal direction.

[0154] In other words, if the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are decimals), the pixel group motion compensation unit 165 can determine the vertical gradient value at position (x,y+β) by filtering the surrounding integer pixels in the vertical direction from position (x,y) using a vertical gradient filter.

[0155] The pixel group motion compensation unit 165 can determine the vertical gradient value at position (x+α,y+β) by filtering the gradient value at position (x,y+β) and the gradient values ​​of surrounding integer pixels located horizontally from position (x,y+β) using a horizontal interpolation filter.

[0156] The order in which the one-dimensional gradient filter and the one-dimensional interpolation filter are used is not restricted. As mentioned above, one can first use the vertical gradient filter to filter pixels at integer positions to generate vertical gradient filtering values, and then use the one-dimensional horizontal interpolation filter to filter the vertical gradient filtering values, but this is not the only way. One can first use the one-dimensional horizontal interpolation filter to filter pixels at integer positions to generate horizontal interpolation filtering values, and then use the one-dimensional vertical gradient filter to filter the horizontal interpolation filtering values.

[0157] The above has described in detail how the pixel group motion compensation unit 165 uses a gradient filter and an interpolation filter to determine the gradient value at decimal pixel positions. However, it is not limited to these methods, and the gradient filter and interpolation filter can also be used to determine the gradient value at integer pixel positions.

[0158] In the case of integer pixels, the pixel value can be determined even without using an interpolation filter. However, for consistent processing with decimal pixels, filtering is sometimes performed using an interpolation filter on the integer pixel and surrounding integer pixels. For example, the interpolation filter coefficient for an integer pixel is {0,0,64,0,0}, and since the interpolation filter coefficient multiplied by the surrounding integer pixels is 0, filtering is performed using only the pixel value of the current integer pixel. As a result, the value generated by filtering using an interpolation filter on the current integer pixel and surrounding integer pixels is determined to be the same as the pixel value of the current integer pixel.

[0159] On the other hand, when using scaled filter coefficients, the motion compensation unit 165 for each pixel group can use a horizontal one-dimensional gradient filter to filter pixels at integer positions and then inversely scale the filtered values. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits can also be determined based on the sample's bit depth. For example, the number of inverse scaling bits can be the sample's bit depth minus 8.

[0160] The pixel group motion compensation unit 165 can utilize a vertical interpolation filter to filter pixels whose vertical components are integers, and then perform inverse scaling. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits can also be determined based on the scaled number of bits of the vertical one-dimensional interpolation filter, the scaled number of bits of the horizontal one-dimensional gradient filter, and the bit depth of the sample.

[0161] If the pixel group motion compensation unit 165 only performs bit shifting to the right by the number of inverse scaling bits, a rounding error may occur. Therefore, after filtering is performed using a one-dimensional interpolation filter, an offset is added to the filtered value, and inverse scaling can be performed on the offset value. In this case, the offset is also 2^(number of bit shifts - 1).

[0162] When the interpretation unit 110 performs block-level motion compensation, it can utilize and save motion vectors. In this case, the motion vector unit is also a 4x4 block. On the other hand, when saving motion vectors after block-level motion compensation, the motion vector storage unit can also be a block of various sizes other than 4x4 (for example, an RxR block; R is an integer). In this case, the motion vector storage unit can also be a block larger than 4x4. For example, it can be a 16x16 block.

[0163] On the other hand, when performing motion compensation at the pixel group level, the size of the target block for motion compensation is expanded based on the window size and interpolation filter length, along with the current block size. The reason the target block size is expanded beyond the current block size based on the window size is that, for pixels located at the edges of the current block, the window is used to perform motion compensation at the pixel group level related to the current block, based on the pixels located at the edges and surrounding pixels.

[0164] Therefore, in order to reduce the number of memory accesses and multiplication operations, the pixel group motion compensation unit 120, in the process of performing motion compensation on a pixel group basis using a window, adjusts the position of a pixel that is currently outside the block to the position of a pixel adjacent to the current block, and determines the pixel value and gradient value at the adjusted pixel position, thereby reducing the number of memory accesses and multiplication operations.

[0165] The pixel group motion compensation unit 120 does not use the pixel values ​​of integer-position pixels to determine the gradient value of the reference pixel, which is the value necessary for motion compensation at the pixel group level. That is, the pixel group compensation unit 120 can apply a horizontal or vertical gradient filter to the pixel value of a decimal-position pixel to calculate the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block or the second corresponding reference pixel in the second reference block. In this case, the gradient filter length is also 5. In this case, the filter coefficients can have coefficients that are symmetrical with respect to the filter's central coefficient. A decimal-position pixel is also a pixel in which at least one of the horizontal and vertical components indicating the pixel's position has a decimal value.

[0166] The pixel values ​​of decimal-position pixels within the first or second reference block are also calculated by applying a horizontal or vertical interpolation filter to the pixel values ​​of integer-position pixels.

[0167] The current horizontal or vertical displacement vector per unit time for a pixel group is also a displacement vector per unit time determined using values ​​determined based on the pixel values ​​and gradient values ​​of the surrounding pixels, the first corresponding reference pixels included in the first corresponding reference pixel group in the first reference picture corresponding to the current pixel group, the second corresponding reference pixels included in the second corresponding reference pixel group in the second reference picture, and the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture.

[0168] The pixel group unit compensation unit 120 can use the pixel value of the boundary-peripheral pixel to derive the pixel value of a pixel located outside the boundary of the first or second reference block if the first corresponding reference pixel or the second corresponding reference pixel is a boundary-peripheral pixel adjacent to the boundary inside the boundary of the first or second reference block.

[0169] The pixel group unit compensation unit 120 can determine the horizontal or vertical displacement vector per unit time of the current block based on the pixel values ​​of the boundary-peripheral pixels and the pixel values ​​of pixels located outside the boundary of the current block, which are derived using the pixel values ​​of the boundary-peripheral pixels. That is, if there are pixels located outside the boundary within a window centered on the boundary-peripheral pixels, then the pixel values ​​and gradient values ​​of the pixels located outside the boundary are also the pixel values ​​and gradient values ​​of pixels derived from boundary-peripheral pixels, which are not values ​​stored in memory.

[0170] The pixel group unit compensation unit 120 can calculate a value related to the current pixel using the pixel value of the first corresponding reference pixel included in the first reference block, the pixel value of the second corresponding reference pixel included in the second reference block, the gradient value of the first corresponding reference pixel, the gradient value of the second corresponding reference pixel, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture. In other words, the value related to the current pixel is also the result value of a function based on the pixel value and gradient value of the corresponding reference pixel of each reference picture, and the POC difference between each reference picture and the current picture.

[0171] The pixel group unit compensation unit 120 can calculate values ​​related to corresponding surrounding pixels calculated using the pixel value of the first corresponding surrounding pixel of the first corresponding reference pixel, the gradient value of the first corresponding surrounding pixel, the pixel value of the second corresponding surrounding pixel of the second corresponding reference pixel, the gradient value of the second corresponding surrounding pixel, the first POC difference between the first reference picture and the current picture, and the second POC difference between the second reference picture and the current picture. In other words, the values ​​related to surrounding pixels are also the result values ​​of a function based on the pixel value and gradient value of the corresponding reference pixel of each reference picture, and the POC difference between each reference picture and the current picture.

[0172] The pixel group unit compensation unit 120 can calculate a weighted average value for the current pixel, which is necessary for calculating the horizontal displacement vector per unit time, by using the value related to the current pixel, the value related to the corresponding surrounding pixel, and the weight value. At this time, the weight value is also determined based on the distance between the current pixel and surrounding pixels, the distance between the pixel and the block boundary, the number of pixels located outside the boundary, or whether the pixel is located inside or outside the boundary.

[0173] The weighted average value for the current pixel is also calculated by applying exponential smoothing in the up, down, left, and right directions to the values ​​related to pixels contained in the first and second reference blocks. By applying exponential smoothing in the up, down, left, and right directions to the values ​​related to pixels, the value calculated for the current pixel has the largest weight related to the current pixel, and the weight related to the values ​​of surrounding pixels decreases exponentially with distance from the current pixel.

[0174] The pixel group unit compensation unit 120 can determine the displacement vector per unit time in the horizontal or vertical direction of the current pixel by utilizing the weighted average value for the current pixel.

[0175] The interpretation unit 155 can generate predicted pixel values ​​for the current block by utilizing the block-level motion compensation value and the pixel group-level motion compensation value related to the current block. For example, the interpretation unit 155 can generate predicted pixel values ​​for the current block by combining the block-level motion compensation value and the pixel group-level motion compensation value related to the current block. In particular, when the motion prediction mode of the current block is the bidirectional motion prediction mode, the interpretation unit 155 can generate predicted pixel values ​​for the current block by utilizing the block-level motion compensation value and the pixel group-level motion compensation value related to the current block.

[0176] The interpretation unit 155 can generate predicted pixel values ​​for the current block by utilizing block-level motion compensation values ​​related to the current block, if the motion prediction mode for the current block is unidirectional motion prediction mode. Here, unidirectional means using one reference picture from among previously encoded pictures. The reference picture is not limited to being a picture prior to the current picture in terms of display order, but can also be a picture after it.

[0177] The interpretation unit 155 can determine the motion prediction mode of the current block and output information indicating the motion prediction mode of the current block. For example, the interpretation unit 155 can determine the bidirectional motion prediction mode in the current block's motion prediction mode and output information indicating the bidirectional motion prediction mode. Here, the bidirectional motion prediction mode refers to a mode in which motion is predicted using reference blocks in two decoded reference pictures.

[0178] The bitstream generation unit 170 can generate a bitstream that includes motion vectors indicating reference blocks. The bitstream generation unit 170 can encode motion vectors indicating reference blocks and generate a bitstream that includes the encoded motion vectors. The bitstream generation unit 170 can encode the difference value of the motion vectors indicating reference blocks and generate a bitstream that includes the encoded difference value of the motion vectors. Here, the difference value of the motion vectors means the difference between the motion vector and the predictor of the motion vector. In this case, the difference value of the motion vectors means the difference value of the motion vectors related to the reference pictures related to each of the prediction directions, including the L0 direction and the L1 direction. Here, the difference value of the motion vectors related to the L0 direction means the difference value of the motion vectors indicating reference blocks in the reference pictures included in the L0 reference picture list, and the difference value of the motion vectors related to the L1 direction means the difference value of the motion vectors indicating reference blocks in the reference pictures included in the L1 reference picture list.

[0179] Furthermore, the bitstream generation unit 170 can generate a bitstream that further includes information indicating the motion prediction mode of the current block. The bitstream generation unit 170 can encode a reference picture index that indicates the reference picture of the current block from among the previously encoded pictures, and generate a bitstream that includes the encoded reference picture index. In this case, the reference picture index means the reference picture index related to each prediction direction, including the L0 direction and the L1 direction. Here, the reference picture index related to the L0 direction means the index that indicates the reference picture among the pictures included in the L0 reference picture list, and the reference picture index related to the L1 direction means the index that indicates the reference picture among the pictures included in the L1 reference picture list.

[0180] The video encoding device 150 may include a video encoding unit (not shown), and the video encoding unit (not shown) may include an interpretation unit 155 and a bitstream generation unit 170. The video encoding unit will be described with reference to Figure 1F.

[0181] Figure 1D illustrates flowcharts of video encoding methods according to various embodiments.

[0182] Referring to Figure 1D, in step S150, the video encoding device 150 performs motion compensation for the current block and motion compensation for pixel groups, and can obtain the predicted block for the current block, the first motion vector and the second motion vector, and the parameters related to motion compensation for pixel groups.

[0183] In step S155, the video encoding device 150 can generate a bitstream containing information about a first motion vector and a second motion vector, and motion prediction mode information indicating whether or not the motion prediction mode related to the current block is a bidirectional motion prediction mode. Here, the first motion vector is also a motion vector that points from the current block to the first reference block of the first reference picture corresponding to the current block in the current picture, and the second motion vector is also a motion vector that points from the current block to the second reference block of the second reference picture corresponding to the current block in the current picture.

[0184] The video encoder 150 can encode the current block's residual block, which shows the difference between the pixels of the current block's predicted block and the current block's original block, and generate a bitstream further containing the encoded residual signal. The video encoder 150 can encode information about the current block's prediction mode and a reference picture index, and generate a bitstream further containing the encoded information about the current block's prediction mode and the reference picture index. For example, the video encoder 150 can encode information indicating whether the current block's prediction mode is interprediction mode and a reference picture index indicating at least one picture from previously decoded pictures, and generate a bitstream further containing the encoded information about the current block's prediction mode and the reference picture index.

[0185] The video encoding device 150 can perform block-level motion compensation and pixel group-level motion compensation for the current block based on the horizontal or vertical gradient value of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group within the current block, the horizontal or vertical gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the horizontal or vertical displacement vector per unit time of the current pixel. The video encoding device 150 can perform block-level motion compensation and pixel group-level motion compensation for the current block and obtain a predicted block for the current block.

[0186] In this case, the horizontal or vertical displacement vector per unit time of pixels in the current block, including pixels adjacent to the current block's boundary, can be determined using values ​​related to the reference pixels contained within the first and second reference blocks, without using values ​​stored for pixels located outside the boundaries of the first and second reference blocks. The values ​​related to the reference pixels contained within the first and second reference blocks are also the pixel values ​​or gradient values ​​of the reference pixels.

[0187] Figure 1E illustrates block diagrams of the video decoding unit 600 according to various embodiments.

[0188] The video decoding unit 600, according to various embodiments, performs the tasks required to encode video data in the video decoding unit (not shown) of the video decoding device 100.

[0189] Referring to Figure 1E, the entropy decoding unit 615 purges the encoded video data to be decoded and the encoding information necessary for decoding from the bitstream 605. The encoded video data consists of quantized conversion coefficients, and the inverse quantization unit 620 and the inverse conversion unit 625 reconstruct the registered data from the quantized conversion coefficients.

[0190] The intra prediction unit 640 performs intra predictions block by block. The inter prediction unit 635 performs inter predictions block by block using the reference video acquired by the restored picture buffer 630. The inter prediction unit 635 in Figure 1E corresponds to the inter prediction unit 110 in Figure 1A.

[0191] By adding the prediction data for each block generated by the intra-prediction unit 640 or the inter-prediction unit 635 to the current data, the spatial region data for the block in the current video 605 is restored. The deblocking unit 645 and the SAO execution unit 650 then perform loop filtering on the restored spatial region data and output the filtered restored video 660. The restored video stored in the restored picture buffer 630 is also output as a reference video.

[0192] In the decoding unit (not shown) of the video decoding device 100, various step-by-step operations of the video decoding unit 600 are performed in blocks according to different embodiments in order to decode the video data.

[0193] Figure 1F illustrates block diagrams of the video encoding unit according to various embodiments.

[0194] The video encoding unit 700, according to various embodiments, performs the tasks required to encode video data in the video encoding unit (not shown) of the video encoding device 150.

[0195] In other words, the intra prediction unit 720 performs intra prediction block by block in the current video 705, and the inter prediction unit 715 performs inter prediction block by block using the current video 705 and the reference video acquired by the restored picture buffer 710. Here, the inter prediction unit 715 in Figure 1E corresponds to the inter prediction unit 160 in Figure 1C.

[0196] The prediction data for each block output from the intra-prediction unit 720 or the inter-prediction unit 715 is subtracted from the data for the currently encoded block of the video 705 to generate residency data. The conversion unit 725 and the quantization unit 730 perform conversion and quantization on the residency data and can output quantized conversion coefficients for each block. The inverse quantization unit 745 and the inverse conversion unit 750 perform inverse quantization and inverse conversion on the quantized conversion coefficients to restore the spatial domain residency data. The restored spatial domain residency data is added to the prediction data for each block output from the intra-prediction unit 720 or the inter-prediction unit 715 to restore the spatial domain data for the block of the current video 705. The deblocking unit 755 and the SAO execution unit 760 perform in-loop filtering on the restored spatial domain data to generate the filtered restored video. The generated restored video is stored in the restored picture buffer 710. The restored image stored in the restored picture buffer 710 is also used as a reference image for interpretation of other images. The entropy coding unit 735 entropy codes the quantized transformation coefficients, and the entropy coded coefficients are output as a bitstream 740.

[0197] In order for the video encoding unit 700 according to various embodiments to be applied to the video encoding device 150, the step-by-step operations of the video encoding unit 700 according to various embodiments are also performed in blocks.

[0198] Figure 2 is a reference diagram illustrating the process of block-based bidirectional motion prediction and compensation according to one embodiment. Referring to Figure 2, the video encoding device 150 performs bidirectional motion prediction by searching for the region in the first reference picture 210 and the second reference picture 220 that is most similar to the current block 201 being encoded in the current picture 200. Here, the first reference picture 210 is assumed to be a picture prior to the current picture 200, and the second reference picture 220 is assumed to be a picture after the current picture 200. Based on the bidirectional motion prediction results, the video encoding device 150 determines the first corresponding region 212 in the first reference picture 210 that is most similar to the current block 201, and the second corresponding region 222 in the second reference picture 220 that is most similar to the current block 201. Here, the first corresponding region and the second corresponding region also become the reference regions of the current block.

[0199] Then, the video encoding device 150 determines a first motion vector MV1 based on the positional difference between block 211, which is in the same position as the current block 201 of the first reference picture 210, and the first corresponding region 212, and determines a second motion vector MV2 based on the positional difference between block 221, which is in the same position as the current block 201 of the second reference picture 220, and the second corresponding region 222.

[0200] The video encoding device 150 uses the first motion vector MV1 and the second motion vector MV2 to perform block-level bidirectional motion compensation for the current block 201.

[0201] For example, if we let P0(i,j) be the pixel value at (i,j) (where i and j are integers) in the first reference picture 210, P1(i,j) be the pixel value at (i,j) in the second reference picture 220, MV1=(MVx1,MVy1), and MV2=(MVx2,MVy2), then the block-level bidirectional motion compensation value P_BiPredBlock(i,j) for the pixel at position (i,j) in block 201 can also be calculated using the following formula: P_BiPredBlock(i,j)={P0(i+MVx1,j+MVy1)+P1(i+MVx2,j+MVy2)} / 2. In this way, the video encoding device 150 can use the average value or weighted sum of the pixels in the first corresponding region 212 and the second corresponding region 222 indicated by the first motion vector MV1 and the second motion vector MV2 to perform block-level motion compensation for the current block 201 and generate block-level motion compensation values.

[0202] Figures 3A to 3C are reference diagrams illustrating the process of performing motion compensation on a pixel group basis according to one embodiment.

[0203] In Figure 3A, the first corresponding region 310 and the second corresponding region 320 correspond to the first corresponding region 212 and the second corresponding region 222 in Figure 2, respectively, and are assumed to have been shifted using bidirectional motion vectors (MV1, MV2) so that they overlap with block 300.

[0204] Furthermore, we define P(i,j) as the bidirectionally predicted pixel at position (i,j) (where i and j are integers) in block 300, P0(i,j) as the first reference pixel value of the first reference picture corresponding to the bidirectionally predicted pixel P(i,j) in block 300, and P1(i,j) as the second reference pixel value of the second reference picture corresponding to the bidirectionally predicted pixel P(i,j) in block 300.

[0205] Alternatively, the first reference pixel value P0(i,j) is the pixel corresponding to pixel P(i,j) of the current block 300 determined by the bidirectional motion vector MV1 indicating the first reference picture, and the pixel value P1(i,j) of the second reference pixel is the pixel corresponding to pixel P(i,j) of the current block 300 determined by the bidirectional motion vector MV2 indicating the second reference picture.

[0206] Also, the horizontal gradient value of the first reference pixel is

[0207]

Number

[0208]

Number

[0209]

Number

[0210]

Number

[0211] Assuming there is small motion determined by the video sequence, the pixel in the first corresponding region 310 of the first reference picture that is most similar to the current pixel P(i,j), which is compensated for bidirectional motion on a pixel group basis, is not the first reference pixel P0(i,j), but rather the first displacement reference pixel PA, which is the first reference pixel P0(i,j) moved by a predetermined displacement vector. As mentioned above, since we assumed there is motion determined by the video sequence, in the second corresponding region 320 of the second reference picture, the pixel most similar to the current pixel P(i,j) can be estimated to be the second displacement reference pixel PB, which is the second reference pixel P1(i,j) moved by a predetermined displacement vector.

[0212] The displacement vector is also composed of the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction mentioned above. Therefore, the pixel group-based motion compensation unit 165 calculates the x-axis direction displacement vector Vx and the y-axis direction displacement vector Vy that constitute such a displacement vector and uses them to perform motion compensation on a pixel group basis.

[0213] Optical flow refers to the pattern of apparent motion of an object or surface induced by the relative movement between the observer (eye or video image acquisition device such as a camera) and the scene. In a video sequence, optical flow can also be represented by calculating the motion between frames acquired at arbitrary times t and t+△t. The pixel value located at (x,y) within the frame at time t is also defined as I(x,y,t). That is, I(x,y,t) is also a value that changes spatiotemporally. Differentiating I(x,y,t) with respect to time t yields the following equation (1).

[0214]

number

[0215]

number

[0216] The pixel group motion compensation unit 165 calculates the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction using equation (2), and performs motion compensation on a pixel group basis using these displacement vectors Vx and Vy. In equation (2), the pixel value I(x,y,t) is the value of the original signal, so using the value of the original signal as is would induce a lot of overhead during encoding. Therefore, the pixel group motion compensation unit 165 can calculate the displacement vectors Vx and Vy using equation (2) by using the pixels of the first reference picture and the second reference picture determined as a result of bidirectional motion prediction on a block basis. That is, the pixel group motion compensation unit 165 determines the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction so that Δ is minimized within a predetermined size window (Ωij) including surrounding pixels, centered on the current pixel P(i,j) whose bidirectional motion compensation is performed. The most desirable case is when Δ is 0. However, since there are no x-axis displacement vector Vx and y-axis displacement vector Vy that satisfy the condition Δ is 0 for all pixels within the window (Ωij), the x-axis displacement vector Vx and y-axis displacement vector Vy that minimize Δ are determined. The process of finding the displacement vectors Vx and Vy will be explained in detail with reference to Figure 8A.

[0217] To determine the predicted pixel value of the current pixel, a function P(t) related to t is also determined, as shown in equation (3) below.

[0218]

number

[0219] Assume that the time distance from the first reference picture (assuming that the first reference picture is at a position temporally prior to the current picture) to the current picture is τ0, and the time distance from the second reference picture (assuming that the second reference picture is at a position temporally subsequent to the current picture) to the current picture is τ1. Then, the reference pixel value in the first reference picture is the same as P(-τ0), and the reference pixel value in the second reference picture is the same as P(τ1). Hereinafter, for the convenience of calculation, assume that both τ0 and τ1 are the same as τ.

[0220] The coefficients of each order of P(t) are also determined by the following mathematical formula (4). Here, P0(i,j) means the pixel value at the (i,j) position of the first reference picture, and P1(i,j) means the pixel value at the (i,j) position of the second reference picture.

[0221]

Equation

[0222]

Equation

[0223]

Equation

[0224] For convenience as described above, we explained the process of determining the predicted pixel value of the current pixel when the temporal distance from the first reference picture to the current picture is τ, and the temporal distance from the second reference picture to the current picture is the same as τ. However, the temporal distance from the first reference picture to the current picture is τ0, and the temporal distance from the second reference picture to the current picture is τ1. In this case, the predicted pixel value P(0) of the current pixel can also be determined as shown in equation (7) below.

[0225]

number

[0226]

number

[0227] For example, as illustrated in Figure 3B, both the first reference picture containing the first corresponding region and the second reference picture containing the second corresponding region can be positioned in the display order, temporally, before the current picture containing the current block.

[0228] In that case, the predicted pixel value P(0) of the current pixel is also determined by equation (9), in which τ1, which represents the temporal distance difference between the second reference picture and the current picture, is replaced with -τ1 in equation (8) derived by referring to Figure 3A.

[0229]

number

[0230] In that case, the predicted pixel value of the current pixel is also determined by equation (10), in which τ0, which represents the temporal distance difference between the first reference picture and the current picture, is replaced with -τ0 in equation (8) derived by referring to Figure 3A.

[0231]

number

[0232] Figure 4 is a reference diagram illustrating the process of calculating horizontal and vertical gradient values ​​according to one embodiment. Referring to Figure 4, the horizontal gradient value of the first reference pixel P0(i,j)410 of the first reference picture is shown.

[0233]

number

[0234]

number

[0235]

number

[0236]

number

[0237]

number

[0238] According to one embodiment, instead of calculating the change in pixel value at decimal pixel positions and then calculating the gradient value, as shown in equation (11), a predetermined filter can be used to calculate the gradient value at each reference pixel. The filter coefficients of the predetermined filter are determined from the coefficients of an interpolation filter used to determine the pixel value at decimal pixel positions, taking into account the linearity of the filter.

[0239] Figure 5 is a reference diagram illustrating the process of calculating horizontal and vertical gradient values ​​according to another embodiment.

[0240] According to another embodiment, the gradient value is also determined by applying a predetermined filter to the pixels of the reference picture. Referring to Figure 5, the video decoder 100 centers on the reference pixel P0500 for which the horizontal gradient value is currently to be determined, and to the left of it, M Max 520 pixels, and |M on the right side Min | A predetermined filter can be applied to the pixels 510 to calculate the horizontal gradient value of P0500. The filter coefficients used in this case are M, which is used to determine the window size, as illustrated in Figures 7A to 7D. Max Integer pixels and M Min It is also determined by the α value, which indicates the interpolation position with integer pixels (fractional pixel position). As an example, referring to Figure 7A, the M that determines the window size Min is -2, and M Max When α is 3, and the reference pixel P0500 is about 1 / 4 of the way away, i.e., α = 1 / 4, the filter coefficients {4, -17, -36, 60, -15, 4} in the second row of Figure 7A are applied to the surrounding pixels P-2, P-1, P0, P1, P2, P3. In that case, the horizontal gradient value of reference pixel 500

[0241]

number

[0242]

number

[0243] Figures 6A and 6B are diagrams illustrating the process of determining horizontal and vertical gradient values ​​using a one-dimensional filter according to one embodiment.

[0244] Referring to Figure 6A, in the reference picture, filtering is performed on integer pixels using multiple one-dimensional filters to determine the horizontal gradient value of the reference pixels. Pixel group motion compensation is additional motion compensation performed after block motion compensation. Therefore, in the block motion compensation process, the reference position of the current block's reference block indicated by the motion vector is also a decimal pixel position, and pixel group motion compensation is performed on the reference pixels within the reference block at decimal pixel positions. Thus, filtering is performed considering the determination of the gradient value of pixels at decimal pixel positions.

[0245] Referring to Figure 6A, the video decoder 100 can first use a first one-dimensional filter to filter pixels located horizontally or vertically from the surrounding integer pixels of the reference pixel within the reference picture. Similarly, the video decoder 100 can use the first one-dimensional filter to filter adjacent integer pixels located in different rows or columns from the reference pixel. The video decoder 100 can then use a second one-dimensional filter to filter the values ​​generated by the above filtering to generate a horizontal gradient value for the reference pixel.

[0246] For example, when the position of the reference pixel is at the position of a fractional pixel of (x + α, y + β) (x and y are integers, and α and β are fractions), for the horizontal integer pixels (x, y) and (x - 1, y), (x + 1, y), …, (x + M Min , y), (x + M Max , y) (M Min , M Max are all integers), a one-dimensional vertical interpolation filter is used, and filtering is performed as in the following formula (12).

[0247]

Equation

[0248] That is, the first one-dimensional filter is also an interpolation filter for determining the fractional pixel value in the vertical direction. offset1 is an offset for preventing rounding errors, and shift1 means the number of inverse scaling bits. Temp[i, j + β] means the pixel value at the fractional pixel position (i, j + β). Temp[i’, j + β] (i’ is an integer from i + M min to i + M max up to) can also be determined by substituting i with i’ in formula (12).

[0249] Next, the video decoder 100 can perform filtering on the pixel value at the fractional pixel position (i, j + β) and the pixel value at the fractional pixel position (i’, j + β) using a second one-dimensional filter.

[0250]

Equation

[0251] In other words, according to equation (13), the video decoder 100 uses a gradient filter (gradFilterα) to filter the pixel value (Temp[i,j+β]) at (i,j+β) and the pixel value (Temp[i',j+β]) located vertically from the pixel position (i,j+β), thereby obtaining the horizontal gradient value at (i+α,j+β).

[0252]

number

[0253] Earlier, we described a method in which an interpolation filter is applied first, followed by a gradient filter to determine the horizontal gradient value. However, this method is not limiting; it is also possible to apply a gradient filter first, followed by an interpolation filter to determine the horizontal gradient value. Below, we will describe an embodiment in which a gradient filter is applied first, followed by an interpolation filter to determine the horizontal gradient value.

[0254] For example, if the position of the reference pixel is a decimal pixel position on a (x+α,y+β) (where x and y are integers and α and β are decimals), then the horizontal integer pixels (x,y) and (x-1,y), (x+1,y), ..., (x+M Min ,y),(x+M Max, y)(M Min , M Mmax is an integer), the first one-dimensional filter is used, and filtering is performed as shown in the following formula (14).

[0255]

Equation

[0256] That is, the first one-dimensional filter is also an interpolation filter for determining the horizontal gradient value of the pixel at the position where the horizontal component of the pixel position is fractional. offset3 is an offset for preventing rounding errors, and shift3 means the inverse scaling bit number. Temp[i + α, j] means the horizontal gradient value at the pixel position (i + α, j). Temp[i + α, j’] (j’ is an integer from j + M min to j + M max is also determined by substituting j with j’ in formula (14).

[0257] Next, the video decoding device 100 can perform filtering as shown in the following formula (15) using the second one-dimensional filter on the horizontal gradient value at the pixel position (i + α, j) and the horizontal gradient value at the pixel position (i + α, j’).

[0258]

Equation

[0259] In other words, according to equation (15), the video decoder 100 filters the horizontal gradient value (Temp[i+α,j]) at (i+α,j) and the horizontal gradient value (Temp[i+α,j']) of a pixel located vertically from the pixel position (i+α,j) using a gradient filter (fracFilterβ), thereby obtaining the horizontal gradient value at (i+α,j+β).

[0260]

number

[0261] Referring to Figure 6B, in the reference picture, filtering is performed on integer pixels using multiple one-dimensional filters to determine the vertical gradient value of the reference pixels. Pixel group motion compensation is additional motion compensation performed after block motion compensation. Therefore, in the block motion compensation process, the reference position of the current block's reference block indicated by the motion vector is also a decimal pixel position, and pixel group motion compensation is also performed on the reference pixels within the reference block at decimal pixel positions. Consequently, filtering is also performed to take into consideration the determination of the gradient value of pixels at decimal pixel positions.

[0262] Referring to Figure 6B, the video decoder 100 can first use a first one-dimensional filter to filter pixels located horizontally or vertically from integer pixels surrounding the reference pixel within the reference picture. Similarly, the video decoder 100 can use the first one-dimensional filter to filter adjacent pixels located in different columns or rows from the reference pixel. The video decoder 100 can then use a second one-dimensional filter to filter the values ​​generated by the first filtering to generate a vertical gradient value for the reference pixel.

[0263] For example, if the position of the reference pixel is a decimal pixel position on a (x+α, y+β) (where x and y are integers and α and β are decimals), then the horizontal integer pixels (x, y) and (x-1, y-1), (x+1, y+1), ..., (x+M Min ,y+M Min ),(x+M Max ,y+M max A first-dimensional filter is applied to (MMin and MMmax, both integers), and filtering is performed as shown in the following equation (16).

[0264]

number

[0265] In other words, the first one-dimensional filter is also an interpolation filter for determining the pixel value at a fractional pixel position α in the horizontal direction. offset5 is an offset to prevent rounding errors, and shift5 represents the number of inverse scaling bits.

[0266] Temp[i+α,j] represents the pixel value at a decimal pixel position (i+α,j). Temp[i+α,j'](j' is j+M with j removed) min From j+M max The integer up to 1 is also determined by formula (16), by substituting j with j'.

[0267] Next, the video decoder 100 can filter the pixel value at pixel position (i+α,j) and the pixel value at pixel position (i+α,j') using a second one-dimensional filter, as shown in the following equation (17).

[0268]

number

[0269] In other words, according to equation (17), the video decoder 100 applies a gradient filter (gradFilter) to the pixel value (Temp[i+α,j]) at (i+α,j) and the pixel value (Temp[i+α,j']) located perpendicular to the pixel position (i+α,j). β By filtering using ), the vertical gradient value at (i+α, j+β)

[0270]

number

[0271] Earlier, we described a method in which an interpolation filter is applied first, followed by a gradient filter to determine the vertical gradient value. However, this method is not limiting; a gradient filter can also be applied first, followed by an interpolation filter to determine the horizontal gradient value. Below, we will describe an embodiment in which a gradient filter is applied first, followed by an interpolation filter to determine the vertical gradient value.

[0272] For example, if the position of the reference pixel is a decimal pixel position on a (x+α,y+β) (where x and y are integers and α and β are decimals), then the vertical integer pixels (x,y) and (x,y-1), (x,y+1), ..., (x,y+M Min ),(x,y+M max )(M Min M max A first-dimensional filter is used on (where all are integers), and filtering is performed as shown in equation (18) below.

[0273]

number

[0274] In other words, the first one-dimensional filter is also an interpolation filter for determining the vertical gradient value of pixels at positions where the vertical component of the pixel position is a decimal. offset7 is an offset to prevent rounding errors, and shift7 represents the number of inverse scaling bits.

[0275] Temp[i,j+β] represents the vertical gradient value at pixel position (i,j+β). Temp[i',j+β](i' is i+M excluding i) min From i+M max The integers up to 1 are also determined by formula (18), where i is replaced by i'.

[0276] Next, the video decoder 100 can filter the vertical gradient value at pixel position (i, j+β) and the vertical gradient value at pixel position (i', j+β) using a second one-dimensional filter, as shown in the following equation (19).

[0277]

number

[0278] In other words, according to equation (19), the video decoder 100 applies an interpolation filter (fracFilter) to the vertical gradient value (Temp[i,j+β]) at (i,j+β) and the vertical gradient value (Temp[i',j+β)]) of the pixel located horizontally from the pixel position (i,j+β). α By filtering using ), the vertical gradient value at (i+α, j+β)

[0279]

number

[0280] In one embodiment, the video decoding device 100 determines the horizontal and vertical gradient values ​​at (i+α, j+β) by a combination of the various filters described above. For example, to determine the horizontal gradient value, an interpolation filter for determining the vertical pixel value is used in the first one-dimensional filter, and a gradient filter for the horizontal gradient value is used in the second one-dimensional filter. To determine the vertical gradient value, a gradient filter for determining the vertical gradient value is used in the first one-dimensional filter, and an interpolation filter for determining the horizontal pixel value is used in the second one-dimensional filter.

[0281] Figures 7A to 7E are tables showing the filter coefficients of a filter used in one embodiment to determine the pixel value at decimal pixel positions in decimal pixel units, as well as the horizontal and vertical gradient values.

[0282] Figures 7A and 7B are tables showing the filter coefficients of a filter used to determine the horizontal or vertical gradient value at fractional pixel positions in units of 1 / 4 pel (pel).

[0283] As mentioned above, one-dimensional gradient filters and one-dimensional interpolation filters are also used to determine horizontal or vertical gradient values. Figure 7A illustrates the filter coefficients of a one-dimensional gradient filter. In this case, a 6-tap gradient filter is also used. The gradient filter coefficients are also scaled coefficients by approximately 2^4. min This represents the difference between the position of the central integer pixel and the position of the central integer pixel, where M is the furthest pixel among the negative integer pixels applied to the filter. maxThis represents the difference between the position of the central integer pixel and the position of the furthest integer pixel in the positive direction to which the filter is applied, relative to the central integer pixel. For example, the gradient filter coefficients for finding the horizontal gradient value of a pixel with a decimal pixel position α = 1 / 4 in the horizontal direction are {4, -17, -36, 60, -15, -4}. The gradient filter coefficients for finding the horizontal gradient values ​​of pixels with decimal pixel positions α = 0, 1 / 2, 3 / 4 in the horizontal direction are also determined by referring to Figure 7A.

[0284] Referring to Figure 7B, the filter coefficients of a one-dimensional interpolation filter are illustrated. In this case, a 6-tap filter is also used. The filter coefficients of the interpolation filter are also coefficients scaled by approximately 2^6. min This represents the difference between the position of the central integer pixel and the position of the central integer pixel, where M is the furthest pixel among the negative integer pixels applied to the filter. max This represents the difference between the position of the central integer pixel and the position of the central integer pixel, specifically the position of the furthest integer pixel in the positive direction applied to the filter, relative to the central integer pixel.

[0285] Figure 7C is a table showing the filter coefficients of a one-dimensional interpolation filter used to determine pixel values ​​at fractional pixel positions in units of 1 / 4 per unit.

[0286] As mentioned above, two identical one-dimensional interpolation filters are also used horizontally and vertically to determine the pixel value at decimal pixel positions.

[0287] Referring to Figure 7C, the filter coefficients of the one-dimensional interpolation filter are illustrated. In this case, the one-dimensional interpolation filter is also a 6-tap filter. The gradient filter coefficients are also scaled by approximately 2^6. min This represents the difference between the position of the central integer pixel and the position of the central integer pixel, where M is the furthest pixel among the negative integer pixels applied to the filter.max This represents the difference between the position of the central integer pixel and the position of the central integer pixel, specifically the position of the furthest integer pixel in the positive direction applied to the filter, relative to the central integer pixel.

[0288] Figure 7D is a table showing the filter coefficients of the filters used to determine the horizontal or vertical gradient values ​​at fractional pixel positions in units of 1 / 16 per unit.

[0289] As mentioned above, one-dimensional gradient filters and one-dimensional interpolation filters are also used to determine horizontal or vertical gradient values. Figure 7D illustrates the filter coefficients of a one-dimensional gradient filter. In this case, a 6-tap gradient filter is also used. The gradient filter coefficients are also scaled by approximately 2^4. For example, the gradient filter coefficients for determining the horizontal gradient value of a pixel where the decimal pixel position α is 1 / 16 are {8, -32, -13, 50, -18, 5}. The gradient filter coefficients for determining the horizontal gradient values ​​of pixels where the decimal pixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, and 1 / 2 can also be determined using Figure 7D. On the other hand, the gradient filter coefficients for determining the horizontal gradient value of pixels with decimal pixel positions α at 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, and 15 / 16 can also be determined using the symmetry of the filter coefficients based on α=1 / 2. That is, the filter coefficients for the left decimal pixel positions, as disclosed in Figure 7D, can be used, and the filter coefficients for the right decimal pixel positions can be determined using the remaining α=1 / 2 reference. For example, the filter coefficient at α=15 / 16 can also be determined using the filter coefficients {8,-32,-13,50,-18,5} at α=1 / 16, which is symmetrical with respect to α=1 / 2. In other words, the filter coefficients at α=15 / 16 can also be determined by reversing the order of the filter coefficients {8,-32,-13,50,-18,5}, resulting in {5,-18,50,-13,-32,8}.

[0290] Referring to Figure 7E, the filter coefficients of a one-dimensional interpolation filter are illustrated. In this case, a 6-tap filter is also used for the interpolation filter. The filter coefficients of the interpolation filter are also coefficients scaled by approximately 2^6. For example, the one-dimensional interpolation filter coefficients for finding the horizontal pixel value of a pixel whose decimal pixel position α is 1 / 16 in the horizontal direction are {1, -3, 64, 4, -2, 0}. The interpolation filter coefficients for finding the horizontal pixel value of a pixel whose decimal pixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, 1 / 2 in the horizontal direction can also be determined using Figure 7E. On the other hand, the interpolation filter coefficients for finding the horizontal pixel value of a pixel whose decimal pixel position α is 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, 15 / 16 in the horizontal direction can also be determined by utilizing the symmetry of the filter coefficients based on α = 1 / 2. In other words, the filter coefficients for the decimal pixel positions on the left, based on α=1 / 2 as disclosed in Figure 7E, are used, and the filter coefficients for the decimal pixel positions on the right, also based on α=1 / 2, are determined. For example, the filter coefficients at α=15 / 16 can also be determined using the filter coefficients {1,-3,64,4,-2,0} at α=1 / 16, which is the symmetrical position based on α=1 / 2. In other words, the filter coefficients at α=15 / 16 can also be determined by reversing the order of the filter coefficients {1,-3,64,4,-2,0} to {0,-2,4,64,-3,1}.

[0291] Figure 8A is a reference diagram illustrating the process of determining the horizontal and vertical displacement vectors related to a pixel according to one embodiment.

[0292] Referring to Figure 8A, a window of a predetermined size (Ωij)800 has a size of (2M+1)*(2N+1) (where M and N are integers) centered on the bidirectionally predicted pixel P(i,j) in the current block.

[0293] The bidirectionally predicted pixel of the current block in the window is P(i',j') (where iM≦i'≦i+M and jN≦j'≦j+N, then (i',j')∈Ωij), the pixel value of the first reference pixel of the first reference picture 810 corresponding to the bidirectionally predicted pixel P(i',j') of the current block is P0(i',j'), the pixel value of the second reference pixel of the second reference picture 820 corresponding to the bidirectionally predicted pixel P(i',j') of the current block is P1(i',j'), and the horizontal gradient value of the first reference pixel is

[0294]

number

[0295]

number

[0296]

number

[0297]

number

[0298]

number

[0299] The difference △i'j' between the first displacement-corresponding pixel PA' and the second displacement-corresponding pixel PB' can also be determined as shown in the following equation (21).

[0300]

number

[0301]

number

[0302]

number

[0303]

number

[0304]

number

[0305]

number

[0306]

number

[0307] By first minimizing in the horizontal direction and then in the vertical direction, a simplified solution to the above equation can be determined. That is, for example, if we assume that only the horizontal displacement vector is changed, we can assume Vy=0 in the first equation of equation (24), and therefore the equation: τVx=s3 / s1 can be determined.

[0308] Furthermore, by rearranging the second equation of equation (24) using the equation τVx=s3 / s1, we can determine that the equation τVy=(s6-τVx*S2) / s5.

[0309] Here, gradient value

[0310]

number

[0311] In the process of determining Vx(i,j) and Vy(i,j), adjustment parameters r and m are introduced to prevent multiplication by 0 or very small values.

[0312] For convenience, let us assume that Vx(i,j) and Vy(i,j) are in the opposite direction to that shown in Figure 3A. For example, Vx(i,j) and Vy(i,j) derived by equation (24) based on the directions of Vx(i,j) and Vy(i,j) shown in Figure 3A are identical in magnitude to Vx(i,j) and Vy(i,j) determined in the opposite direction to that shown in Figure 3A, with only a sign difference in their values.

[0313] The first displacement-corresponding pixel PA' and the second displacement-corresponding pixel PB' can also be determined as shown in equation (26) below. In this case, PA' and PB' can also be determined using the first linear term of local Taylor expansion.

[0314]

number

[0315]

number

[0316]

Number

[0317]

Number

[0318]

Number

[0319]

Number

[0320] Through the minimization, Vy is also determined as in the following mathematical formula (32).

[0321]

Number

[0322] At this time, s1, s2, s3, and s5 are also determined as in the following mathematical formula (33). s4 can have the same value as s2.

[0323]

Number

[0324]

Number

[0325] However, it is not limited to this; the values ​​of the adjustment parameters r,m and thBIO are also determined based on information about the adjustment parameters obtained from the bitstream. In this case, the information about the adjustment parameters is also included in the slice header, picture parameter set, sequence parameter set, and various forms of high-level syntax carriers.

[0326] Furthermore, the adjustment parameters are determined by the availability of bidirectional predictions at different time intervals. For example, thBIO when bidirectional predictions at different time intervals are available. diff However, if bidirectional predictions that are the same in time are available, thBIO same Larger, thBIO diff The size is thBIO same It's also twice the size.

[0327] Figure 8B is a reference diagram illustrating the process of determining the horizontal and vertical displacement vectors related to a pixel group according to one embodiment.

[0328] Referring to Figure 8B, a predetermined-size window (Ωij) 810 centers around a KxK pixel group 820 of multiple pixels that are not currently predicted bidirectionally in the block, and has a size of (2M+K+1)*(2N+K+1) (where M and N are integers).

[0329] In this case, the only difference from Figure 8A is that the window size is larger; otherwise, the same method can be used to determine the horizontal and vertical displacement vectors related to the pixel group.

[0330] Figure 8C is a reference diagram illustrating the process of determining the horizontal and vertical displacement vectors related to a pixel according to one embodiment.

[0331] Referring to Figure 8C, the video decoder 100 can determine the horizontal displacement vector and the vertical displacement vector for each pixel. Therefore, the displacement vector 835 per unit time for each pixel can be determined. At this time, the horizontal displacement vector Vx[i,j] and the vertical displacement vector Vy[i,j] of the displacement vector 835 for each pixel can also be determined by the following formula (35). Here, i and j represent the x and y coordinate components of the pixel. Here, σ1[i,j], σ2[i,j], σ3[i,j], σ5[i,j] and σ6 [i,j] are also s1, s2, s3, s5, and s6 of equation (33), respectively.

[0332]

number

[0333] Referring to Figure 8D, the video decoder 100 can determine σ1[i,j], σ2[i,j], σ3[i,j], σ5[i,j], and σ6[i,j] for each pixel included in each of the pixel groups 840, as shown in formula (35).

[0334] The video decoder 100 can determine the horizontal displacement vector Vx[i,j] related to the pixel group 840 using the pixel's σ1[i,j] and σ3[i,j] as shown in the following equation (36). Here, i and j represent the x and y components of the left upper end coordinate of the pixel group.

[0335]

number

[0336] The video decoder 100 can use the pixel σ2[i,j], σ5[i,j], σ6[i,j] and Vx[i,j] to determine the vertical displacement vector Vy[i,j] related to the pixel group 840 as shown in the following equation (37). Here, the horizontal displacement vector Vx[i,j] is also the value determined by equation (36).

[0337]

number

[0338] On the other hand, as previously explained in detail using Figure 8D, assuming that the pixel group has a 2x2 size, it is not limited to this, and the pixel group can have an LxL size (where L is an integer).

[0339] In this case, the size L of the pixel group is also determined by the following formula (38), where W and H represent the width and height of the current block, respectively.

[0340]

number

[0341] [Table 1] In other words, when the video decoding device 100 performs motion compensation on a pixel group basis, it can reduce the number of division operations by a value that is divided by L^2, which is a value corresponding to the size of the pixel group, and perform the division operations in this manner.

[0342] Figure 9A is a diagram illustrating the process of determining horizontal or vertical gradient values ​​by adding an offset and performing inverse scaling after filtering, according to one embodiment.

[0343] Referring to Figure 9A, the video decoder 100 can determine horizontal or vertical gradient values ​​by filtering pixels where a predetermined directional component is at an integer position using a first one-dimensional filter and a second one-dimensional filter. However, the value obtained by filtering using the first or second one-dimensional filter for pixels where a predetermined directional component is at an integer position may fall outside a predetermined range. This phenomenon is called an overflow phenomenon. The coefficients of the one-dimensional filter are determined as integers in order to perform integer arithmetic instead of inaccurate and complex decimal arithmetic. Because the coefficients of the one-dimensional filter are determined as integers, they are also scaled. If filtering is performed using the scaled coefficients of the one-dimensional filter, integer arithmetic becomes possible, but compared to the case where filtering is performed using an unscaled one-dimensional filter, the magnitude of the filtered value becomes larger, and an overflow phenomenon may occur. Therefore, in order to prevent this overflow phenomenon, inverse scaling is performed after filtering using the one-dimensional filter. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits. The number of inverse scaling bits is determined while maximizing the accuracy of the calculation, taking into account the maximum number of bits in the register for the filtering operation and the maximum number of bits in the temporary buffer for storing the filtering operation results. In particular, the number of inverse scaling bits is also determined based on the internal bit depth, the number of scaling bits for the interpolation filter, and the number of scaling bits for the gradient filter.

[0344] The following describes how, in order to determine the horizontal gradient value, a vertical interpolation filter is first used to filter pixels at integer positions to generate a vertical interpolation filtering value, and then a horizontal gradient filter is used to perform inverse scaling on the vertical interpolation filtering value during the filtering process.

[0345] According to the aforementioned formula (12), the video decoder 100 can first filter pixels at integer positions using a vertical interpolation filter in order to determine the horizontal gradient value. At this time, shift1 is also determined to be b-8. In this case, b is also the internal bit depth of the input video. Below, referring to Table 2, we will explain the register bit depth (Reg Bitdepth) and the temporary buffer bit depth (Temp Bitdepth) when actual inverse scaling is performed based on shift1.

[0346] [Table 2] At this time, the values ​​of the variables in Table 2 are also determined by the following formula (39).

[0347]

number

[0348] For example, assuming that the 1 / 4 Pell unit gradient filter fracFilter disclosed in Figure 7C is used, FilterSumPos is 88 and FilterSumNeg is -24.

[0349] The Ceiling(x) function is also a function that outputs the smallest integer that is greater than or equal to x for an error x. offset1 is an offset value added to the filtered value to prevent rounding errors that may occur during the descaling process using shift1, and offset1 can also be determined as 2^(shift1-1).

[0350] Referring to Table 2, when the internal bit depth b is 8, the register bit depth (RegBitdpeth) is 16; when the internal bit depth b is 9, the register bit depth is 17; and so on, when the internal bit depth b is 10, 11, 12, and 16, the register bit depths are 18, 19, and 24, respectively. If the register used for filtering is a 32-bit register, the bit depths of the registers in Table 2 will not exceed 32, and therefore, an overflow will not occur.

[0351] Similarly, if the internal bit depth b is 8, 9, 10, 11, 12, and 16, the temporary buffer's bit depth (TempBitDepth) is 16 in all cases. If the temporary buffer used to store filtered and descaled values ​​is a 16-bit buffer, the temporary buffer's bit depth in Table 2 is 16, and since none of them exceed 16, no overflow phenomenon occurs.

[0352] According to equation (12), the video decoder 100 first uses a vertical interpolation filter to filter pixels at integer positions to determine the horizontal gradient value, thereby generating a vertical interpolation filtering value. Then, using equation (13), it uses a horizontal gradient filter to filter the vertical interpolation filtering value. In this case, shift2 is also determined as p + q - shift1. Here, p represents the number of bits scaled for the interpolation filter including the filter coefficients shown in Figure 7C, and q represents the number of bits scaled for the gradient filter including the filter coefficients shown in Figure 7A. For example, p is 6 and q is 4, and therefore shift2 = 18 - b.

[0353] The reason shift2 is determined as described above is that, since the final filtered result is the same whether the filter coefficients are upscaled or not, shift1 + shift2, which is the sum of the bits that are descaled, must be the same as the sum of the bits that are upscaled for the filter (p + q).

[0354] Below, referring to Table 3, we will explain the register bit depth (Reg Bitdepth) and the temporary buffer bit depth (Temp Bitdepth) when actual inversion scaling is performed based on the aforementioned shift2.

[0355] [Table 3] At this time, the values ​​of the variables in Table 3 are also determined by the following formula (40).

[0356]

number

[0357] offset2 is an offset value added to the filtered value to prevent rounding errors that may occur during the descaling process using shift2, and offset2 can also be determined as 2^(shift2-1).

[0358] While shift1 and shift2 can be determined as described above, they are not limited to that method. They can also be determined in various ways such that their sum equals the sum of the scaling bits applied to the filter. In this case, the shift1 and shift2 values ​​are determined assuming that no overflow occurs. shift1 and shift2 can also be determined based on the internal bit depth of the input video and the scaling bits applied to the filter.

[0359] However, it is not always necessary to determine shift1 and shift2 such that their sum is equal to the sum of the scaling bits for the filter. For example, shift1 can be determined as d-8, while shift2 can be determined as a fixed number.

[0360] If shift1 is the same as before, and shift2 is the solid integer 7, then OutMax, OutMin, and Temp Bitdepth will also be different in Table 3 mentioned earlier. Below, we will refer to Table 4 and explain the temporary buffer bit depth (Temp Bitdepth).

[0361] [Table 4] Unlike Table 3, in Table 4, the temporary buffer bit depth is the same as 16 in all cases b. If we assume that a 16-bit temporary buffer is used to save the result data, then because the temporary buffer bit depth is less than 16, no overflow occurs for the internal bit depth of any input video. On the other hand, referring to Table 3, if the internal bit depth of the input video is 12 or 16, then if we assume that a 16-bit temporary buffer is used to save the result data, then because the temporary buffer bit depth is greater than 16, an overflow phenomenon will occur.

[0362] If shift2 is a fixed number, the result obtained when filtering is performed without using scaled filter coefficients will differ from the result obtained when filtering is performed using scaled filter coefficients and then inverse scaling is performed. In that case, it will be easy for a person skilled in the art to understand that additional inverse scaling is necessary.

[0363] Earlier, we described how, in order to determine the horizontal gradient value, a vertical interpolation filter is first used to filter pixels at integer positions to generate a vertical interpolation filtered value, and then a horizontal gradient filter is used to perform inverse scaling on the vertical interpolation filtered value during the filtering process. However, we are not limited to this, and it will be easy for those skilled in the art to understand that inverse scaling can be similarly performed when filtering is performed on pixels with integer directional components in order to determine the horizontal and vertical gradient values ​​using various combinations of one-dimensional filters.

[0364] Figure 9B is a diagram illustrating the process of determining horizontal or vertical gradient values ​​by adding an offset and performing inverse scaling after filtering, according to another embodiment.

[0365] Referring to Figure 9B, the video decoder 100 can take fractional and integer pixels of a reference picture as input and perform filtering. Here, it is assumed that the fractional pixels of the reference picture are values ​​determined by applying horizontal and vertical one-dimensional filters to the integer pixels of the reference picture.

[0366] The video decoder 100 can determine a horizontal or vertical gradient value by filtering pixels with decimal position components in a predetermined direction and integer pixels using a horizontal or vertical one-dimensional filter. However, the value obtained by filtering using the first one-dimensional filter for pixels with decimal position components in a predetermined direction and integer pixels may fall outside a predetermined range. This phenomenon is called an overflow phenomenon. The coefficients of the one-dimensional filter are determined to be integers in order to perform integer arithmetic instead of inaccurate and complex decimal arithmetic. The coefficients of the one-dimensional filter are also scaled in order to be determined as integers. If filtering is performed using the scaled coefficients of the one-dimensional filter, integer arithmetic becomes possible, but compared to when filtering is performed using an unscaled one-dimensional filter, the magnitude of the filtered value becomes larger, and an overflow phenomenon may occur. Therefore, in order to prevent an overflow phenomenon, inverse scaling is performed after filtering using a one-dimensional filter. In this case, the inverse scaling may include bit shifting to the right by the number of inverse scaling bits (shift1). The number of inverse scaling bits is determined while maximizing the accuracy of the calculation, taking into account the maximum number of bits in the register for the filtering operation and the maximum number of bits in the temporary buffer for storing the filtering operation results. In particular, the number of inverse scaling bits is also determined based on the internal bit depth and the scaling bits for the gradient filter.

[0367] Figure 9C is a diagram illustrating the range required to determine the horizontal and vertical displacement vectors in the process of performing pixel-level motion compensation on the current block.

[0368] Referring to Figure 9C, in the process of performing pixel-level motion compensation for the reference block 910 corresponding to the current block, the video decoder 100 can use the window 920 around pixel 915 located at the upper left end of the reference block 910 to determine the horizontal displacement vector per unit time and the vertical displacement vector per unit time at pixel 915. At this time, the pixel value and gradient value of pixels located outside the reference block 910 can be used to determine the horizontal or vertical displacement vector per unit time. Similarly, in the process of determining the horizontal and vertical displacement vectors for pixels located at the boundary of the reference block 910, the video decoder 100 will determine the pixel value and gradient value of pixels located outside the reference block 910. Therefore, the video decoder 100 can use block 925, which is larger than the reference block 910, to determine the horizontal displacement vector and the vertical displacement vector per unit time. For example, if the current block size is A x B and the pixel-level window size is (2M + 1) x (2N + 1), then the size of the range for determining the horizontal and vertical displacement vectors is also (A + 2M) x (B + 2N).

[0369] Figures 9D and 9E are diagrams illustrating the range of regions used in the process of performing pixel-level motion compensation according to various embodiments.

[0370] Referring to Figure 9D, in the process of performing pixel-level motion compensation, the video decoder 100 can determine the horizontal displacement vector and the vertical displacement vector per unit time for each pixel included in the reference block 930, based on the block 935, which is an expanded range determined by the size of the window of pixels located at the boundary of the reference block 930. However, in the process of determining the horizontal and vertical displacement vectors per unit time, the video decoder 100 needs the pixel value and gradient value of the pixels located in block 935, and in this case, an interpolation filter or gradient filter can be used to obtain the aforementioned pixel value and gradient value. In the process of using an interpolation filter or gradient filter on the boundary pixels of block 935, the pixel values ​​of surrounding pixels can be used, and therefore, pixels located outside the block boundary can be used. Accordingly, pixel-level motion compensation can also be performed using a block 940, which is an expanded range obtained by subtracting approximately 1 from the number of taps of the interpolation filter or gradient filter. Therefore, if the block size is NxN, the pixel-by-pixel window size is (2M+1)x(2M+1), and the interpolation filter or gradient filter length is T, then it is also (N+2M+T-1)x(N+2M+T-1).

[0371] Referring to Figure 9E, in the process of performing pixel-level motion compensation, the video decoder 100 does not expand the reference block based on the size of the window of pixels located at the boundary of the reference block 945. Instead, it uses the pixel value and gradient value of pixels located within the reference block 945 to determine the horizontal displacement vector and the vertical displacement vector per unit time for each pixel. Specifically, the process by which the video decoder 100 determines the horizontal displacement vector per unit time and the vertical displacement vector per unit time without expanding the reference block will be explained with reference to Figure 9E. However, in order to determine the pixel value and gradient value of a pixel, the interpolation filter or gradient filter of the reference block 945 is used, and pixel-level motion compensation is also performed using the expanded block 950. Therefore, if the size of the block is NxN, the pixel-level window size is (2M+1)x(2M+1), and the interpolation filter length or gradient filter length is T, then it is also (N+T-1)x(N+T-1).

[0372] Figure 9F is a diagram illustrating the process for determining the horizontal and vertical displacement vectors without extending the reference block.

[0373] Referring to Figure 9F, for pixels located outside the boundary of reference block 955, the video decoder 100 adjusts the position of the pixel to the position of the nearest available pixel located within the boundary of reference block 955, and the pixel value and gradient value of the pixel located outside the boundary are determined to be the same as the pixel value and gradient value of the nearest available pixel. At this time, the video decoder 100 determines the position of the pixel located outside the boundary of reference block 955 using the formula:

[0374]

number

[0375]

number

[0376] Here, i' is the x-coordinate of the pixel, j' is the y-coordinate of the pixel, and H and W represent the height and width of the reference block. We assume the top-left corner of the reference block is at (0,0). If the top-left corner of the reference block is at (xP,yP), then the position of the final pixel is also (i'+xP,j'+yP).

[0377] Referring to Figure 9D, in block 935, which is expanded by the size of the pixel-by-pixel window, the positions of pixels located outside the boundary of reference block 930 are adjusted to the positions of adjacent pixels inside the boundary of reference block 930. As shown in Figure 9E, the video decoder 100 can determine the horizontal displacement vector and the vertical displacement vector per unit time for each pixel within reference block 945 using the pixel values ​​and gradient values ​​of reference block 945.

[0378] Therefore, the video decoder 100 does not expand the reference block 945 according to the pixel-by-pixel window size, and by performing pixel-level motion compensation, the number of memory accesses for pixel value referencing is reduced, the number of multiplication operations is reduced, and the complexity of the operation is also reduced.

[0379] When the video decoder 100 performs block-level motion compensation (as when operating according to the HEVC standard), when it performs pixel-level motion compensation along with block expansion based on the window size, or when it performs pixel-level motion compensation without block expansion, the number of memory accesses and multiplication operations performed by the video decoder 100 are as shown in Table 5 below, corresponding to the number of memory accesses and multiplication operations involved in each case. At this time, we assume that the signal (interpolation) filter length is 8, the gradient filter length is 6, the block size is NxN, and the pixel-level window size 2M+1 is 5.

[0380] [Table 5] In block-level motion compensation according to the HEVC standard, an 8-tap interpolation filter is used for each sample, requiring 8 peripheral samples. Therefore, if the size of the reference block is NxN, 8-tap interpolation requires (N+7)x(N+7) reference samples. Since it is bidirectional motion prediction compensation, two reference blocks are used. Thus, block-level motion compensation according to the HEVC standard requires 2*(N+7)x(N+7) memory accesses, as shown in Table 4. When performing pixel-level motion compensation along with block expansion, if M=2, then using an 8-tap interpolation filter or gradient filter for the expanded (N+4)x(N+4) size block requires (N+4+7)x(N+4+7) reference samples. Since it is bidirectional motion prediction compensation, two reference blocks are used. Therefore, ultimately, pixel-level motion compensation performed along with block expansion requires 2*(N+4+7)x(N+4+7) memory accesses, as shown in Table 5.

[0381] When performing pixel-level motion compensation without block expansion, (N+7)x(N+7) reference samples are required, similar to block-level motion compensation using the HEVC standard, because there is no block expansion. Furthermore, since it is bidirectional motion prediction compensation, two reference blocks are used. Therefore, pixel-level motion compensation without block expansion requires 2*(N+7)x(N+7) memory accesses, as shown in Table 5.

[0382] On the other hand, in block-level motion compensation according to the HEVC standard, an 8-tap interpolation filter is used for each sample, so the number of samples required for the first horizontal interpolation is (N+7)xN samples. The number of samples required for the second vertical interpolation is NxN samples. On the other hand, the number of multiplication operations required per 8-tap interpolation filter is 8, and since it is bidirectional motion prediction compensation, two reference blocks are used, so in block-level motion compensation according to the HEVC standard, as shown in Table 5, 2*8*{(N+7)*N+N*N) multiplication operations are required.

[0383] When performing pixel-level motion compensation along with block expansion, the block size is expanded to perform pixel-level motion compensation. Therefore, to use an 8-tap interpolation filter on the expanded (N+4)x(N+4) size block and determine the pixel value at a decimal pixel position, a total of 2*8*{(N+4+7)x(N+4)+(N+4)x(N+4)} multiplication operations are required, as shown in Table 5.

[0384] On the other hand, when performing pixel-level motion compensation along with block expansion, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine the horizontal or vertical gradient value. Because the block size is expanded, determining the gradient value for the expanded (N+4)x(N+4) block using the 6-tap interpolation filter and gradient filter requires a total of 2*6*{(N+4+5)x(N+4)+(N+4)x(N+4)}*2 multiplication operations, as shown in Table 5.

[0385] When performing pixel-level motion compensation without block expansion, (N+7)x(N+7) reference samples are required, similar to block-level motion compensation using the HEVC standard, because there is no block expansion. Furthermore, since it is bidirectional motion prediction compensation, two reference blocks are used. Therefore, in pixel-level motion compensation performed without block expansion, an 8-tap interpolation filter is used for an NxN size block, and 2*8*{(N+7)xN+NxN} multiplication operations are required, as shown in Table 5, to determine the pixel value at a fractional pixel position.

[0386] On the other hand, when performing pixel-level motion compensation without block expansion, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine the horizontal or vertical gradient value. To determine the gradient value for an NxN size block using a 6-tap interpolation filter and gradient filter, a total of 2*6*{(N+5)xN+NxN}*2 multiplication operations are required, as shown in Table 5.

[0387] Figures 9G to 9I are diagrams illustrating the process for determining the horizontal and vertical displacement vectors without extending the reference block, according to other embodiments.

[0388] As explained earlier with reference to Figure 9D, in the process of performing pixel-level motion compensation, the video decoder 100 can determine the horizontal displacement vector and the vertical displacement vector per unit time for each pixel included in the reference block 930, based on the block 935, which is an expanded range of pixels located at the boundary of the reference block 930 by the size of the window. For example, if the window size is (2M+1)x(2M+1), the video decoder 100 can determine the horizontal displacement vector and the vertical displacement vector for each pixel by referencing the pixel values ​​and gradient values ​​of pixels that are M away from the reference block 930, provided that the window is applied to pixels located at the boundary of the reference block 930.

[0389] The following describes a method, using another embodiment, in which the video decoding device 100 does not refer to the pixel values ​​and gradient values ​​of pixels outside the reference block, but uses only the pixel values ​​and gradient values ​​of the reference block corresponding to the current block to determine the values ​​(s1 to s6 in equation (33)) for determining the horizontal and vertical displacement vectors for each pixel. Here, we assume that the window size is 5x5. For convenience, we will describe pixels located horizontally with respect to the current pixel. Those skilled in the art will easily understand that the weighted values ​​for pixels located vertically with respect to the current pixel can also be determined using the same method.

[0390] The video decoder 100 determines the horizontal or vertical displacement vector for each pixel included in the window centered on the current pixel, determining the pixel value P0(i',j') and P1(i',j'), and the horizontal or vertical gradient value.

[0391]

number

[0392] In this case, by multiplying the result value of the calculation performed on each pixel by the weight value, and combining these values, it is possible to determine values ​​s1 to s6 for determining the horizontal and vertical displacement vectors for each pixel.

[0393] Referring to Figure 9G, in the process of determining the horizontal and vertical displacement vectors for the current pixel 961 within block 960, the video decoder 100 can determine weighting values ​​such that each pixel in the window has the same value of 1. The video decoder 100 multiplies the result value calculated for each pixel by the weighting value determined for each pixel, adds all of these results together, and can determine values ​​s1 to s6 for determining the horizontal and vertical displacement vectors for the current pixel.

[0394] Referring to Figure 9H, if pixel 971 is currently adjacent to the boundary of block 970 within block 970, the video decoder 100 can determine the weighting value of pixels adjacent to the boundary of block 970, which are located outside the boundary of block 970, to be 3. The video decoder 100 can determine the weighting value of the remaining pixels 973 to be 1.

[0395] Referring to Figure 9H, if pixel 981 is currently located near the boundary of block 980 (i.e., the pixel is about 1 pixel away from the boundary), the video decoder 100 can determine the weighting value for pixel 982, which is located outside the boundary of block 980, to be 0, and the weighting value for pixel 983, which is adjacent to the boundary of block 980, to be 2. The video decoder 100 can determine the weighting value for the remaining pixels 984 to be 1.

[0396] As explained with reference to Figures 9G to 9I, the video decoder 100 assigns different weighting values ​​to each pixel within the window depending on the current pixel position. This allows it to determine values ​​s1 to s6 for determining the horizontal and vertical displacement vectors for each pixel by utilizing the pixel values ​​and gradient values ​​of pixels located inside the reference block, rather than utilizing the pixel values ​​and gradient values ​​of pixels located outside the reference block corresponding to the current block.

[0397] Figure 9J is a diagram illustrating, in one embodiment, the process of determining pixel-specific horizontal and vertical displacement vectors by referencing the pixel and gradient values ​​of a reference block and applying an exponential smoothing technique to the top, bottom, left, and right directions, without expanding the block.

[0398] Referring to Figure 9J, the video decoder 100 currently calculates, for each pixel contained in block 990, the pixel values ​​P0(i',j') and P1(i',j') of the corresponding reference pixels contained in the corresponding reference block, and the horizontal or vertical gradient value of the corresponding reference pixels contained in the pixels of the corresponding reference block.

[0399]

number

[0400] The video decoding device 100 combines the result of the function calculation performed on the current pixel with the result of the function calculation performed on the surrounding pixels to determine the horizontal and vertical displacement vectors of the current pixel, using values ​​s1 to s6 (σ k(k=1,2,3,4,5,6) ) is also determined. That is, the values ​​s1 to s6 for determining the horizontal and vertical displacement vectors of the current pixel can also be expressed by the weighted average of the calculation values ​​for the current pixel and surrounding pixels, as shown in equation (41) below. At this time, the position coordinates of the pixel contained in window Ω are (i',j'). Also, W[i',j'] represents the weighted value related to the pixel contained in window Ω. Here, the size of window Ω is also (2M+1)x(2M+1) (M is an integer). Also, function A k [i',j'] represents the pixel values ​​P0(i',j') and P1(i',j') (I[i',j'](0,1)) of the corresponding reference pixel related to the pixel at position (i',j') contained within the window Ω, and the horizontal or vertical gradient value of the corresponding reference pixel contained within the corresponding reference block's pixels.

[0401]

number

[0402]

number

[0403] [Table 6] Here I x (0) [i',j'], I x (1) [i',j'] are, respectively

[0404]

number

[0405]

number

[0406] The video decoding device 100 is A k In order to determine the weighted average involving [i',j'], A k For [i',j'], exponential smoothing can be applied in the up, down, left, and right directions, and then averaged.

[0407] Referring to Figure 9J, first the video decoding device 100 moves in the left-right direction, A k Exponential smoothing can be applied to [i',j']. Below, the video decoder 100 controls A in the left-right direction. k Apply the exponential smoothing technique to [i',j'], A k This section provides a detailed explanation of how the weighted average value related to [i',j'] is determined.

[0408] First, we will explain how the video decoder 100 applies exponential smoothing to average to the right. The video decoder 100 uses the following pseudocode 1 to perform A k For [i',j'], averaging can be performed to the right. In this case, H represents the height of the current block, W represents the width of the current block, and Stride represents the distance between one line and the next line in a one-dimensional array. That is, a two-dimensional array A[i,j] can also be represented by a one-dimensional array A[i+j*Stride]. [Pseudocode 1]

[0409]

number

[0410] The following describes how the video decoder 100 applies exponential smoothing to average to the left. The video decoder 100 uses pseudocode 2 as shown below to perform A k For [i',j'], averaging can be performed to the left. In this case, H represents the height of the current block, W represents the width of the current block, and Stride represents the distance between one line and the next line in a one-dimensional array. That is, a two-dimensional array A[i,j] can also be represented by a one-dimensional array A[i+j*Stride]. [Pseudocode 2]

[0411]

number

[0412] The following describes how the video decoder 100 applies exponential smoothing to average downwards. The video decoder 100 uses pseudocode 3 as shown below to perform A k Averaging can be performed downwards on [i',j']. In this case, H represents the height of the current block, W represents the width of the current block, and Stride represents the distance between one line and the next line in a one-dimensional array. That is, a two-dimensional array A[i,j] can also be represented by a one-dimensional array A[i+j*Stride]. [Pseudocode 3]

[0413]

number

[0414]

number

[0415] Therefore, the video decoder 100 currently uses exponential smoothing techniques on block 990 and averages in the up, down, left, and right directions to determine the weighted average σ necessary for determining the horizontal or vertical displacement vector for each pixel. k[i,j] can be determined. That is, the video decoder 100 does not refer to the pixel values ​​and gradient values ​​of the reference block corresponding to block 996 which is an extension of block 990, but only refers to the pixel values ​​and gradient values ​​of the reference block corresponding to block 990, and can determine the horizontal or vertical displacement vector for each pixel.

[0416] Figure 9K is a diagram illustrating the process of using a filter to determine the pixel values ​​of reference pixels within a reference block, as well as the horizontal and vertical gradient values, in order to perform motion compensation related to the current block, according to one embodiment.

[0417] Referring to Figure 9K, the video decoder 100 can perform pixel-level and block-level motion compensation for the current block by utilizing the pixel value and gradient value of the reference pixel in the reference block corresponding to the current block. Therefore, in order to perform pixel-level and block-level motion compensation for the current block, the pixel value and gradient value of the reference pixel in the reference block corresponding to the current block must be determined. In this case, the units of the pixel value and gradient value of the reference pixel in the reference block are also decimal pixel units. For example, the units of the pixel value and gradient value of the reference pixel in the reference block are also 1 / 16 pixel (1 / 16pel) units.

[0418] The video decoder 100 can filter the pixel values ​​of integer pixels in a reference block in order to determine the pixel values ​​and gradient values ​​of reference pixels in a reference block in units of fractional pixels.

[0419] First, the video decoder 100 can determine the pixel values ​​of the reference pixels within a reference block by applying a horizontal 8-tap signal filter (also called an interpolation filter) and a vertical 8-tap signal filter to the pixel values ​​of the integer pixels of the reference pixels.

[0420] The video decoder 100 can determine the pixel values ​​of reference pixels that have position components in units of decimal pixels in the horizontal direction by applying a horizontal 8-tap signal filter to the integer pixel values ​​of the reference block and storing them in a buffer. The video decoder 100 can also determine the pixel values ​​of reference pixels that have position components in units of decimal pixels in the vertical direction by applying a vertical 8-tap signal filter to the pixel values ​​of reference pixels that have position components in units of decimal pixels in the vertical direction.

[0421] The video decoder 100 can determine the horizontal gradient value of the reference pixels within a reference block by first applying a vertical 6-tap signal filter to the integer pixel values ​​of the reference block, and then applying a horizontal 6-tap gradient filter to perform filtering.

[0422] In other words, the video decoder 100 can determine the pixel values ​​of reference pixels that have position components in the vertical direction in units of fractional pixels by applying a vertical 6-tap signal filter to the integer pixel values ​​of the reference block and storing them in a buffer.

[0423] The video decoder 100 can apply a horizontal 6-tap gradient filter to the pixel values ​​of reference pixels that have position components in integer units in the horizontal direction, thereby determining the horizontal gradient values ​​of reference pixels that have position components in decimal units in the horizontal direction.

[0424] The video decoder 100 can determine the vertical gradient value of the reference pixels within a reference block by first applying a vertical 6-tap gradient filter to the integer pixel values ​​of the reference block, and then applying a horizontal 6-tap signal filter to perform filtering.

[0425] In other words, the video decoder 100 can determine the gradient value of a reference pixel having a position component of a fractional pixel in the vertical direction by applying a vertical 6-tap gradient filter to the integer pixel value of the reference block and storing it in a buffer.

[0426] The video decoder 100 can determine the vertical gradient value of a reference pixel having a position component in decimal pixels in the horizontal direction by applying a horizontal 6-tap signal filter to the vertical gradient value of a reference pixel having a position component in decimal pixels in the horizontal direction.

[0427] In other words, the video decoder 100 applies two one-dimensional filters to determine the pixel value of a reference pixel within a reference block, the horizontal gradient value of a reference pixel within a reference block, and the vertical gradient value of a reference pixel within a reference block, respectively. At this time, a multiplication operation is also performed between the coefficient of each filter and the corresponding value related to the pixel. For example, to determine the horizontal gradient value of a reference pixel within a reference block, two 6-tap signals / gradient filters are used, and a total of 12 multiplication operations are performed per pixel. Similarly, to determine the vertical gradient value of a reference pixel within a reference block, two 6-tap signals / gradient filters are used, and a total of 12 multiplication operations are performed per pixel.

[0428] Figure 9L is a diagram illustrating the process of using a filter to perform motion compensation for a current block, and determining the pixel values ​​of reference pixels within the reference block, as well as the horizontal and vertical gradient values, according to another embodiment.

[0429] Referring to Figure 9L, the video decoder 100 can perform pixel-level and block-level motion compensation for the current block by utilizing the pixel value and gradient value of the reference pixel in the reference block corresponding to the current block. Therefore, in order to perform pixel-level and block-level motion compensation for the current block, the pixel value and gradient value of the reference pixel in the reference block corresponding to the current block must be determined. In this case, the units of the pixel value and gradient value of the reference pixel in the reference block are also decimal pixel units. For example, the units of the pixel value and gradient value of the reference pixel in the reference block are also 1 / 16 pixel (1 / 16pel) units.

[0430] The video decoder 100 can filter the pixel values ​​of integer pixels in a reference block in order to determine the pixel values ​​and gradient values ​​of reference pixels in a reference block in units of fractional pixels.

[0431] Unlike the explanation with reference to Figure 9K, the video decoder 100 first applies a horizontal 8-tap signal filter and a vertical 8-tap signal filter to the integer pixel values ​​of the reference block to determine the pixel values ​​of the reference pixels within the reference block. Then, it applies a horizontal 5-tap gradient filter to the pixel values ​​of the reference pixels within the reference block to determine the horizontal gradient values ​​of the reference pixels within the reference block. Furthermore, the video decoder 100 can apply a vertical 5-tap gradient filter to the pixel values ​​of the reference pixels within the reference block to determine the vertical gradient values ​​of the reference pixels within the reference block.

[0432] The video decoder 100 can determine the horizontal or vertical gradient value of a reference pixel within a reference block by applying two one-dimensional signal (interpolation) filters to determine the pixel value of a reference pixel having a position in fractional pixel units, and then applying two one-dimensional gradient filters in parallel to the pixel value of the reference pixel at the fractional pixel position.

[0433] The video decoder 100 can determine the horizontal gradient value Ix(k) of a reference pixel in a reference block by applying a 5-tap horizontal gradient filter (with filter coefficients {9, -48, 0, 48, 9}; however, the filter coefficients are not limited to these) to the pixel value of a reference pixel, as shown in the following formula (42). Here, k can have a value of 0 or 1, which can represent reference picture 0 or 1, respectively. I(k)[i,j] is also the pixel value of the reference pixel in the reference block at position (i,j). i is the horizontal position component of the pixel, and j is the vertical position component of the pixel, both in units of decimal pixels.

[0434]

number

[0435] Furthermore, the video decoder 100 can determine the vertical gradient value Iy(k) of a reference pixel in a reference block by applying a 5-tap vertical gradient filter (with filter coefficients {9,-48,0,48,9}; however, it is not limited to these) to the pixel value of a reference pixel, as shown in the following formula (43). Here, k can have a value of 0 or 1, which can represent reference picture 0 or 1, respectively. I(k)[i,j] is also the pixel value of the reference pixel in the reference block at position (i,j). i is the horizontal position component of the pixel, and j is the vertical position component of the pixel, both in units of decimal pixels.

[0436]

number

[0437] When the video decoder 100 performs pixel-level motion compensation along with block expansion based on the window size, or when it performs pixel-level motion compensation by reducing the gradient filter length, the number of memory accesses and multiplication operations involved in each case are also performed in the video decoder 100 as shown in Table 7 below. At this time, we assume that the signal filter length T is 8, the gradient filter length T is 6, the simplified gradient filter length T is 5, the block size is NxN, and the pixel-level window size 2M+1 is 5.

[0438] [Table 7] In other words, the video decoder 100 performs two multiplication operations per one-dimensional gradient filter according to equations (41) and 42, applying the gradient filter to two reference blocks, and then applying the gradient filter to an expanded (N+4)x(N+4) reference block based on the window size. Therefore, a total of 2*2*{(N+4)x(N+4)}*2 multiplication operations are performed to determine the gradient values ​​of the horizontal and vertical reference pixels.

[0439] When the video decoder 100 performs pixel-level motion compensation along with block expansion based on the window size, or when it performs pixel group-level motion compensation by reducing the gradient filter length without block expansion, the number of memory access operations and multiplication operations involved in each case are also performed in the video decoder 100 as shown in Table 8 below. At this time, we assume that the signal filter length T is 8, the gradient filter length T is 6, the reduced gradient filter length T is 5, the pixel group size is LxL, the block size is NxN, and the pixel-specific window size 2M+1 is 5.

[0440] [Table 8] Therefore, when the video decoder 100 reduces the gradient filter length and performs motion compensation on a pixel group basis without block expansion, the number of memory accesses, multiplication operations, and multiplication operations are significantly reduced compared to when pixel-level motion compensation is performed with block expansion.

[0441] The following describes a method for determining the data units that can be used by a video decoding device 100 in the process of decoding video, with reference to Figures 10 to 23. The operation of the video encoding device 150 may be similar to, or even the opposite of, the operation of the video decoding device 100, which will be described later.

[0442] Figure 10 illustrates, in one embodiment, the process by which the video decoding device 100 divides the current encoding unit and determines at least one encoding unit.

[0443] In one embodiment, the video decoding device 100 can use block configuration information to determine the configuration of the encoded unit, and use the division configuration information to determine how the encoded unit will be divided. That is, the method of dividing the encoded unit indicated by the division configuration information is determined by the block configuration information used by the video decoding device 100.

[0444] In one embodiment, the video decoding device 100 can utilize block shape information indicating that the current encoding unit is square-shaped. For example, the video decoding device 100 can use division shape information to decide whether to not divide the square encoding unit, to divide it vertically, to divide it horizontally, or to divide it into four encoding units. Referring to Figure 10, if the block shape information of the current encoding unit 1000 indicates a square shape, the video decoding device 100 can use division shape information indicating that it will not be divided to decide whether to not divide the encoding unit 1010a which has the same size as the current encoding unit 1000, or to determine the divided encoding units 1010b, 1010c, and 1010d based on division shape information indicating a predetermined division method.

[0445] Referring to Figure 10, in one embodiment, the video decoding device 100 can determine two coding units 1010b obtained by vertically dividing the current coding unit 1000 based on division pattern information indicating that it is divided vertically. The video decoding device 100 can determine two coding units 1010c obtained by horizontally dividing the current coding unit 1000 based on division pattern information indicating that it is divided horizontally. The video decoding device 100 can determine four coding units 1010d obtained by vertically and horizontally dividing the current coding unit 1000 based on division pattern information indicating that it is divided vertically and horizontally. However, the division patterns in which a square coding unit is divided are not limited to the above-described patterns, and may include a variety of patterns that the division pattern information can indicate. The predetermined division patterns in which a square coding unit is divided will be described in detail below through various embodiments.

[0446] Figure 11 illustrates the process by which, in one embodiment, a video decoding device 100 divides an encoding unit that is in a non-square shape and determines at least one encoding unit.

[0447] In one embodiment, the video decoding device 100 can utilize block shape information indicating that the current encoding unit is non-square. The video decoding device 100 can determine, based on the division shape information, whether to divide the non-square current encoding unit or to divide it in a predetermined way. Referring to Figure 11, if the block shape information of the current encoding unit 1100 or 1150 indicates a non-square shape, the video decoding device 100 can determine, based on the division shape information indicating that it will not divide the encoding unit 1110 or 1160 which has the same size as the current encoding unit 1100 or 1150, or to determine the divided encoding units 1120a, 1120b, 1130a, 1130b, 1130c, 1170a, 1170b, 1180a, 1180b, and 1180c based on the division shape information indicating a predetermined division method. The predetermined division method for dividing the non-square encoding unit will be specifically described below through various embodiments.

[0448] In one embodiment, the video decoding device 100 can use the division configuration information to determine the configuration in which the coding unit is divided, in which case the division configuration information can indicate the number of at least one coding unit generated by the division of the coding unit. Referring to Figure 11, if the division configuration information indicates that the current coding unit 1100 or 1150 is divided into two coding units, the video decoding device 100 can divide the current coding unit 1100 or 1150 based on the division configuration information and determine the two coding units 1120a, 11420b or 1170a, 1170b included in the current coding unit.

[0449] In one embodiment, when the video decoding device 100 divides a non-square current coding unit 1100 or 1150 based on the division shape information, it can divide the current coding unit by considering the position of the longer side of the non-square current coding unit 1100 or 1150. For example, the video decoding device 100 can consider the shape of the current coding unit 1100 or 1150 and divide the current coding unit 1100 or 1150 in a direction that divides the longer side of the current coding unit 1100 or 1150, thereby determining a plurality of coding units.

[0450] In one embodiment, if the division information indicates that the coding unit is to be divided into an odd number of blocks, the video decoding device 100 can determine the odd number of coding units currently included in coding unit 1100 or 1150. For example, if the division information indicates that the current coding unit 1100 or 1150 is to be divided into three coding units, the video decoding device 100 can divide the current coding unit 1100 or 1150 into three coding units 1130a, 1130b, 1130c, 1180a, 1180b, and 1180c. In one embodiment, the video decoding device 100 can determine the odd number of coding units currently included in coding unit 1100 or 1150, and the sizes of the determined coding units are not all the same. For example, in a determined odd number of coding units 1130a, 1130b, 1130c, 1180a, 1180b, 1180c, the size of a given coding unit 1130b or 1180b may be different from the size of the other coding units 1130a, 1130c, 1180a, 1180c. That is, the coding units determined by dividing the currently assigned coding unit 1100 or 1150 may have multiple sizes, and in some cases, the odd number of coding units 1130a, 1130b, 1130c, 1180a, 1180b, 1180c may each have different sizes from one another.

[0451] In one embodiment, if the division information indicates that the coding unit is divided into an odd number of blocks, the video decoding device 100 can determine the odd number of coding units currently included in the coding unit 1100 or 1150, and furthermore, the video decoding device 100 can impose a predetermined restriction on at least one of the odd number of coding units generated by the division. Referring to Figure 11, the video decoding device 100 can make the decoding process for the central coding unit 1130b, 1180b, among the three coding units 1130a, 1130b, 1130c, 1180a, 1180b, 1180c generated by the division of the current coding unit 1100 or 1150, different from that for the other coding units 1130a, 1130c, 1180a, 1180c. For example, the video decoding device 100 can restrict the central coding units 1130b and 1180b from being further divided, unlike the other coding units 1130a, 1130c, 1180a, and 1180c, or restrict them to being divided only a predetermined number of times.

[0452] Figure 12 illustrates, in one embodiment, the process by which a video decoding device 100 divides an encoding unit based on at least one of block-type information and division-type information.

[0453] In one embodiment, the video decoding device 100 can decide whether or not to divide the square-shaped first coding unit 1200 into coding units based on at least one of block shape information and division shape information. In one embodiment, if the division shape information indicates that the first coding unit 1200 is divided horizontally, the video decoding device 100 can divide the first coding unit 1200 horizontally and determine the second coding unit 1210. In one embodiment, the terms first coding unit, second coding unit, and third coding unit used are terms used to understand the relationship between the coding units before and after division. For example, if the first coding unit is divided, the second coding unit is determined, and if the second coding unit is divided, the third coding unit is also determined. Hereafter, the relationship between the first coding unit, second coding unit, and third coding unit used will be understood to be due to the aforementioned features.

[0454] In one embodiment, the video decoding device 100 can decide whether to divide the determined second coding unit 1210 into coding units or not, based on at least one of block shape information and division shape information. Referring to Figure 12, the video decoding device 100 divides the second coding unit 1210, which is determined by dividing the first coding unit 1200, into at least one third coding unit 1220a, 1220b, 1220c, 1220d, or does not divide the second coding unit 1210, based on at least one of block shape information and division shape information. The video decoding device 100 can acquire at least one of block-type information and segmented-type information, and based on at least one of the acquired block-type information and segmented-type information, the video decoding device 100 can divide the first coding unit 1200 into a plurality of second coding units (e.g., 1210) of various forms, and the second coding unit 1210 is also divided in the same way that the first coding unit 1200 was divided, based on at least one of the block-type information and segmented-type information. In one embodiment, if the first coding unit 1200 is divided into a second coding unit 1210 based on at least one of the block-type information and segmented-type information relating to the first coding unit 1200, the second coding unit 1210 is also divided into a third coding unit (e.g., 1220a, 1220b, 1220c, 1220d) based on at least one of the block-type information and segmented-type information relating to the second coding unit 1210. In other words, the coding unit is also recursively divided based on at least one of the division shape information and block shape information related to each coding unit. Therefore, in a non-square coding unit, a square coding unit is determined, and such a square coding unit is recursively divided to determine a non-square coding unit. Referring to Figure 12, in the odd number of third coding units 1220b, 1220c, 1220d determined by the division of the non-square second coding unit 1210, a predetermined coding unit (for example, the coding unit located in the middle, or a square coding unit) is also recursively divided.In one embodiment, a square-shaped third coding unit 1220c, which is one of an odd number of third coding units 1220b, 1220c, and 1220d, is divided horizontally into multiple fourth coding units. A non-square-shaped fourth coding unit 1240, which is one of multiple fourth coding units, is also divided into multiple coding units. For example, a non-square-shaped fourth coding unit 1240 is further divided into an odd number of coding units 1250a, 1250b, and 1250c.

[0455] The methods used for recursive partitioning of coding units will be described later through various embodiments.

[0456] In one embodiment, the video decoding device 100 can decide whether to divide each of the third coding units 1220a, 1220b, 1220c, and 1220d into coding units, or whether to leave the second coding unit 1210 undivided, based on at least one of block shape information and division shape information. In one embodiment, the video decoding device 100 can divide a non-square second coding unit 1210 into an odd number of third coding units 1220b, 1220c, and 1220d. The video decoding device 100 can impose predetermined restrictions on a given third coding unit among the odd number of third coding units 1220b, 1220c, and 1220d. For example, the video decoding device 100 can restrict the coding unit 1220c located in the middle of the odd number of third coding units 1220b, 1220c, and 1220d so that it cannot be divided any further, or so that it must be divided a set number of times. Referring to Figure 12, the video decoding device 100 can restrict the middle coding unit 1220c among the odd number of third coding units 1220b, 1220c, and 1220d contained in the non-square second coding unit 1210 to not be further divided, or to be divided into a predetermined division pattern (for example, to be divided into only four coding units, or to be divided in a pattern corresponding to the division pattern of the second coding unit 1210), or to be divided only a predetermined number of times (for example, to be divided only n times, n>0). However, the above restrictions on the middle coding unit 1220c are merely a simple embodiment and should not be interpreted as being limited to the above embodiment, but rather should be interpreted as including a variety of restrictions that allow the middle coding unit 1220c to be decoded differently from the other coding units 1220b and 1220d.

[0457] In one embodiment, the video decoding device 100 can acquire at least one of the block-type information and the division-type information used to divide the currently encoded unit at a predetermined position within the currently encoded unit.

[0458] Figure 13 illustrates, in one embodiment, a method by which a video decoder 100 determines a predetermined encoding unit from an odd number of encoding units. Referring to Figure 13, at least one of the block shape information and division shape information of the current encoding unit 1300 is obtained from a sample at a predetermined position among a plurality of samples contained in the current encoding unit 1300 (for example, a sample 1340 located in the middle). However, the predetermined position within the current encoding unit 1300 from which at least one of such block shape information and division shape information is obtained should not be interpreted as being limited to the middle position shown in Figure 13, but rather should be interpreted as including a variety of positions contained within the current encoding unit 1300 (for example, the top edge, bottom edge, left side, right side, upper left edge, lower left section, upper right edge, or lower right section). The video decoder 100 can obtain at least one of the block shape information and division shape information obtained from the predetermined position and decide whether to divide the current encoding unit into encoding units of various shapes and sizes, or not.

[0459] In one embodiment, the video decoding device 100 can select one encoding unit if the currently encoded unit has been divided into a predetermined number of encoding units. There are various methods for selecting one of multiple encoding units, and such methods will be described later through the various embodiments described below.

[0460] In one embodiment, the video decoding device 100 can divide the current encoding unit into a plurality of encoding units and determine the encoding unit at a predetermined position.

[0461] Figure 13 illustrates a method by which a video decoding device 100 determines a coding unit at a predetermined position among an odd number of coding units, according to one embodiment.

[0462] In one embodiment, the video decoding device 100 can utilize information indicating the position of each of the odd-numbered coding units in order to determine the coding unit located in the middle of the odd-numbered coding units. Referring to Figure 13, the video decoding device 100 can divide the current coding unit 1300 and determine the odd-numbered coding units 1320a, 1320b, and 1320c. The video decoding device 100 can use information relating to the positions of the odd-numbered coding units 1320a, 1320b, and 1320c to determine the middle coding unit 1320b. For example, the video decoding device 100 can determine the middle coding unit 1320b by determining the positions of the coding units 1320a, 1320b, and 1320c based on information indicating the positions of predetermined samples contained in the coding units 1320a, 1320b, and 1320c. Specifically, the video decoding device 100 can determine the middle coding unit 1320b by determining the positions of coding units 1320a, 1320b, and 1320c based on information indicating the positions of samples 1330a, 1330b, and 1330c at the upper left end of coding units 1320a, 1320b, and 1320c.

[0463] In one embodiment, the information indicating the position of the upper left-hand sample 1330a, 1330b, 1330c included in the coding units 1320a, 1320b, 1320c, respectively, may include information relating to the position or coordinates of the coding units 1320a, 1320b, 1320c within the picture. In one embodiment, the information indicating the position of the upper left-hand sample 1330a, 1330b, 1330c included in the coding units 1320a, 1320b, 1320c, respectively, may include information indicating the width or height of the coding units 1320a, 1320b, 1320c currently included in coding unit 1300, and such width or height corresponds to information indicating the difference between coordinates within the picture of the coding units 1320a, 1320b, 1320c. In other words, the video decoding device 100 can determine the middle coding unit 1320b by directly using information relating to the position or coordinates of the coding units 1320a, 1320b, and 1320c within the picture, or by using information relating to the width or height of the coding units corresponding to the difference between the coordinates.

[0464] In one embodiment, information indicating the position of sample 1330a at the upper left end of the upper coding unit 1320a can be expressed in (xa,ya) coordinates, information indicating the position of sample 1330b at the upper left end of the middle coding unit 1320b can be expressed in (xb,yb) coordinates, and information indicating the position of sample 1330c at the upper left end of the lower coding unit 1320c can be expressed in (xc,yc) coordinates. The video decoding device 100 can determine the middle coding unit 1320b by using the coordinates of the upper left end samples 1330a, 1330b, and 1330c contained in coding units 1320a, 1320b, and 1320c, respectively. For example, when the coordinates of samples 1330a, 1330b, and 1330c at the top left are sorted in ascending or descending order, the coding unit 1320b, which contains the coordinates (xb,yb) of sample 1330b located in the middle, can be determined as the middle coding unit among the coding units 1320a, 1320b, and 1320c determined by the division of the currently determined coding unit 1300. However, the coordinates indicating the positions of samples 1330a, 1330b, and 1330c at the upper left edge can represent absolute positions within the picture. Furthermore, it is also possible to use the (dxb, dyb) coordinates, which indicate the relative position of sample 1330b at the upper left edge of the middle coding unit 1320b, and the (dxc, dyc) coordinates, which indicate the relative position of sample 1330c at the upper left edge of the lower coding unit 1320c, based on the position of sample 1330a at the upper left edge of the upper coding unit 1320a. Moreover, the method of determining the coding unit at a given position by using the coordinates of the sample as information indicating the position of the sample included in the coding unit should not be interpreted as being limited to the method described above, but rather as a variety of arithmetic methods that can utilize the coordinates of the sample.

[0465] In one embodiment, the video decoding device 100 can divide the currently encoded unit 1300 into a plurality of encoded units 1320a, 1320b, and 1320c, and can select an encoded unit from among the encoded units 1320a, 1320b, and 1320c according to a predetermined criterion. For example, the video decoding device 100 can select an encoded unit 1320b of a different size from the encoded units 1320a, 1320b, and 1320c.

[0466] In one embodiment, the video decoding device 100 can determine the width or height of each coding unit 1320a, 1320b, and 1320c by using the (xa,ya) coordinates, which are information indicating the position of sample 1330a at the upper left end of the upper coding unit 1320a; the (xb,yb) coordinates, which are information indicating the position of sample 1330b at the upper left end of the middle coding unit 1320b; and the (xc,yc) coordinates, which are information indicating the position of sample 1330c at the upper left end of the lower coding unit 1320c. The video decoding device 100 can determine the size of each coding unit 1320a, 1320b, and 1320c by using the coordinates (xa,ya), (xb,yb), and (xc,yc), which are the coordinates indicating the positions of the coding units 1320a, 1320b, and 1320c.

[0467] In one embodiment, the video decoding device 100 can determine the width of the upper coding unit 1320a as xb-xa and its height as yb-ya. In one embodiment, the video decoding device 100 can determine the width of the middle coding unit 1320b as xc-xb and its height as yc-yb. In one embodiment, the video decoding device 100 can determine the width or height of the lower coding unit using the width or height of the current coding unit and the width and height of the upper coding unit 1320a and the middle coding unit 1320b. Based on the determined widths and heights of the coding units 1320a, 1320b, and 1320c, the video decoding device 100 can determine a coding unit having a different size from the other coding units. Referring to Figure 13, the video decoding device 100 can determine the middle coding unit 1320b, which has a different size from the upper coding unit 1320a and the lower coding unit 1320c, as the coding unit at a predetermined position. However, the process by which the aforementioned video decoding device 100 determines coding units having different coding units and different sizes is merely one embodiment of determining coding units at a predetermined position using the size of coding units determined based on sample coordinates. Therefore, various processes are also used to determine coding units at a predetermined position by comparing the sizes of coding units determined by predetermined sample coordinates.

[0468] However, the sample positions considered in determining the location of the coding unit are not limited to the upper left corner as described above; it can also be interpreted that information relating to any sample position included in the coding unit may be used.

[0469] In one embodiment, the video decoding device 100 can take into account the shape of the current coding unit and select a coding unit at a predetermined position from an odd number of coding units determined by dividing the current coding unit. For example, if the current coding unit is non-square in shape, with a width greater than its height, the video decoding device 100 can determine a coding unit at a predetermined position along the horizontal direction. That is, the video decoding device 100 can determine one of coding units at different positions in the horizontal direction and impose restrictions on that coding unit. If the current coding unit is non-square in shape, with a height greater than its width, the video decoding device 100 can determine a coding unit at a predetermined position along the vertical direction. That is, the video decoding device 100 can determine one of coding units at different positions in the vertical direction and impose restrictions on that coding unit.

[0470] In one embodiment, the video decoding device 100 can use information indicating the position of each of the even-numbered coding units to determine the coding unit at a predetermined position among the even-numbered coding units. The video decoding device 100 can currently divide the coding unit and determine the even-numbered coding units, and can use the information relating to the positions of the even-numbered coding units to determine the coding unit at the predetermined position. The specific process related to this is the same process as the process for determining the coding unit at a predetermined position (for example, the middle position) among the odd-numbered coding units described in Figure 13, so it will be omitted here.

[0471] In one embodiment, when a non-square current coding unit is divided into multiple coding units, predetermined information about the coding unit at a predetermined position can be used during the division process to determine the coding unit at a predetermined position among the multiple coding units. For example, the video decoding device 100 can use at least one of the block shape information and division shape information stored in the sample included in the middle coding unit during the division process to determine the coding unit located in the middle of the multiple coding units into which the current coding unit has been divided.

[0472] Referring to Figure 13, the video decoding device 100 can divide the current encoded unit 1300 into multiple encoded units 1320a, 1320b, and 1320c based on at least one of the block shape information and the division shape information, and can determine the encoded unit 1320b located in the middle of the multiple encoded units 1320a, 1320b, and 1320c. Furthermore, the video decoding device 100 can determine the encoded unit 1320b located in the middle by considering the position from which at least one of the block shape information and the division shape information is acquired. That is, if at least one of the block shape information and the division shape information of the current encoded unit 1300 is acquired from a sample 1340 located in the middle of the current encoded unit 1300, and the current encoded unit 1300 is divided into multiple encoded units 1320a, 1320b, and 1320c based on at least one of the block shape information and the division shape information, the encoded unit 1320b containing the sample 1340 can be determined to be the encoded unit located in the middle. However, the information used to determine the central coding unit is not limited to interpreting at least one of block-type information and segment-type information; various types of information are also used in the process of determining the central coding unit.

[0473] In one embodiment, predetermined information for identifying an encoding unit at a predetermined location is also obtained from a predetermined sample included in the encoding unit to be determined. Referring to Figure 13, the video decoding device 100 can use at least one of block shape information and division shape information obtained from a predetermined sample within the current encoding unit 1300 (for example, a sample located in the middle of the current encoding unit 1300) to determine an encoding unit at a predetermined location (for example, an encoding unit located in the middle of the multiple divided encoding units) in a plurality of encoding units 1320a, 1320b, 1320c determined by dividing the current encoding unit 1300. In other words, the video decoding device 100 can determine a sample at a predetermined position by considering the block-block configuration of the current encoding unit 1300, and the video decoding device 100 can determine an encoding unit 1320b from among the multiple encoding units 1320a, 1320b, and 1320c determined by dividing the current encoding unit 1300, which contains a sample from which predetermined information (for example, at least one of block configuration information and division configuration information) is obtained, and can set a predetermined restriction. Referring to Figure 13, in one embodiment, the video decoding device 100 can determine a sample 1340 located in the middle of the current encoding unit 1300 as a sample from which predetermined information is obtained, and the video decoding device 100 can set a predetermined restriction in the decoding process for the encoding unit 1320b containing such a sample 1340. However, the position of the sample from which predetermined information is obtained is not limited to the aforementioned position, but can also be interpreted as a sample at any position in the encoding unit 1320b that is being determined in order to set the restriction.

[0474] In one embodiment, the location of a sample from which predetermined information is obtained is also determined by the shape of the currently encoded unit 1300. In one embodiment, block shape information can determine whether the shape of the currently encoded unit is square or non-square, and the location of a sample from which predetermined information is obtained can be determined by the shape. For example, the video decoder 100 can use at least one of the information relating to the width of the currently encoded unit and the information relating to the height to determine that a sample located on a boundary that divides at least one of the width and height of the currently encoded unit in half is the sample from which predetermined information is obtained. To give another example, if the block shape information relating to the currently encoded unit indicates that it is non-square, the video decoder 100 can determine that one of the samples adjacent to the boundary that divides the longer side of the currently encoded unit in half is the sample from which predetermined information is obtained.

[0475] In one embodiment, when the current coding unit is divided into multiple coding units, the video decoding device 100 can use at least one of block-type information and division-type information to determine the coding unit at a predetermined position among the multiple coding units. In one embodiment, the video decoding device 100 can obtain at least one of block-type information and division-type information from a sample at a predetermined position included in the coding unit, and the video decoding device 100 can divide the multiple coding units generated by the division of the current coding unit using at least one of the division-type information and block-type information obtained from a sample at a predetermined position included in each of the multiple coding units. That is, the coding unit can also be recursively divided using at least one of the block-type information and division-type information obtained from a sample at a predetermined position included in each coding unit. The recursive division process of the coding unit has been explained with reference to Figure 12, so a detailed explanation will be omitted.

[0476] In one embodiment, the video decoding device 100 can divide the current encoding unit and determine at least one encoding unit, and can determine the order in which such at least one encoding unit is decoded by a predetermined block (e.g., the current encoding unit).

[0477] Figure 14 illustrates the order in which multiple encoding units are processed when the video decoding device 100 divides the current encoding unit and determines multiple encoding units according to one embodiment.

[0478] In one embodiment, the video decoding device 100 can, based on block shape information and division shape information, vertically divide the first coding unit 1400 to determine second coding units 1410a and 1410b, horizontally divide the first coding unit 1400 to determine second coding units 1430a and 1430b, or divide the first coding unit 1400 vertically and horizontally to determine second coding units 1450a, 1450b, 1450c, and 1450d.

[0479] Referring to Figure 14, the video decoder 100 can determine the processing order of the second coding units 1410a and 1410b, which are determined by vertically dividing the first coding unit 1400, so that they are processed horizontally (1410c). The video decoder 100 can also determine the processing order of the second coding units 1430a and 1430b, which are determined by horizontally dividing the first coding unit 1400, so that they are processed vertically (1430c). The video decoder 100 can determine the processing order of the second coding units 1450a, 1450b, 1450c, and 1450d, which are determined by vertically and horizontally dividing the first coding unit 1400, so that the coding units located in one row are processed before the coding units located in the next row are processed (for example, this can be determined by a raster scan order or a z scan order (1450e)).

[0480] In one embodiment, the video decoding device 100 can recursively divide an encoding unit. Referring to Figure 14, the video decoding device 100 can divide the first encoding unit 1400 and determine a plurality of encoding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d, and can recursively divide each of the determined plurality of encoding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d. The method for dividing the plurality of encoding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d also corresponds to the method for dividing the first encoding unit 1400. As a result, the multiple coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d are each independently divided into multiple coding units. Referring to Figure 14, the video decoding device 100 can vertically divide the first coding unit 1400 to determine the second coding units 1410a and 1410b, and furthermore, it can decide whether to divide the second coding units 1410a and 1410b independently or not.

[0481] In one embodiment, the video decoding device 100 can horizontally divide the left second encoding unit 1410a into third encoding units 1420a and 1420b, while the right second encoding unit 1410b is not divided.

[0482] In one embodiment, the processing order of coding units is also determined based on the coding unit division process. In other words, the processing order of divided coding units is also determined based on the processing order of the coding unit immediately before division. The video decoding device 100 can determine the processing order of the third coding units 1420a and 1420b, which are determined when the left second coding unit 1410a is divided, independently of the right second coding unit 1410b. Since the left second coding unit 1410a is divided horizontally and the third coding units 1420a and 1420b are determined, the third coding units 1420a and 1420b are also processed vertically (1420c). Furthermore, since the processing order of the left-side second coding unit 1410a and the right-side second coding unit 1410b corresponds to the horizontal direction 1410c, the third coding units 1420a and 1420b included in the left-side second coding unit 1410a are processed vertically (1420c) before the right-side coding unit 1410b is processed. The above description is intended to explain the process by which the processing order of coding units is determined by the coding units before division, and should not be interpreted as being limited to the above-described embodiment. Rather, it should be interpreted as being usable in various ways in which coding units determined by being divided into various forms are processed independently in a predetermined order.

[0483] Figure 15 illustrates, in one embodiment, the process by which the video decoding device 100 determines that, if the encoding unit cannot be processed in a predetermined order, the current encoding unit will be divided into an odd number of encoding units.

[0484] In one embodiment, the video decoding device 100 can determine, based on the acquired block shape information and division shape information, that the current coding unit is divided into an odd number of coding units. Referring to Figure 15, the square-shaped first coding unit 1500 is divided into non-square-shaped second coding units 1510a and 1510b, and the second coding units 1510a and 1510b are each independently divided into third coding units 1520a, 1520b, 1520c, 1520d, and 1520e. In another embodiment, the video decoding device 100 can divide the left coding unit 1510a of the second coding unit horizontally to determine multiple third coding units 1520a and 1520b, and the right coding unit 1510b can be divided into an odd number of third coding units 1520c, 1520d, and 1520e.

[0485] In one embodiment, the video decoding device 100 can determine whether the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e are processed in a predetermined order, and whether there are coding units that have been divided into an odd number of parts. Referring to Figure 15, the video decoding device 100 can recursively divide the first coding unit 1500 to determine the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e. Based on at least one of the block configuration information and the division configuration information, the video decoding device 100 can determine whether the first coding unit 1500, the second coding units 1510a, 1510b, or the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e are divided into an odd number of coding units in the configuration in which they are divided. For example, in the second coding units 1510a and 1510b, the coding unit located on the right side may be divided into an odd number of third coding units 1520c, 1520d, and 1520e. The order in which the multiple coding units contained in the first coding unit 1500 are processed may be a predetermined order (for example, a z-scan order (1530)), and the video decoding device 100 can determine whether the third coding units 1520c, 1520d, and 1520e, which are determined by dividing the right-side second coding unit 1510b into an odd number, satisfy the condition that they are processed in the predetermined order.

[0486] In one embodiment, the video decoding device 100 can determine whether the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e included in the first coding unit 1500 satisfy the condition that they be processed in a predetermined order, and the condition relates to whether at least one of the width and height of the second coding units 1510a and 1510b is divided in half by the boundary of the third coding units 1520a, 1520b, 1520c, 1520d, and 1520e. For example, the third coding units 1520a and 1520b, determined by dividing the height of the non-square left second coding unit 1510a in half, satisfy the condition. However, the boundaries of the third coding units 1520c, 1520d, and 1520e, determined by dividing the right second coding unit 1510b into three coding units, cannot divide the width or height of the right second coding unit 1510b in half. Therefore, the third coding units 1520c, 1520d, and 1520e do not satisfy the condition. In such cases of non-satisfaction, the video decoding device 100 determines that there is a disconnection in the scan order, and based on the determination result, it can determine that the right second coding unit 1510b is divided into an odd number of coding units. In one embodiment, when the video decoding device 100 is divided into an odd number of encoding units, a predetermined restriction can be placed on an encoding unit at a predetermined position among the divided encoding units. Since the content of such restrictions or predetermined positions have been explained through various embodiments, a detailed explanation will be omitted here.

[0487] Figure 16 illustrates the process by which a video decoding device 100 divides a first coding unit 1600 and determines at least one coding unit, according to one embodiment. According to one embodiment, the video decoding device 100 can divide the first coding unit 1600 based on at least one of the block shape information and division shape information acquired via the acquisition unit 110. The square-shaped first coding unit 1600 may be divided into coding units having four square shapes, or it may be divided into multiple non-square coding units. For example, referring to Figure 16, if the block shape information indicates that the first coding unit 1600 is square, and the division shape information indicates that it is divided into non-square coding units, the video decoding device 100 divides the first coding unit 1600 into multiple non-square coding units. Specifically, if the division configuration information indicates that the first coding unit 1600 is divided horizontally or vertically to determine an odd number of coding units, the video decoding device 100 can divide the square-shaped first coding unit 1600 into an odd number of second coding units, namely second coding units 1610a, 1610b, 1610c determined by vertical division, or second coding units 1620a, 1620b, 1620c determined by horizontal division.

[0488] In one embodiment, the video decoding device 100 can determine whether the second coding units 1610a, 1610b, 1610c, 1620a, 1620b, and 1620c included in the first coding unit 1600 satisfy the condition that they are processed in a predetermined order. This condition relates to whether the boundaries of the second coding units 1610a, 1610b, 1610c, 1620a, 1620b, and 1620c divide at least one of the width and height of the first coding unit 1600 in half. Referring to Figure 16, the boundaries of the second coding units 1610a, 1610b, and 1610c, determined by vertically dividing the square-shaped first coding unit 1600, cannot divide the width of the first coding unit 1600 in half, so it is determined that the first coding unit 1600 does not satisfy the condition that it is processed in a predetermined order. Furthermore, since the boundaries of the second coding units 1620a, 1620b, and 1620c, which are determined by horizontally dividing the square-shaped first coding unit 1600, cannot divide the width of the first coding unit 1600 in half, it is also determined that the first coding unit 1600 cannot satisfy the condition for being processed in a predetermined order. In the event that such a condition is not satisfied, the video decoding device 100 determines that there is a discontinuation in the scan order, and based on the determination result, it can determine that the first coding unit 1600 is divided into an odd number of coding units. In one embodiment, when the first coding unit 1600 is divided into an odd number of coding units, the video decoding device 100 can place a predetermined restriction on the coding unit at a predetermined position among the divided coding units. Since the content of such restrictions or predetermined positions have been explained through various embodiments, a detailed explanation will be omitted here.

[0489] In one embodiment, the video decoding device 100 can divide the first encoding unit and determine various forms of encoding units.

[0490] Referring to Figure 16, the video decoding device 100 can divide the square-shaped first coding unit 1600, the non-square-shaped first coding unit 1630, or 1650 into coding units of various shapes.

[0491] Figure 17 illustrates, in one embodiment, that when the non-square second encoding unit determined by dividing the first encoding unit 1700 satisfies predetermined conditions, the form in which the second encoding unit can be divided is limited.

[0492] In one embodiment, the video decoding device 100 can decide to divide the square-shaped first coding unit 1700 into non-square-shaped second coding units 1710a, 1710b, 1720a, and 1720b based on at least one of the block shape information and division shape information acquired via the acquisition unit 105. The second coding units 1710a, 1710b, 1720a, and 1720b can also be divided independently. As a result, the video decoding device 100 can decide whether to divide each of the second coding units 1710a, 1710b, 1720a, and 1720b into multiple coding units or not to divide them, based on at least one of the block shape information and division shape information relating to each of them. In one embodiment, the video decoding device 100 can divide the non-square left second coding unit 1710a, which is determined by dividing the first coding unit 1700 vertically, horizontally to determine the third coding units 1712a and 1712b. However, if the left second coding unit 1710a is divided horizontally, the video decoding device 100 can restrict the right second coding unit 1710b from being divided horizontally in the same direction as the left second coding unit 1710a. If the right second coding unit 1710b is divided in the same direction and the third coding units 1714a and 1714b are determined, the left second coding unit 1710a and the right second coding unit 1710b are also divided horizontally independently, thereby determining the third coding units 1712a, 1712b, 1714a, and 1714b. However, this is the same result as when the video decoding device 100 divides the first encoding unit 1700 into four square-shaped second encoding units 1730a, 1730b, 1730c, and 1730d based on at least one of the block-type information and the division-type information, and this is inefficient in terms of video decoding.

[0493] In one embodiment, the video decoding device 100 can divide the non-square second coding units 1720a or 1720b, which are determined by dividing the first coding unit 11300 horizontally, vertically to determine third coding units 1722a, 1722b, 1724a, and 1724b. However, if the video decoding device 100 divides one of the second coding units (for example, the upper second coding unit 1720a) vertically, for the reasons mentioned above, it can restrict the other second coding units (for example, the lower coding unit 1720b) ​​from being divided vertically in the same direction as the upper second coding unit 1720a.

[0494] Figure 18 illustrates the process by which the video decoder 100 divides the square-shaped coding units, in cases where it is not possible to show that the division pattern information is divided into four square-shaped coding units according to one embodiment.

[0495] In one embodiment, the video decoding device 100 can divide the first coding unit 1800 and determine the second coding units 1810a, 1810b, 1820a, and 1820b based on at least one of block shape information and division shape information. The division shape information may include information relating to various shapes in which the coding unit is divided, but the information relating to various shapes may not include information for dividing it into four square coding units. According to such division shape information, the video decoding device 100 cannot divide the square first coding unit 1800 into four square second coding units 1830a, 1830b, 1830c, and 1830d. ​​Based on the division shape information, the video decoding device 100 can determine the non-square second coding units 1810a, 1810b, 1820a, and 1820b.

[0496] In one embodiment, the video decoding device 100 can independently divide the non-square second coding units 1810a, 1810b, 1820a, and 1820b. Each of the second coding units 1810a, 1810b, 1820a, and 1820b is divided in a predetermined order via a recursive method, which is also a division method corresponding to the method in which the first coding unit 1800 is divided based on at least one of block shape information and division shape information.

[0497] For example, the video decoding device 100 can determine square-shaped third coding units 1812a and 1812b by horizontally dividing the left second coding unit 1810a, and can determine square-shaped third coding units 1814a and 1814b by horizontally dividing the right second coding unit 1810b. Furthermore, the video decoding device 100 can also determine square-shaped third coding units 1816a, 1816b, 1816c, and 1816d by horizontally dividing either the left second coding unit 1810a or the right second coding unit 1810b. In such a case, the coding unit can be determined in the same form as when the first coding unit 1800 is divided into four square-shaped second coding units 1830a, 1830b, 1830c, and 1830d.

[0498] To give another example, the video decoder 100 can determine square-shaped third coding units 1822a and 1822b by vertically dividing the upper second coding unit 1820a, and can determine square-shaped third coding units 1824a and 1824b by vertically dividing the lower second coding unit 1820b. Furthermore, the video decoder 100 can also determine square-shaped third coding units 1822a, 1822b, 1824a and 1824b by vertically dividing either the upper second coding unit 1820a or the lower second coding unit 1820b. In such a case, the coding unit can be determined in the same form as when the first coding unit 1800 is divided into four square-shaped second coding units 1830a, 1830b, 1830c, and 1830d.

[0499] Figure 19 illustrates, in one embodiment, that the processing order between multiple coding units may differ depending on the coding unit division process.

[0500] In one embodiment, the video decoding device 100 can divide the first coding unit 1900 based on block shape information and division shape information. If the block shape information indicates a square shape and the division shape information indicates that the first coding unit 1900 is divided in at least one of the horizontal and vertical directions, the video decoding device 100 can divide the first coding unit 1900 and determine, for example, second coding units 1910a, 1910b, 1920a, 1920b, 1930a, 1930b, 1930c, and 1930d. Referring to Figure 19, the non-square second coding units 1910a, 1910b, 1920a, and 1920b, which are determined by dividing the first coding unit 1900 only in the horizontal or vertical direction, can also be divided independently based on the block shape information and division shape information relating to each of them. For example, the video decoding device 100 can determine third coding units 1916a, 1916b, 1916c, and 1916d by dividing the second coding units 1910a and 1910b, which are generated by dividing the first coding unit 1900 vertically, horizontally, and can determine third coding units 1926a, 1926b, 1926c, and 1926d by dividing the second coding units 1920a and 1920b, which are generated by dividing the first coding unit 1900 horizontally, horizontally. The division process of such second coding units 1910a, 1910b, 1920a, and 1920b has been explained in relation to Figure 17, so a detailed explanation will be omitted.

[0501] In one embodiment, the video decoding device 100 can process coding units in a predetermined order. The features related to the processing of coding units in a predetermined order have been explained in relation to Figure 14, so a detailed explanation will be omitted. Referring to Figure 19, the video decoding device 100 can divide a square-shaped first coding unit 1900 and determine four square-shaped third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d. In one embodiment, the video decoding device 100 can determine the processing order of the third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d depending on the form in which the first coding unit 1900 is divided.

[0502] In one embodiment, the video decoding device 100 can divide the second coding units 1910a and 1910b, which are generated by vertical division, horizontally, and determine the third coding units 1916a, 1916b, 1916c, and 1916d. The video decoding device 100 can process the third coding units 1916a, 1916b, 1916c, and 1916d in the order (1917) of first processing the third coding units 1916a and 1916b contained in the left second coding unit 1910a vertically, and then processing the third coding units 1916c and 1916d contained in the right second coding unit 1910b vertically.

[0503] In one embodiment, the video decoding device 100 can divide the second coding units 1920a and 1920b, which are generated by dividing them horizontally, vertically, and determine the third coding units 1926a, 1926b, 1926c, and 1926d. The video decoding device 100 can process the third coding units 1926a, 1926b, 1926c, and 1926d in the order (1927) of first processing the third coding units 1926a and 1926b contained in the upper second coding unit 1920a horizontally, and then processing the third coding units 1926c and 1926d contained in the lower second coding unit 1920b horizontally.

[0504] Referring to Figure 19, the second coding units 1910a, 1910b, 1920a, and 1920b are each divided, and the square-shaped third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d are determined. The second coding units 1910a and 1910b determined by vertical division, and the second coding units 1920a and 1920b determined by horizontal division, are divided into different forms from each other. However, according to the third coding units 1916a, 1916b, 1916c, 1916d, 1926a, 1926b, 1926c, and 1926d determined thereafter, the result is that the first coding unit 1900 is divided into coding units of the same form. As a result, the video decoding device 100 can recursively divide the coding unit through different processes based on at least one of the block shape information and the division shape information, thereby determining coding units of the same shape, and processing multiple coding units determined to be of the same shape in different orders.

[0505] Figure 20 illustrates the process by which the depth of a coding unit is determined when, according to one embodiment, a coding unit is recursively divided and multiple coding units are determined, as the shape and size of the coding unit change.

[0506] In one embodiment, the video decoding device 100 can determine the depth of an encoding unit according to a predetermined criterion. For example, the predetermined criterion may be the length of the long side of the encoding unit. If the length of the long side of the current encoding unit is divided into 2n (n>0) times the length of the long side of the encoding unit before division, the video decoding device 100 can determine that the depth of the current encoding unit is increased by n compared to the depth of the encoding unit before division. Hereinafter, the encoding unit with increased depth will be referred to as the lower-depth encoding unit.

[0507] Referring to Figure 20, in one embodiment, based on block shape information indicating a square shape (for example, the block shape information can indicate "0:SQUARE"), the video decoding device 100 can divide the square-shaped first coding unit 2000 and determine the lower-depth second coding unit 2002, third coding unit 2004, etc. If the size of the square-shaped first coding unit 2000 is 2Nx2N, then the width and height of the first coding unit 2000 can be divided by 1 / 2. 1 The second coding unit 2002, determined by dividing it in half, can have a size of N x N. Furthermore, the third coding unit 2004, determined by dividing the width and height of the second coding unit 2002 into 1 / 2 sizes, can have a size of N / 2 x N / 2. In that case, the width and height of the third coding unit 2004 are 1 / 2 of the first coding unit 2000. 2 This corresponds to double. If the depth of the first coding unit 2000 is D, then it is half the width and height of the first coding unit 2000. 1 The depth of the second coding unit 2002, which is double, is also D+1, and is half the width and height of the first coding unit 2000. 2 The depth of the third coding unit 2004, which is double, is also D+2.

[0508] In one embodiment, based on block shape information indicating a non-square shape (for example, the block shape information may indicate "1:NS_VER" indicating that the height is longer than the width, or "2:NS_HOR" indicating that the width is longer than the height), the video decoder 100 can divide the non-square first coding unit 2010 or 2020 and determine the lower-depth second coding unit 2012 or 2022 and the third coding unit 2014 or 2024.

[0509] The video decoder 100 can divide at least one of the width and height of the first coding unit 2010, which is of size Nx2N, to determine, for example, second coding units 2002, 2012, and 2022. That is, the video decoder 100 can divide the first coding unit 2010 horizontally to determine a second coding unit 2002 of size NxN, or a second coding unit 2022 of size NxN / 2, or it can divide it horizontally and vertically to determine a second coding unit 2012 of size N / 2xN.

[0510] In one embodiment, the video decoding device 100 can also divide at least one of the width and height of the 2NxN size first coding unit 2020 to determine, for example, second coding units 2002, 2012, and 2022. That is, the video decoding device 100 can divide the first coding unit 2020 vertically to determine an NxN size second coding unit 2002 or an N / 2xN size second coding unit 2012, and can also divide it horizontally and vertically to determine an NxN / 2 size second coding unit 2022.

[0511] In one embodiment, the video decoding device 100 can also divide at least one of the width and height of the NxN size second coding unit 2002 to determine, for example, third coding units 2004, 2014, 2024. That is, the video decoding device 100 divides the second coding unit 2002 vertically and horizontally to determine a third coding unit 2004 of N / 2xN / 2 size, or N / 2 2 Determine a third coding unit 2014 of size xN / 2, or N / 2xN / 2 2 The third coding unit of size, 2024, can be determined.

[0512] In one embodiment, the video decoding device 100 can also divide at least one of the width and height of the N / 2xN size second coding unit 2012 to determine, for example, third coding units 2004, 2014, 2024. That is, the video decoding device 100 divides the second coding unit 2012 horizontally to determine N / 2xN / 2 size third coding units 2004, or N / 2xN / 2 2 Determine the third coding unit of size, 2024, or divide it vertically and horizontally, N / 2 2 A third coding unit 2014 of size xN / 2 can be determined.

[0513] In one embodiment, the video decoding device 100 can also divide at least one of the width and height of the NxN / 2 size second coding unit 2014 to determine, for example, third coding units 2004, 2014, 2024. That is, the video decoding device 100 can divide the second coding unit 2012 vertically to determine the N / 2xN / 2 size third coding unit 2004, or N / 2 2 Determine a third coding unit 2014 of size xN / 2, or divide it vertically and horizontally to N / 2xN / 2 2 The third coding unit of size, 2024, can be determined.

[0514] In one embodiment, the video decoding device 100 can divide, for example, square coding units 2000, 2002, and 2004 horizontally or vertically. For example, a 2Nx2N size first coding unit 2000 can be divided vertically to determine an Nx2N size first coding unit 2010, or it can be divided horizontally to determine a 2NxN size first coding unit 2020. In one embodiment, if the depth is determined based on the length of the longest side of the coding unit, the depth of the coding unit determined by dividing a 2Nx2N size first coding unit 2000, 2002, or 2004 horizontally or vertically is also the same as the depth of the first coding unit 2000, 2002, or 2004.

[0515] According to an embodiment, the width and height of the third encoding unit 2014 or 2024 are 1 / 2 of those of the first encoding unit 2010 or 2020. 2 Correspondingly, it is twice. When the depth of the first encoding unit 2010 or 2020 is D, the depth of the second encoding unit 2012 or 2014, which is 1 / 2 times the width and height of the first encoding unit 2010 or 2020, is also D + 1, and the depth of the third encoding unit 2014 or 2024, which is 1 / 2 2 times the width and height of the first encoding unit 2010 or 2020, is also D + 2.

[0516] FIG. 21 illustrates, according to an embodiment, the depth and the index (PID: part index) for classifying encoding units, which are also determined by the form and size of the encoding units.

[0517] According to an embodiment, the video decoding device 100 can divide the square first encoding unit 2100 and determine second encoding units in various forms. Referring to FIG. 21, the video decoding device 100 can divide the first encoding unit 2100 in at least one of the vertical and horizontal directions according to the division form information, and determine the second encoding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, 2106d. That is, the video decoding device 100 can determine the second encoding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, 2106d based on the division form information related to the first encoding unit 2100.

[0518] In one embodiment, the depth of the second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d, which are determined by the division shape information relating to the square-shaped first coding unit 2100, is also determined based on the length of the longer side. For example, since the length of one side of the square-shaped first coding unit 2100 is the same as the length of the longer side of the non-square-shaped second coding units 2102a, 2102b, 2104a, and 2104b, the depth of the first coding unit 2100 and the non-square-shaped second coding units 2102a, 2102b, 2104a, and 2104b can be considered to be the same as D. Conversely, if the video decoding device 100 divides the first coding unit 2100 into four square-shaped second coding units 2106a, 2106b, 2106c, and 2106d based on the division pattern information, the length of one side of the square-shaped second coding units 2106a, 2106b, 2106c, and 2106d is half the length of one side of the first coding unit 2100. Therefore, the depth of the second coding units 2106a, 2106b, 2106c, and 2106d is also D+1, which is one depth lower than the depth D of the first coding unit 2100.

[0519] In one embodiment, the video decoding device 100 can divide a first coding unit 2110, which has a height greater than its width, horizontally according to the division configuration information, and divide it into a plurality of second coding units 2112a, 2112b, 2114a, 2114b, and 2114c. In another embodiment, the video decoding device 100 can divide a first coding unit 2120, which has a width greater than its height, vertically according to the division configuration information, and divide it into a plurality of second coding units 2122a, 2122b, 2124a, 2124b, and 2124c.

[0520] In one embodiment, the depth of the second coding units 2112a, 2112b, 2114a, 2114b, 2116a, 2116b, 2116c, and 2116d, which are determined by the division shape information relating to the non-square first coding unit 2110 or 2120, is also determined based on the length of the longer side. For example, since the length of one side of the square second coding units 2112a and 2112b is half the length of one side of the non-square first coding unit 2110, whose height is longer than its width, the depth of the square second coding units 2102a, 2102b, 2104a, and 2104b is D+1, which is one depth lower than the depth D of the non-square first coding unit 2110.

[0521] Furthermore, the video decoding device 100 can divide the non-square-shaped first coding unit 2110 into an odd number of second coding units 2114a, 2114b, and 2114c based on the division shape information. The odd number of second coding units 2114a, 2114b, and 2114c may include non-square-shaped second coding units 2114a and 2114c, and a square-shaped second coding unit 2114b. In that case, since the length of the longer side of the non-square-shaped second coding units 2114a and 2114c, and the length of one side of the square-shaped second coding unit 2114b are 1 / 2 the length of one side of the first coding unit 2110, the depth of the second coding units 2114a, 2114b, and 2114c is also a depth of D+1, which is one depth lower than the depth D of the first coding unit 2110. The video decoding device 100 can determine the depth of an encoding unit related to a non-square first encoding unit 2120, in a manner corresponding to the method for determining the depth of an encoding unit related to the first encoding unit 2110.

[0522] In one embodiment, the video decoding device 100 can determine the index (PID) for the division of a divided coding unit based on the ratio of the sizes of the coding units when the odd number of divided coding units are not the same size. Referring to Figure 21, the middle coding unit 2114b among the odd number of divided coding units 2114a, 2114b, 2114c has the same width as the other coding units 2114a, 2114c, but its height is twice that of the other coding units 2114a, 2114c. That is, the middle coding unit 2114b in this case may include both of the other coding units 2114a, 2114c. Therefore, depending on the scan order, if the index (PID) of the middle coding unit 2114b is 1, then the next coding unit 2114c in the sequence will have an index that is 2 higher, which is 3. In other words, there is a discontinuity in the index values. In one embodiment, the video decoding device 100 can determine whether an odd number of divided coding units are of the same size as each other, based on the presence or absence of discontinuities in the indices for the divisions between such divided coding units.

[0523] In one embodiment, the video decoding device 100 can determine whether or not a plurality of coding units, which have been divided and determined from the current coding unit, have been divided into a specific division pattern, based on the value of an index used to distinguish them. Referring to Figure 21, the video decoding device 100 can divide a rectangular first coding unit 2110, where the height is longer than the width, and determine an even number of coding units 2112a, 2112b, or an odd number of coding units 2114a, 2114b, 2114c. The video decoding device 100 can use an index (PID) that represents each coding unit to distinguish each of the plurality of coding units. In one embodiment, the PID is also obtained from a sample at a predetermined position in each coding unit (for example, the upper left sample).

[0524] In one embodiment, the video decoding device 100 can determine a coding unit at a predetermined position among coding units that have been divided and determined using an index for the division of coding units. In one embodiment, if the division pattern information relating to a rectangular first coding unit 2110, which is taller than its width, indicates that it is divided into three coding units, the video decoding device 100 can divide the first coding unit 2110 into three coding units 2114a, 2114b, and 2114c. The video decoding device 100 can assign an index relating to each of the three coding units 2114a, 2114b, and 2114c. The video decoding device 100 can compare the indices relating to each coding unit in order to determine the middle coding unit among the odd number of coding units that have been divided. Based on the index of the coding units, the video decoding device 100 can determine the coding unit 2114b, which has an index corresponding to the middle value among the indices, as the coding unit at the middle position among the coding units that have been divided and determined from the first coding unit 2110. In one embodiment, the video decoder 100 can determine the index for the division of the divided coding units based on the size ratio between the coding units if the coding units are not the same size as each other. Referring to Figure 21, the coding unit 2114b generated by the division of the first coding unit 2110 has the same width as the other coding units 2114a and 2114c, but its height is different, and the coding unit 2114b is twice the height of the coding units 2114a and 2114c. In that case, if the index (PID) of the coding unit 2114b located in the middle is 1, then the coding unit 2114c located in the next order will also have an index that is 2, which is 3. In such cases, where the index increases uniformly but by different increments, the video decoder 100 can determine that the division has been made into multiple coding units, including coding units that have different sizes from the other coding units.In one embodiment, if the division configuration information indicates that the code is divided into an odd number of coding units, the video decoding device 100 can divide the current coding unit into a configuration in which a coding unit at a predetermined position (for example, the middle coding unit) is of a different size from the other coding units. In this case, the video decoding device 100 can use an index (PID) related to the coding unit to determine the middle coding unit having a different size. However, the aforementioned index, the size of the coding unit at the predetermined position to be determined, or the position are specified for the purpose of explaining one embodiment and should not be interpreted as being limited to these, but rather as various indices, coding unit positions, and sizes being used.

[0525] In one embodiment, the video decoding device 100 can utilize a predetermined data unit in which the recursive division of the encoding unit begins.

[0526] Figure 22 illustrates how, in one embodiment, multiple encoding units are determined from multiple predetermined data units contained in a picture.

[0527] In one embodiment, a predetermined data unit is also defined as a data unit from which an encoded unit begins to recursively divide using at least one of block-type information and division-type information. That is, it corresponds to the highest-depth encoded unit used in the process of determining multiple encoded units that currently divide the picture. Hereinafter, for explanatory convenience, such a predetermined data unit will be referred to as a reference data unit.

[0528] In one embodiment, the reference data unit can have a predetermined size and shape. In one embodiment, the reference coding unit may contain MxN samples, where M and N may be the same as each other, or they may be integers expressed as powers of 2. That is, the reference data unit can have a square or non-square shape, and may then be divided into an integer number of coding units.

[0529] In one embodiment, the video decoding device 100 can divide the current picture into a plurality of reference data units. In another embodiment, the video decoding device 100 can divide the current picture into a plurality of reference data units using the division information related to each reference data unit. Such a division process of reference data units corresponds to a division process using a quad-tree structure.

[0530] In one embodiment, the video decoding device 100 can predetermine the minimum size that a reference data unit currently contained in a picture can have. This allows the video decoding device 100 to determine reference data units of various sizes that are larger than or equal to the minimum size, and, using the determined reference data units as a basis, to determine at least one encoding unit using block shape information and segmentation shape information.

[0531] Referring to Figure 22, the video decoding device 100 can utilize a square-shaped reference coding unit 2200, or it can utilize a non-square-shaped reference coding unit 2202. In one embodiment, the shape and size of the reference coding unit can also be determined by a variety of data units (e.g., sequence, picture, slice, slice segment, maximum coding unit, etc.) that include at least one reference coding unit.

[0532] In one embodiment, the acquisition unit 105 of the video decoding device 100 can acquire at least one of the following from the bitstream for each of the various data units: information relating to the shape of the reference coding unit and information relating to the size of the reference coding unit. The process for determining at least one coding unit included in the square-shaped reference coding unit 2200 was explained through the process of dividing the current coding unit 300 in Figure 10, and the process for determining at least one coding unit included in the non-square-shaped reference coding unit 2200 was explained through the process of dividing the current coding unit 1100 or 1150 in Figure 11, so a detailed explanation will be omitted.

[0533] In one embodiment, the video decoding device 100 can use an index to identify the size and shape of a reference coding unit in order to determine the size and shape of a reference coding unit based on a subset of data units predetermined based on predetermined conditions. That is, the acquisition unit 105 can acquire only an index for identifying the ...

Claims

1. A method for decoding video, This is the stage where the first extended reference block is acquired within the first reference picture, and the second extended reference block is acquired within the second reference picture. The first extended reference block includes a first reference block and a first portion extended from the first reference block. The aforementioned second extended reference block comprises a second reference block and a second portion extended from the aforementioned second reference block, and The step is to determine the displacement vector of a pixel group that currently includes at least one pixel adjacent to the inside of the block boundary, using the gradient value of at least one reference pixel in the first extended reference block and the gradient value of at least one reference pixel in the second extended reference block. The first portion of the first extended reference block is used to calculate the gradient value of at least one reference pixel within the first reference block. The second portion of the second extended reference block is used to calculate the gradient value of at least one reference pixel within the second reference block, and includes steps and A step of obtaining a predicted pixel value of the current block by performing optical flow-based compensation on the current block using the gradient value of the at least one reference pixel in the first reference block, the gradient value of the at least one reference pixel in the second reference block, and the displacement vector of the pixel group, The step includes restoring the current block based on the predicted pixel values, Currently, pictures are divided into multiple maximum encoding units. One of the multiple maximum coding units is divided into at least one coding unit that includes the current block. How to decrypt a video.

2. A video encoding method, This step involves obtaining a predicted pixel value for the current block by performing optical flow-based compensation on the current block using the gradient value of at least one reference pixel in the first reference block, the gradient value of at least one reference pixel in the second reference block, and the displacement vector of the pixel group in the current block. The pixel group includes a step that includes at least one pixel adjacent to the boundary inside the current block, The step of generating a bitstream that includes the result of encoding the current block based on the predicted pixel value, A first extended reference block is acquired within a first reference picture, a second extended reference block is acquired within a second reference picture, the first extended reference block includes the first reference block and a first portion extended from the first reference block, and the second extended reference block includes the second reference block and a second portion extended from the second reference block. The displacement vector of the pixel group is determined using the gradient value of at least one reference pixel in the first extended reference block and the gradient value of at least one reference pixel in the second extended reference block. The first portion of the first extended reference block is used to calculate the gradient value of the at least one reference pixel within the first reference block. The second portion of the second extended reference block is used to calculate the gradient value of the at least one reference pixel within the second reference block. Currently, a picture is divided into multiple maximum encoding units. One of the multiple maximum coding units is divided into at least one coding unit that includes the current block. Video encoding method.

3. A method for transmitting the bitstream generated by the video encoding method described in claim 2.

Images