Video decoding method and apparatus and video encoding method and apparatus

Abstract

A video decoding method and apparatus, as well as a video encoding method and apparatus, are disclosed. The decoding method includes determining a displacement vector per unit time in the horizontal or vertical direction of a plurality of pixels of the current block, including pixels adjacent to the boundary of the current block, using the values ​​of reference pixels included in first and second reference blocks, without using stored values ​​related to pixels outside the boundaries of the first and second reference blocks; performing block-unit and pixel-group-unit motion compensation of the current block based on the gradient values ​​of a first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group in the current block, the gradient values ​​of a second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel values ​​of the first corresponding reference pixel, the pixel values ​​of the second corresponding reference pixel, and the displacement vector per unit time in the horizontal or vertical direction of the current pixel to obtain a predicted block of the current block. The pixel group may include at least one pixel.

Description

[0001] This application is a divisional application of the invention patent application filed on January 4, 2018, with application number "201880017180.0" and titled "Video Decoding Method and Apparatus and Video Encoding Method and Apparatus". Technical Field

[0002] This disclosure relates to a video decoding method and apparatus, as well as a video encoding method and apparatus. More specifically, this disclosure relates to a video decoding method and apparatus, as well as a video encoding method and apparatus, for performing inter-frame prediction in a bidirectional motion prediction mode. Background Technology

[0003] With the development and widespread availability of hardware capable of reproducing and storing high-resolution or high-definition video content, the demand for video codecs that can effectively encode or decode such content is increasing. Based on existing video codecs, video is encoded using tree-structured coding units according to a limited coding scheme.

[0004] Image data in the spatial domain is transformed into coefficients in the frequency domain using a frequency transform. To perform fast computation in the frequency transform, the video codec divides the image into blocks of predetermined size and performs a Discrete Cosine Transform (DCT) on each block, thereby encoding the frequency coefficients in each block unit. The coefficients in the frequency domain are easier to compress than the image data in the spatial domain. Because the image pixel values ​​in the spatial domain are represented as prediction errors by the video codec via inter-frame or intra-frame prediction, a large amount of data can be transformed to zero when performing a frequency transform on the prediction errors. The video codec reduces the amount of data by changing sequentially repeating data into smaller data sizes. Summary of the Invention

[0005] Technical issues

[0006] According to various embodiments, the predicted pixel values ​​of the current block can be generated not only using the pixel values ​​of a first reference block from a first reference frame and the pixel values ​​of a second reference block from a second reference frame in bidirectional motion prediction mode, but also using the first gradient value of the first reference block and the second gradient value of the second reference block. Therefore, a predicted block similar to the original block can be generated, thereby improving encoding and decoding efficiency.

[0007] When performing pixel-group unit motion compensation, the pixel values ​​of the first reference block, the pixel values ​​of the second reference block, the first gradient value of the first reference block, and the second gradient value of the second reference block can be used to determine the displacement vector of the current block in the horizontal or vertical direction. Specifically, to determine the displacement vector of the current pixel in the current block in the horizontal or vertical direction, not only the pixel values ​​and gradient values ​​of the first reference pixel corresponding to the current pixel in the first reference block and the second reference pixel in the second reference block are used, but also the pixel values ​​and gradient values ​​of neighboring pixels included in a window of predetermined size with respect to the first and second reference pixels are used. Therefore, when the current pixel is located at the boundary, there is a problem that additional memory access is required because when the neighboring pixels of the reference pixel corresponding to the current pixel are located outside the reference block, the pixel values ​​and gradient values ​​of the pixels located outside the reference block must be referenced.

[0008] According to various embodiments, the displacement vector of the current block in the horizontal or vertical direction is determined by referring only to the pixel values ​​and gradient values ​​of pixels located in the reference block, and not to the pixel values ​​and gradient values ​​stored for pixels located outside the reference block, thereby minimizing the number of memory accesses.

[0009] According to various embodiments, in order to determine the gradient value of a reference pixel in the horizontal or vertical direction, the horizontal or vertical gradient filter and interpolation filter, which are inputs of integer pixel values, are not used. Instead, the pixel value of the pixel at a fractional pixel unit position is determined by applying an interpolation filter to the pixel value of the integer pixel value, and the gradient value of the reference pixel in the horizontal or vertical direction is determined by applying a horizontal or vertical gradient filter to the pixel value of the pixel at a fractional pixel unit position. The gradient filter has a relatively short length, which minimizes the number of complex multiplication operations.

[0010] According to various embodiments, motion compensation is performed on a pixel-by-pixel basis, which minimizes the number of complex multiplication operations compared to motion compensation performed on a pixel-by-pixel basis.

[0011] A computer-readable recording medium is provided, wherein a program for performing methods according to various embodiments is recorded on the computer-readable recording medium.

[0012] The technical problems of the various embodiments are not limited to the features described above, and other undescribed technical problems will be clearly understood by those skilled in the art in light of the following description.

[0013] Technical solution

[0014] The technical problem addressed in this disclosure is not limited to the features described above, and other undescribed technical problems will be clearly understood by those skilled in the art in light of the following description.

[0015] According to various embodiments, a video decoding method may include: obtaining motion prediction mode information about a current block in a current frame from a bitstream; when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtaining information about a first motion vector and information about a second motion vector from the bitstream, wherein the first motion vector indicates a first reference block in a first reference frame, and the second motion vector indicates a second reference block in a second reference frame; determining, in the horizontal or vertical direction, a plurality of pixels of the current block, including pixels adjacent to the boundary of the current block, in relation to reference pixels included in the first and second reference blocks, using values ​​of reference pixels included in the first and second reference blocks, without using stored values ​​of pixels located outside the boundaries of the first and second reference blocks. The displacement vector per unit time in the direction; by using the gradient value in the horizontal or vertical direction of the first corresponding reference pixel in the first reference block corresponding to the current pixel included in the current pixel group in the current block, the gradient value in the horizontal or vertical direction of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction per unit time, block unit motion compensation and pixel group unit motion compensation are performed on the current block to obtain the prediction block of the current block; information about the residual block of the current block is obtained from the bit stream; and the current block is reconstructed based on the prediction block and the residual block, wherein the current pixel group includes at least one pixel.

[0016] In the video decoding method according to various embodiments, the step of obtaining the prediction block of the current block may further include: calculating the gradient value of the first corresponding reference pixel or the second corresponding reference pixel in the horizontal or vertical direction by applying a gradient filter in the horizontal or vertical direction to the pixel value of the pixel at the fractional position in the first reference block or the second reference block, wherein the gradient filter is a 5-tap filter, and the pixels at the fractional position are all pixels with a fractional value in at least one of the horizontal or vertical components of the coordinates indicating the position of the pixel.

[0017] In video decoding methods according to various embodiments, the pixel value of a pixel at a fractional position in a first reference block or a second reference block can be calculated by applying a horizontal or vertical interpolation filter to the pixel value of a pixel at an integer position.

[0018] In video decoding methods according to various embodiments, the size of the current pixel group can be determined based on the minimum of the height and width values ​​of the current block.

[0019] In the video decoding method according to various embodiments, the displacement vector per unit time in the horizontal or vertical direction of the current pixel group can be a displacement vector per unit time determined by using the following: the pixel value of the first corresponding reference pixel in the first corresponding reference pixel group corresponding to the current pixel group in the first reference frame and the pixel value of the neighboring pixels of the first corresponding reference pixel; the pixel value of the second corresponding reference pixel in the second corresponding reference pixel group corresponding to the current pixel group in the second reference frame and the pixel value of the neighboring pixels of the second corresponding reference pixel; the gradient value; the first frame sequence count (POC) difference between the first reference frame and the current frame; and the second POC difference between the second reference frame and the current frame.

[0020] In a video decoding method according to various embodiments, the step of determining the displacement vector of the plurality of pixels of the current block per unit time in the horizontal or vertical direction may include: when a first corresponding reference pixel or a second corresponding reference pixel is a boundary neighbor pixel adjacent to the interior of the boundary of a first reference block or a second reference block, deriving the pixel value and gradient value of a pixel located outside the boundary of the first reference block or the second reference block using the pixel value of the boundary neighbor pixel, and determining the displacement vector of the current pixel per unit time in the horizontal or vertical direction based on the pixel value and gradient value of the boundary neighbor pixel and the pixel value and gradient value of the pixel located outside the boundary of the current block, wherein the pixel value and gradient value of the pixel located outside the boundary of the current block are derived using the pixel value of the boundary neighbor pixel.

[0021] In a video decoding method according to various embodiments, the step of determining the displacement vector of the plurality of pixels of the current block per unit time in the horizontal or vertical direction may include: calculating a value for the current pixel using pixel values ​​of a first corresponding reference pixel included in a first reference block, pixel values ​​of a second corresponding reference pixel included in a second reference block, gradient values ​​of the first corresponding reference pixel, gradient values ​​of the second corresponding reference pixel, a first POC difference between a first reference frame and the current frame, and a second POC difference between the second reference frame and the current frame; calculating a value for neighboring pixels using pixel values ​​of first corresponding neighboring pixels of the first corresponding reference pixel, pixel values ​​of second corresponding neighboring pixels of the second corresponding reference pixel, gradient values ​​of the first corresponding neighboring pixels, gradient values ​​of the second corresponding neighboring pixels, the first POC difference between the first reference frame and the current frame, and the second POC difference between the second reference frame and the current frame; calculating a weighted average value for the current pixel required to calculate the displacement vector per unit time in the horizontal or vertical direction using the value for the current pixel, the value for neighboring pixels, and weights; and determining the displacement vector of the current pixel per unit time in the horizontal or vertical direction using the calculated weighted average value for the current pixel.

[0022] In video decoding methods according to various embodiments, the weighted average value for the current pixel may be a value calculated by applying exponential smoothing techniques to the values ​​of pixels included in the first and second reference blocks in the vertical and horizontal directions.

[0023] According to various embodiments, a video decoding apparatus may include: a receiver configured to obtain motion prediction mode information about a current block in a current frame from a bitstream; when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtain from the bitstream information about a first motion vector of a first reference block indicating the current block in a first reference frame and information about a second motion vector of a second reference block indicating the current block in a second reference frame; and obtain from the bitstream information about a residual block of the current block; and an inter-frame predictor configured to determine the current block's inclusion within the boundaries of the current block by using values ​​of reference pixels included in the first and second reference blocks, without using stored values ​​of pixels located outside the boundaries of the first and second reference blocks. The current block is obtained by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block using the displacement vectors of multiple pixels of neighboring pixels in the horizontal or vertical direction per unit time, and by using the gradient values ​​of the first corresponding reference pixels in the first reference block corresponding to the current pixel included in the current pixel group in the current block in the horizontal or vertical direction, the gradient values ​​of the second corresponding reference pixels in the second reference block corresponding to the current pixel in the horizontal or vertical direction, the pixel values ​​of the first corresponding reference pixels, the pixel values ​​of the second corresponding reference pixels, and the displacement vectors of the current pixel in the horizontal or vertical direction per unit time; and a decoder is configured to reconstruct the current block based on the prediction block and the residual block, wherein the current pixel group includes at least one pixel.

[0024] In a video decoding apparatus according to various embodiments, an inter-frame predictor may be configured to calculate the gradient value of a first corresponding reference pixel or a second corresponding reference pixel in the horizontal or vertical direction by applying a gradient filter in the horizontal or vertical direction to the pixel value of a pixel at a fractional position in a first reference block or a second reference block, wherein the gradient filter is a 5-tap filter, and the pixels at the fractional positions are pixels having a fractional value in at least one of the horizontal or vertical components of the coordinates indicating the position of the pixel.

[0025] In a video decoding apparatus according to various embodiments, the displacement vector per unit time in the horizontal or vertical direction of the current pixel group can be a displacement vector per unit time determined by an inter-frame predictor using the following: the pixel value of a first corresponding reference pixel in a first corresponding reference pixel group corresponding to the current pixel group in a first reference frame and the pixel values ​​of the neighboring pixels of the first corresponding reference pixel; the pixel value of a second corresponding reference pixel in a second corresponding reference pixel group corresponding to the current pixel group in a second reference frame and the pixel values ​​of the neighboring pixels of the second corresponding reference pixel; a gradient value; a first frame sequence count (POC) difference between the first reference frame and the current frame; and a second POC difference between the second reference frame and the current frame.

[0026] In a video decoding apparatus according to various embodiments, an inter-frame predictor may be configured to: calculate a value for a current pixel using pixel values ​​of a first corresponding reference pixel included in a first reference block, pixel values ​​of a second corresponding reference pixel included in a second reference block, gradient values ​​of the first corresponding reference pixel, gradient values ​​of the second corresponding reference pixel, a first POC difference between a first reference frame and the current frame, and a second POC difference between the second reference frame and the current frame; calculate a value for neighboring pixels using pixel values ​​of first corresponding neighboring pixels of the first corresponding reference pixel, pixel values ​​of second corresponding neighboring pixels of the second corresponding reference pixel, gradient values ​​of the first corresponding neighboring pixels, gradient values ​​of the second corresponding neighboring pixels, the first POC difference between the first reference frame and the current frame, and the second POC difference between the second reference frame and the current frame; calculate a weighted average value for the current pixel used to calculate the displacement vector per unit time in the horizontal or vertical direction using the value for the current pixel, the value for neighboring pixels, and weights; and determine the displacement vector of the current pixel per unit time in the horizontal or vertical direction using the calculated weighted average value for the current pixel.

[0027] According to various embodiments, a video coding method may include: obtaining a prediction block, a first motion vector, and a second motion vector of the current block by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block; and generating a bitstream including information about the first motion vector and the second motion vector, as well as motion prediction mode information indicating that the motion prediction mode of the current block is a bidirectional motion prediction mode, wherein the pixel group for pixel-group-unit motion compensation includes at least one pixel, wherein the first motion vector is a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, wherein the second motion vector is a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block, wherein the prediction block of the current block is obtained by using the first reference block corresponding to the current block. The current pixel group includes the gradient value of the first corresponding reference pixel in the horizontal or vertical direction corresponding to the current pixel, the gradient value of the second corresponding reference pixel in the horizontal or vertical direction corresponding to the current pixel in the second reference block, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction per unit time. The current pixel group includes the gradient value of the first corresponding reference pixel in the horizontal or vertical direction corresponding to the current pixel in the second reference block, the gradient value of the second corresponding reference pixel in the horizontal or vertical direction corresponding to the current pixel in the second reference block, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction per unit time. The displacement vector of the current pixel group including the pixels adjacent to the boundary of the current pixel in the horizontal or vertical direction per unit time is determined by using the values ​​of the reference pixels included in the first reference block and the second reference block, without using the stored values ​​of the pixels located outside the boundary of the first reference block and the second reference block.

[0028] According to various embodiments, a video encoding apparatus may include: an inter-frame predictor configured to obtain a predicted block of the current block, a first motion vector, and a second motion vector by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block; and a bitstream generator configured to generate a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating that the motion prediction mode of the current block is a bidirectional motion prediction mode, wherein the pixel group for pixel-group-unit motion compensation includes at least one pixel, wherein the first motion vector is a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, and the second motion vector is a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block, wherein the predicted block of the current block is obtained by using a first... The current block is obtained by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block using the gradient values ​​of a first corresponding reference pixel in a reference block corresponding to the current pixel in the current pixel group of the current block, the gradient values ​​of a second corresponding reference pixel in a second reference block corresponding to the current pixel in the horizontal or vertical direction, the pixel values ​​of the first corresponding reference pixel, the pixel values ​​of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction per unit time. The displacement vectors of multiple pixels in the current block, including pixels adjacent to the boundary of the current block, in the horizontal or vertical direction per unit time are determined using values ​​of reference pixels included in the first and second reference blocks, without using stored values ​​of pixels located outside the boundary of the first and second reference blocks.

[0029] A computer-readable recording medium is provided, the program being used to perform methods according to various embodiments.

[0030] Beneficial effects

[0031] According to various embodiments, by performing inter-frame prediction on the current block using the gradient value of the reference block of the reference frame in bidirectional motion prediction mode to predict a value similar to the value of the original block of the current block, encoding and decoding efficiency can be improved. Attached Figure Description

[0032] Figure 1a A block diagram of a video decoding device according to various embodiments is shown.

[0033] Figure 1b A flowchart of a video decoding method according to various embodiments is shown.

[0034] Figure 1c A block diagram of a video encoding apparatus according to various embodiments is shown.

[0035] Figure 1d A flowchart of a video encoding method according to various embodiments is shown.

[0036] Figure 1e A block diagram of an image decoder according to various embodiments is shown.

[0037] Figure 1f A block diagram of an image encoder according to various embodiments is shown.

[0038] Figure 2 This is a reference diagram illustrating block-based bidirectional motion prediction and compensation processing according to an embodiment.

[0039] Figures 3a to 3c This is a reference diagram illustrating the process of performing pixel group unit motion compensation according to an embodiment.

[0040] Figure 4 This is a reference diagram used to describe the process of calculating gradient values ​​in the horizontal and vertical directions according to an embodiment.

[0041] Figure 5 This is a reference diagram used to describe the process of calculating gradient values ​​in the horizontal and vertical directions according to another embodiment.

[0042] Figures 6a to 6b This is a diagram illustrating the process of determining gradient values ​​in the horizontal and vertical directions using a 1D filter according to an embodiment.

[0043] Figures 7a to 7e This is a table showing the filter coefficients of a filter used to determine the pixel value at a fractional pixel position and the gradient values ​​in the horizontal and vertical directions, according to an embodiment.

[0044] Figure 8a This is a reference diagram used to describe the process of determining the horizontal and vertical displacement vectors of a pixel according to an embodiment.

[0045] Figure 8b This is a reference diagram used to describe the process of determining the horizontal and vertical displacement vectors of a pixel group according to an embodiment.

[0046] Figure 8c This is a reference diagram used to describe the process of determining the horizontal and vertical displacement vectors of a pixel according to an embodiment.

[0047] Figure 8d This is a reference diagram used to describe the process of determining the horizontal and vertical displacement vectors of a pixel group according to an embodiment.

[0048] Figure 9aThis is a diagram illustrating the process of adding an offset value after performing filtering and determining the gradient value in the horizontal or vertical direction by performing descaling, according to an embodiment.

[0049] Figure 9b This is a diagram illustrating a process according to another embodiment of adding an offset value after performing filtering and determining the gradient value in the horizontal or vertical direction by performing descaling.

[0050] Figure 9c This is a diagram used to describe the necessary range for determining the horizontal and vertical displacement vectors during the process of performing pixel-unit motion compensation on the current block.

[0051] Figures 9d to 9e This is an illustration of the range of regions used in the process of performing pixel-unit motion compensation, according to various embodiments.

[0052] Figure 9f This is a diagram illustrating the process of determining the horizontal and vertical displacement vectors without enlarging the reference block.

[0053] Figures 9g to 9i This is a diagram illustrating the process of determining the horizontal and vertical displacement vectors without enlarging the reference block, according to other embodiments.

[0054] Figure 9j This is a diagram illustrating the process of determining the horizontal and vertical displacement vectors for each pixel by applying exponential smoothing techniques in the vertical and horizontal directions, without expanding the block, by referring to the pixel values ​​and gradient values ​​of a reference block.

[0055] Figure 9k This is a diagram illustrating a process, according to an embodiment, of determining the pixel values ​​and gradient values ​​of reference pixels in a reference block in the horizontal and vertical directions using a filter to perform motion compensation on the current block.

[0056] Figure 9l This is a diagram illustrating a process, according to another embodiment, of determining the pixel values ​​and gradient values ​​of reference pixels in a reference block in the horizontal and vertical directions using a filter to perform motion compensation on the current block.

[0057] Figure 10 The process of determining at least one coding unit by dividing the current coding unit is illustrated according to an embodiment.

[0058] Figure 11 The process of determining at least one coding unit by dividing non-square coding units according to an embodiment is illustrated.

[0059] Figure 12 The process of dividing the coding unit based on at least one of the block shape information and the partition shape information according to an embodiment is illustrated.

[0060] Figure 13 A method for determining a predetermined coding unit from an odd number of coding units according to an embodiment is shown.

[0061] Figure 14 The following illustration shows the order in which the plurality of coding units are processed when a plurality of coding units are determined by dividing the current coding unit, according to an embodiment.

[0062] Figure 15 The process described in the embodiment is to determine that the current coding unit will be divided into an odd number of coding units when the coding units cannot be processed in a predetermined order.

[0063] Figure 16 The process of determining at least one coding unit by dividing a first coding unit, according to an embodiment, is illustrated.

[0064] Figure 17 The illustration shows a limitation on the shape into which the second coding unit can be divided when a second coding unit with a non-square shape is determined by dividing the first coding unit and the second coding unit satisfies predetermined conditions, according to an embodiment.

[0065] Figure 18 The process of dividing a square coding unit according to an embodiment is illustrated when the division shape information indicates that the square coding unit is not divided into four square coding units.

[0066] Figure 19 This illustrates that, according to an embodiment, the processing order among multiple coding units can be changed based on the process of dividing the coding units.

[0067] Figure 20 The process of determining the depth of a coding unit as the shape and size of the coding unit change is illustrated according to an embodiment when coding units are recursively divided to determine multiple coding unit determinations.

[0068] Figure 21 The diagram illustrates a depth that can be determined based on the shape and size of the coding unit, according to an embodiment, and a partial index (PID) used to distinguish the coding unit.

[0069] Figure 22 The illustration shows how multiple encoding units are determined based on multiple predetermined data units included in a screen, according to an embodiment.

[0070] Figure 23A processing block is shown according to an embodiment, which serves as a unit for determining the order of reference coding units included in a frame.

[0071] Best practice

[0072] According to various embodiments, a video decoding method may include: obtaining motion prediction mode information about a current block in a current frame from a bitstream; when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtaining information about a first motion vector and information about a second motion vector from the bitstream, wherein the first motion vector indicates a first reference block in a first reference frame, and the second motion vector indicates a second reference block in a second reference frame; determining, in the horizontal or vertical direction, a plurality of pixels of the current block, including pixels adjacent to the boundary of the current block, in relation to reference pixels included in the first and second reference blocks, without using stored values ​​of pixels located outside the boundaries of the first and second reference blocks. The displacement vector upwards per unit time; performing block unit motion compensation and pixel group unit motion compensation on the current block using the gradient value in the horizontal or vertical direction of the first corresponding reference pixel in the first reference block corresponding to the current pixel in the current pixel group in the current block, the gradient value in the horizontal or vertical direction of the second corresponding reference pixel in the second reference block corresponding to the current pixel, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction per unit time to obtain the prediction block of the current block; obtaining information about the residual block of the current block from the bitstream; and reconstructing the current block based on the prediction block and the residual block, wherein the current pixel group includes at least one pixel.

[0073] According to various embodiments, a video decoding apparatus includes: a receiver configured to obtain motion prediction mode information about a current block in a current frame from a bitstream; when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtain from the bitstream information about a first motion vector indicating a first reference block in a first reference frame and information about a second motion vector indicating a second reference block in a second reference frame; and obtain from the bitstream information about a residual block of the current block; and an inter-frame predictor configured to determine the current block's internal adjacency by using values ​​of reference pixels included in the first and second reference blocks without using stored values ​​of pixels located outside the boundaries of the first and second reference blocks. The current block is obtained by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block using the gradient values ​​of a first corresponding reference pixel in a first reference block corresponding to the current pixel in the current pixel group in the current block, the gradient values ​​of a second corresponding reference pixel in a second reference block corresponding to the current pixel in the current block, the pixel values ​​of the first corresponding reference pixel, the pixel values ​​of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction per unit time; and a decoder is configured to reconstruct the current block based on the prediction block and the residual block, wherein the current pixel group includes at least one pixel.

[0074] According to various embodiments, a video coding method includes: obtaining a prediction block, a first motion vector, and a second motion vector of the current block by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block; and generating a bitstream including information about the first motion vector and the second motion vector, as well as motion prediction mode information indicating that the motion prediction mode of the current block is a bidirectional motion prediction mode, wherein the pixel group for pixel-group-unit motion compensation includes at least one pixel, wherein the first motion vector is a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, wherein the second motion vector is a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block, wherein the prediction block of the current block is obtained by using the first reference block corresponding to the current block. The current pixel group includes the gradient value of the first corresponding reference pixel in the horizontal or vertical direction corresponding to the current pixel, the gradient value of the second corresponding reference pixel in the horizontal or vertical direction corresponding to the current pixel in the second reference block, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction per unit time. The current pixel group includes the gradient value of the first corresponding reference pixel in the horizontal or vertical direction corresponding to the current pixel in the second reference block, the gradient value of the second corresponding reference pixel in the horizontal or vertical direction corresponding to the current pixel in the second reference block, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction per unit time. The displacement vector of the current pixel group including the pixels adjacent to the boundary of the current pixel in the horizontal or vertical direction per unit time is determined by using the values ​​of the reference pixels included in the first reference block and the second reference block, without using the stored values ​​of the pixels located outside the boundary of the first reference block and the second reference block.

[0075] According to various embodiments, a video encoding apparatus includes: an inter-frame predictor configured to obtain a predicted block of the current block, a first motion vector, and a second motion vector by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block; and a bitstream generator configured to generate a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating that the motion prediction mode of the current block is a bidirectional motion prediction mode, wherein the pixel group for pixel-group-unit motion compensation includes at least one pixel, wherein the first motion vector is a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, and the second motion vector is a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block, wherein the predicted block of the current block is obtained by using a first... The current block is obtained by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block using the gradient values ​​of a first corresponding reference pixel in a reference block corresponding to the current pixel in the current pixel group of the current block, the gradient values ​​of a second corresponding reference pixel in a second reference block corresponding to the current pixel in the horizontal or vertical direction, the pixel values ​​of the first corresponding reference pixel, the pixel values ​​of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction per unit time. The displacement vectors of multiple pixels in the current block, including pixels adjacent to the boundary of the current block, in the horizontal or vertical direction per unit time are determined using values ​​of reference pixels included in the first and second reference blocks, without using stored values ​​of pixels located outside the boundary of the first and second reference blocks.

[0076] According to various embodiments, a computer-readable recording medium may include a recorded program for performing the method.

[0077] Specific implementation methods

[0078] In the following text, "image" may refer to a still image of the video or to a moving image that is part of the video itself.

[0079] In the following text, "sample point" can refer to a sampling location assigned to an image and is data that is the target of processing. For example, pixels in an image in the spatial domain can be samples.

[0080] In the following text, "current block" may refer to a block of the image that will be encoded or decoded.

[0081] Figure 1a A block diagram of a video decoding device according to various embodiments is shown.

[0082] Video decoding device 100 according to various embodiments includes a receiver 105, an inter-frame predictor 110, and a reconstructor 125.

[0083] The receiver 105 receives a bit stream including prediction mode information about the current block, motion prediction mode information about the current block, and motion vector information.

[0084] The acquirer 105 can obtain prediction mode information, motion prediction mode information, and motion vector information about the current block from the received bitstream. In addition, the acquirer 105 can obtain a reference frame index indicating a reference frame in a previously decoded frame from the received bitstream.

[0085] When the prediction mode of the current block is inter-frame prediction mode, the inter-frame predictor 110 performs inter-frame prediction on the current block. That is, the inter-frame predictor 110 can generate the predicted pixel value of the current block by using at least one of the frames that have been decoded before the current frame including the current block. For example, when the motion prediction mode of the current block is bidirectional motion prediction mode, the inter-frame predictor 110 can generate the predicted pixel value of the current block by using two frames that have been decoded before the current frame. In other words, when the motion prediction mode information received from the bitstream indicates bidirectional motion prediction mode, the inter-frame predictor 110 can generate the predicted pixel value of the current block by using two frames that have been decoded before the current frame.

[0086] The inter-frame predictor 110 may include a block unit motion compensator 115 and a pixel group unit motion compensator 120.

[0087] The block unit motion compensator 115 can perform block unit motion compensation on the current block.

[0088] The block unit motion compensator 115 can determine at least one reference frame among the previously decoded frames by using a reference frame index obtained from the bitstream. In this regard, the reference frame index can refer to the reference frame index with respect to each prediction direction, including the L0 direction and the L1 direction. Here, the reference frame index with respect to the L0 direction refers to the index indicating the reference frame among the frames included in the L0 reference frame list, and the reference frame index with respect to the L1 direction refers to the index indicating the reference frame among the frames included in the L1 reference frame list.

[0089] The block-unit motion compensator 115 can determine the reference block of the current block located in at least one reference frame by using motion vector information received from the bit stream. In this regard, the corresponding block in the reference frame that corresponds to the current block in the current frame can be the reference block. That is, the block-unit motion compensator 115 can determine the reference block of the current block by using a motion vector from the current block indicating the reference block. In this regard, the motion vector means a vector indicating the change between the reference coordinates of the current block in the current frame and the reference coordinates of the reference block in the reference frame. For example, when the top-left coordinate of the current block is (1, 1) and the top-left coordinate of the reference block in the reference frame is (3, 3), the motion vector can be (2, 2).

[0090] In this regard, motion vector information may include differences in motion vectors, and the block unit motion compensator 115 may reconstruct motion vectors using a prediction factor for the motion vectors and the differences in motion vectors obtained from the bitstream, and may determine reference blocks of the current block located in at least one reference frame by using the reconstructed motion vectors. The differences in motion vectors may indicate differences in motion vectors related to various prediction directions, including the L0 direction and the L1 direction. In this regard, the differences in motion vectors with respect to the L0 direction refer to the differences in motion vectors indicating reference blocks in reference frames included in the L0 reference frame list, and the differences in motion vectors with respect to the L1 direction refer to the differences in motion vectors indicating reference blocks in reference frames included in the L1 reference frame list.

[0091] The block-unit motion compensator 115 can perform block-unit motion compensation on the current block by using the pixel values ​​of a reference block. Alternatively, the block-unit motion compensator 115 can perform block-unit motion compensation on the current block by using the value of a reference pixel in the reference block that corresponds to the current pixel in the current block. In this respect, the reference pixel can be a pixel included in the reference block and can correspond to the current pixel in the current block.

[0092] The block-unit motion compensator 115 can perform block-unit motion compensation on the current block by using multiple reference blocks included in multiple reference frames. For example, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the block-unit motion compensator 115 can identify two reference frames among the previously encoded frames, and can identify two reference blocks included in the two reference frames.

[0093] The block unit motion compensator 115 can perform block unit motion compensation on the current block using the pixel values ​​of two reference pixels from two reference blocks. The block unit motion compensator 115 can generate a block unit motion compensation value by performing block unit motion compensation on the current block using the average or weighted sum of the pixel values ​​of the two reference pixels.

[0094] The reference position of a reference block can be an integer pixel position, but is not limited to this, and therefore can be a fractional pixel position. In this respect, an integer pixel can refer to a pixel whose positional component is an integer and is located at an integer pixel position. A fractional pixel can refer to a pixel whose positional component is a fraction and is located at a fractional pixel position.

[0095] For example, when the top-left coordinate of the current block is (1, 1) and the motion vector is (2.5, 2.5), the top-left coordinate of the reference block in the reference frame could be (3.5, 3.5). In this respect, the position of the fractional pixel can be determined in 1 / 4 or 1 / 16 pixel element (pel) units. However, this disclosure is not limited to this, and therefore the position of the fractional pixel can be determined according to various fractional pel units.

[0096] When the reference position of the reference block is the position of a fraction of pixels, the block unit motion compensator 115 can apply an interpolation filter to a first neighboring region of a first pixel in the first reference block indicated by a first motion vector and a second neighboring region of a second pixel in the second reference block indicated by a second motion vector, thereby generating the pixel value of the first pixel and the pixel value of the second pixel.

[0097] In other words, the value of a reference pixel in a reference block can be determined by using the pixel values ​​of neighboring pixels whose predetermined direction component is an integer. In this respect, the predetermined direction can be either horizontal or vertical.

[0098] For example, the block unit motion compensator 115 can determine a reference pixel value as the result of filtering by using an interpolation filter to filter pixel values ​​of pixels whose predetermined direction components are integers, and can determine the value of block unit motion compensation for the current block by using the reference pixel value. The value of block unit motion compensation can be determined by using the value of a weighted sum or the average value of the reference pixels. In this respect, the interpolation filter can be an M-tap interpolation filter based on Discrete Cosine Transform (DCT). The coefficients of the DCT-based M-tap interpolation filter can be derived from DCT and Inverse Discrete Cosine Transform (IDCT). In this respect, the coefficients of the interpolation filter can be filter coefficients scaled to integer coefficients to reduce real number operations when performing filtering. In this respect, the interpolation filter can be a one-dimensional (1D) interpolation filter in the horizontal or vertical direction. For example, when the position of a pixel is represented as orthogonal x and y coordinate components, the horizontal direction can refer to the direction parallel to the x-axis. The vertical direction can refer to the direction parallel to the y-axis.

[0099] The block unit motion compensator 115 can first perform filtering on the pixel values ​​at integer positions using a 1D vertical interpolation filter, and then determine the reference pixel values ​​at fractional pixel positions by performing filtering on the values ​​generated by the filtering using a 1D horizontal interpolation filter.

[0100] When using scaled filter coefficients, the value generated by filtering can be greater than that generated using unscaled filter coefficients. Therefore, the block unit motion compensator 115 can perform descaling on the value generated by filtering.

[0101] The block unit motion compensator 115 can perform descaling by filtering the pixel values ​​at integer positions using a 1D vertical interpolation filter. In this regard, the descaling step may include a right bit shift by the number of descaling bits. The number of descaling bits can be determined based on the bit depth of the sample points in the input image. For example, the number of descaling bits can be a value obtained by subtracting 8 from the bit depth of the sample points.

[0102] Furthermore, the block unit motion compensator 115 can perform filtering on the pixel values ​​at integer positions using a 1D vertical interpolation filter, and can perform filtering on the values ​​generated by the filtering using a 1D horizontal interpolation filter, and then perform descaling. In this regard, the descaling step may include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the number of scaling bits of the 1D vertical interpolation filter, the number of scaling bits of the 1D horizontal interpolation filter, and the bit depth of the sample. For example, when the number of scaling bits p of the 1D vertical interpolation filter is 6, the number of scaling bits q of the 1D horizontal interpolation filter is 6, and the bit depth of the sample is b, the number of descaling bits can be p + q + 8 - b, i.e., 20 - b.

[0103] When the block unit motion compensator 115 performs filtering on pixels where the predetermined direction component is an integer using a 1D interpolation filter, and then performs a bit shift to the right according to the number of descaling bits, rounding errors may occur. Therefore, the block unit motion compensator 115 can perform filtering on pixels where the predetermined direction component is an integer using a 1D interpolation filter, add it to the offset value, and then perform descaling. In this respect, the offset value can be 2^(number of descaling bits - 1).

[0104] The pixel group unit motion compensator 120 can generate pixel group unit motion compensated values ​​by performing pixel group unit motion compensation on the current block. When the motion prediction mode of the current block is bidirectional motion prediction mode, the pixel group unit motion compensator 120 can generate pixel group unit motion compensated values ​​by performing pixel group unit motion compensation on the current block.

[0105] The pixel group unit motion compensator 120 generates pixel group unit motion compensated values ​​by performing pixel group unit motion compensation on the current block based on the optical flow of the pixel group of the first reference frame and the second reference frame. This will be referred to in the following description. Figure 3a Describe optical flow.

[0106] The pixel group unit motion compensator 120 generates pixel group unit motion compensated values ​​by performing pixel group unit motion compensation on pixel groups included in a reference block of the current block. A pixel group may include at least one pixel. For example, a pixel group may be a pixel. Optionally, a pixel group may be a plurality of pixels including two or more pixels. A pixel group may be a plurality of pixels included in a block of size K×K (where K is an integer).

[0107] The pixel group unit motion compensator 120 can determine a pixel group and perform pixel group unit motion compensation on the current block based on the determined pixel group.

[0108] The pixel group unit motion compensator 120 can determine the size of the pixel group based on the size of the current block. For example, the pixel group unit motion compensator 120 can determine the height and width of the pixel group as the maximum value selected from 2 and the value obtained by dividing the minimum value of the width and height of the current block by 8.

[0109] Compared to performing motion compensation pixel-by-pixel at high image resolution, the pixel group unit motion compensator 120 can perform motion compensation on a pixel group basis, including multiple pixels, thus reducing encoding / decoding complexity. Furthermore, compared to performing motion compensation pixel-by-pixel at high frame rates, the pixel group unit motion compensator 120 can perform motion compensation on a pixel group basis, including multiple pixels, thus reducing encoding / decoding complexity.

[0110] The acquirer 105 can acquire pixel group size information included in the bitstream. When the size of the pixel group is K×K, the pixel group size information can be information indicating the height or width K. The pixel group size information can be included in the high-level syntax carrier.

[0111] The pixel group unit motion compensator 120 can determine at least one pixel group region comprising pixels with similar pixel values ​​among a plurality of pixels included in a pixel group, and can perform motion compensation on the pixel group region. In this respect, the pixel group region comprising pixels with similar pixel values ​​is likely to be the same object and its motion is similar to each other; therefore, the pixel group unit motion compensator 120 can perform further detailed motion compensation on a pixel group basis.

[0112] Pixel group unit motion compensation is performed when the motion prediction mode information indicates bidirectional motion prediction mode. However, even in this case, pixel group unit motion compensation is not always performed; it can be performed selectively.

[0113] The pixel group unit motion compensator 120 can determine a reference pixel group in the reference block that corresponds to the current pixel group in the current block, and can determine the gradient value of the reference pixel group. For example, the pixel group unit motion compensator 120 can determine the gradient value of the reference pixel group by using the gradient value of at least one pixel value included in the reference pixel group.

[0114] The pixel group unit motion compensator 120 can generate pixel group unit motion compensated values ​​by performing pixel group unit motion compensation on the current block using the gradient values ​​of the reference pixel group.

[0115] The pixel group unit motion compensator 120 can apply a filter to a first neighboring region of a first pixel group in a pixel group indicated by a first motion vector and a second neighboring region of a second pixel group in a pixel group indicated by a second motion vector, thereby generating gradient values ​​for the first pixel group and the second pixel group.

[0116] The pixel group unit motion compensator 120 can determine the pixel value and gradient value of a pixel in a first window of a predetermined size including the first pixel group for a first pixel group in a first reference frame, and can determine the pixel value and gradient value of a pixel in a second window of a predetermined size including the second pixel group for a second pixel group in a second reference frame.

[0117] The pixel group unit motion compensator 120 determines the displacement vector of the current pixel group per unit time using the pixel values ​​and gradient values ​​of pixels in a first window and pixels in a second window. In this regard, the value of the displacement vector of the current pixel group per unit time can be adjusted by a regularization parameter. The regularization parameter is a parameter used to prevent errors from occurring when an ill-conditioned displacement vector of the current pixel group per unit time is determined for performing motion compensation on the pixel group. The pixel group unit motion compensator 120 can perform pixel group unit motion compensation on the current block based on the regularization parameter regarding the displacement vector per unit time in the horizontal or vertical direction. The regularization parameter will be referred to below. Figure 8a It is described in the description.

[0118] The pixel group unit motion compensator 120 can perform pixel group unit motion compensation on the current block by using the displacement vector of the current pixel group per unit time and the gradient value of the reference pixel.

[0119] The reference position of the reference block can be an integer pixel position, but is not limited to this, and therefore can be a fraction of a pixel position.

[0120] When the reference position of the reference block is the position of a fractional pixel, the gradient value of the reference pixel in the reference block can be determined by using the pixel values ​​of neighboring pixels whose predetermined direction components are integers.

[0121] For example, the pixel group unit motion compensator 120 can perform filtering on the pixel values ​​of neighboring pixels whose predetermined direction components are integers using a gradient filter, and the gradient value of a reference pixel can be determined as the filtered result. In this regard, the filter coefficients of the gradient filter can be determined using coefficients predetermined for a DCT-based interpolation filter. The coefficients of the gradient filter can be filter coefficients scaled to integers to reduce real-number operations when performing filtering.

[0122] In this regard, the gradient filter can be a 1D horizontal gradient filter or a 1D vertical gradient filter.

[0123] The pixel group unit motion compensator 120 can determine the gradient value in the horizontal or vertical direction with respect to the reference pixel by performing filtering on neighboring pixels with corresponding integer direction components using a 1D horizontal gradient filter or a 1D vertical gradient filter.

[0124] For example, the pixel group unit motion compensator 120 can determine the gradient value in the horizontal direction with respect to the reference pixel by using a 1D horizontal gradient filter to filter the pixels that are adjacent to the reference pixel and are located in the horizontal direction starting from the pixel whose horizontal component is an integer.

[0125] When the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are fractions), the pixel group unit motion compensator 120 can perform filtering on the pixel at the (x, y) position and the pixels whose vertical components are integers and located in the vertical direction starting from the pixel at the (x, y) position by using a 1D vertical direction interpolation filter, and the pixel value at (x, y+β) can be determined as the result of the filtering.

[0126] The pixel group unit motion compensator 120 can perform filtering on the pixel value at position (x, y+β) and the pixels whose horizontal components are integers and located in the horizontal direction starting from position (x, y+β) by using a horizontal gradient filter, and can determine the gradient value at position (x+α, y+β) as the result of the filtering.

[0127] The order in which the 1D gradient filter and the 1D interpolation filter are used is not restricted. In the above description, firstly, interpolated values ​​in the vertical direction are generated by filtering pixels at integer positions using a 1D vertical interpolation filter, and then the interpolated values ​​in the vertical direction are filtered using a 1D horizontal gradient filter. However, it is also possible to first generate interpolated values ​​in the horizontal direction by filtering pixels at integer positions using a 1D horizontal gradient filter, and then filter the interpolated values ​​in the horizontal direction using a 1D vertical interpolation filter.

[0128] The method for determining the gradient value in the horizontal direction at the position (x+α, y+β) by the pixel group unit motion compensator 120 has been described in detail above. The pixel group unit motion compensator 120 determines the gradient value in the vertical direction at the position (x+α, y+β) in a similar manner to the method for determining the gradient value in the horizontal direction, and therefore its detailed description is omitted.

[0129] In the above description, a method for determining the gradient value at the position of a fractional pixel using a 1D gradient filter and a 1D interpolation filter, performed by the pixel group unit motion compensator 120, has been specifically described. However, this disclosure is not limited thereto, and the gradient filter and interpolation filter can also be used to determine the gradient value at the position of an integer pixel. Even though the pixel value of an integer pixel can be determined without using an interpolation filter, in order to be consistent with the processing in fractional pixels, the pixel value of the integer pixel can still be determined by performing filtering on the integer pixel and neighboring pixels whose predetermined direction components are integers using an interpolation filter. For example, the interpolation filter coefficients at the integer pixel can be {0, 0, 64, 0, 0}, and since the interpolation filter coefficients related to neighboring integer pixels are 0, filtering can be performed using only the pixel value of the current integer pixel, and as a result, the pixel value of the current integer pixel can be determined by performing filtering on the current integer pixel and neighboring integer pixels using an interpolation filter.

[0130] The pixel group unit motion compensator 120 can perform descaling after filtering pixels at integer positions using a 1D vertical interpolation filter. In this regard, the descaling step may include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the bit depth of the sample. Furthermore, the number of descaling bits can be determined based on specific input data in the block.

[0131] For example, the number of descaling bits can be obtained by subtracting 8 from the bit depth of the sample.

[0132] The pixel group unit motion compensator 120 can perform descaling after filtering the values ​​generated via descaling using a horizontal gradient filter. Similarly, the descaling step may include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the number of scaling bits of the 1D vertical interpolation filter, the number of scaling bits of the 1D horizontal gradient filter, and the bit depth of the sample. For example, when the number of scaling bits p of the 1D interpolation filter is 6, the number of scaling bits q of the 1D gradient filter is 4, and the bit depth of the sample is b, the number of descaling bits can be p + q + 8 - b, i.e., 18 - b.

[0133] When the pixel group unit motion compensator 120 performs a right bit shift only on the value generated by filtering according to the number of descaling bits after filtering, rounding errors may occur. This allows the pixel group unit motion compensator 120 to add the offset value to the value generated by filtering before performing descaling. Here, the offset value can be 2^(number of descaling bits - 1).

[0134] Inter-frame predictor 110 can generate predicted pixel values ​​for the current block using a value with pixel group-unit motion compensation and a value with block-unit motion compensation. For example, inter-frame predictor 110 can generate predicted pixel values ​​for the current block by adding the value with block-unit motion compensation and the value with pixel group-unit motion compensation. In this respect, the value with block-unit motion compensation refers to the value generated by performing block-unit motion compensation, and the value with pixel group-unit motion compensation refers to the value generated by performing pixel group-unit motion compensation, wherein the value with block-unit motion compensation can be an average or weighted sum with respect to a reference pixel, and the value with pixel group-unit motion compensation can be a value determined based on the displacement vector of the current pixel per unit time and the gradient value of the reference pixel.

[0135] The pixel group unit motion compensator 120 can obtain a shift value for descaling after performing interpolation or gradient calculation based on at least one of the bit depth of the sample, the input range of the filter used in the interpolation calculation, or the coefficients of the filter. The pixel group unit motion compensator 120 can perform interpolation or gradient calculation with respect to pixels included in the first reference block and the second reference block by using the shift value for descaling, and then perform descaling.

[0136] The inter-frame predictor 110 can use motion vectors while performing block-unit motion compensation and can store motion vectors. In this respect, the unit of motion vectors can be a 4×4 block. When storing motion vectors after performing block-unit motion compensation, the size of the motion vector storage unit can not be 4×4, but can be a block of various sizes (e.g., an R×R block; where R is an integer). In this respect, the motion vector storage unit can be a block larger than 4×4. For example, the motion vector storage unit can be a 16×16 block.

[0137] When performing pixel group unit motion compensation, the size of the target block to which pixel group unit motion compensation is performed can be increased based on the size of the current block, the size of the window, and the length of the interpolation filter. The reason why the size of the target block is increased based on the size of the window compared to the size of the current block is that, for pixels located at the edges of the current block, pixel group unit motion compensation for the current block is performed based on pixels located at the edges of the current block and neighboring pixels using the window.

[0138] Therefore, in order to reduce the number of memory accesses and the number of multiplication operations, when performing pixel group unit motion compensation by using a window, the pixel group unit motion compensator 120 can adjust the position of pixels in the window that are off-center from the current block to the position of pixels adjacent to the current block. Thus, the pixel value and gradient value at the adjusted pixel position can be determined, thereby reducing the number of memory accesses and the number of multiplication operations.

[0139] The pixel group unit motion compensator 120 can determine the gradient value of a reference pixel, which is necessary for pixel group unit motion compensation, without using the pixel values ​​of pixels at integer positions. That is, the pixel group unit motion compensator 120 can calculate the gradient value of a first corresponding reference pixel in the first reference block in the horizontal or vertical direction, or the gradient value of a second corresponding reference pixel in the second reference block, by applying a horizontal gradient filter or a vertical gradient filter to the pixel values ​​of pixels at fractional positions. In this regard, the length of the gradient filter can be 5. The coefficients of the gradient filter can have coefficients symmetrical with respect to the center coefficient of the filter. The pixel at the fractional position can be a pixel having a fractional value in at least one of the horizontal and vertical components indicating the pixel's position.

[0140] The pixel values ​​of pixels at fractional positions in the first or second reference block can be calculated by applying a horizontal or vertical interpolation filter to the pixel values ​​of pixels at integer positions.

[0141] The displacement vector per unit time in the horizontal or vertical direction of the current pixel group can be the displacement vector per unit time determined by using the following: the pixel value of the first corresponding reference pixel in the first corresponding reference pixel group corresponding to the current pixel group in the first reference frame, the pixel value of the second corresponding reference pixel in the second corresponding reference pixel group in the second reference frame, and the pixel values ​​of the neighboring pixels of the first and second corresponding reference pixels, the gradient value, the first frame sequence count (POC) difference between the first reference frame and the current frame, and the second POC difference between the second reference frame and the current frame.

[0142] When the first corresponding reference pixel or the second corresponding reference pixel is a boundary neighbor pixel adjacent to the boundary inside the first reference block or the second reference block, the pixel group unit motion compensator 120 can derive the pixel value of the pixel located outside the boundary of the first reference block or the second reference block by using the pixel value of the boundary neighbor pixel.

[0143] The pixel group unit motion compensator 120 can determine the displacement vector of the current block per unit time in the horizontal or vertical direction based on the pixel values ​​of boundary-adjacent pixels and the pixel values ​​of pixels located outside the boundary of the current block, wherein the pixel values ​​of pixels located outside the boundary of the current block are derived using the pixel values ​​of boundary-adjacent pixels. That is, pixels located outside the boundary may exist in the pixels included in the window about boundary-adjacent pixels, and in this respect, the pixel values ​​and gradient values ​​of pixels located outside the boundary may be pixel values ​​and gradient values ​​of pixels not included in memory, but rather pixel values ​​and gradient values ​​of pixels derived from boundary-adjacent pixels.

[0144] The pixel group unit motion compensator 120 can calculate the value of the current pixel using the pixel values ​​of the first corresponding reference pixels included in the first reference block, the pixel values ​​of the second corresponding reference pixels included in the second reference block, the gradient values ​​of the first and second corresponding reference pixels, the first POC difference between the first reference frame and the current frame, and the second POC difference between the second reference frame and the current frame. In other words, the value of the current pixel can be the result of a function based on the pixel values, gradient values, and POC differences between each reference frame and the current frame.

[0145] The pixel group unit motion compensator 120 can calculate values ​​for neighboring pixels using the pixel values ​​of the first corresponding neighboring pixels of the first corresponding reference pixel, the gradient values ​​of the first corresponding neighboring pixels, the pixel values ​​of the second corresponding neighboring pixels of the second corresponding reference pixel, the gradient values ​​of the second corresponding neighboring pixels, the first POC difference between the first reference frame and the current frame, and the second POC difference between the second reference frame and the current frame. In other words, the values ​​for neighboring pixels can be the result of a function based on the pixel values ​​of the corresponding reference pixels of each reference frame, the gradient values ​​of the corresponding reference pixels of each reference frame, and the POC difference between each reference frame and the current frame.

[0146] The pixel group unit motion compensator 120 can calculate a weighted average value for the displacement vector per unit time in the horizontal direction for the current pixel by using values ​​about the current pixel, values ​​about neighboring pixels, and weights. In this regard, the weights can be determined based on the following: the distance between the current pixel and neighboring pixels, the distance between the pixel and the boundary of the block, the number of pixels located outside the boundary, or whether the pixel is located inside or outside the boundary.

[0147] The weighted average value for the current pixel can be a value calculated by applying exponential smoothing techniques in both the vertical and horizontal directions to the values ​​of pixels included in the first and second reference blocks. The value calculated for the current pixel by applying exponential smoothing techniques in both the vertical and horizontal directions to the values ​​of pixels included in the first and second reference blocks can be such that the weight of the current pixel's value is maximized, and the weight of the values ​​of the current pixel's neighboring pixels decreases exponentially with respect to their distance from the current pixel.

[0148] The pixel group unit motion compensator 120 can determine the displacement vector of the current pixel in the horizontal or vertical direction per unit time by using the weighted average value of the current pixel.

[0149] Reconstructor 125 can obtain the residual block of the current block from the bitstream and can reconstruct the current block using the prediction block and the residual block. For example, reconstructor 125 can generate the pixel values ​​of the reconstructed block by summing the pixel values ​​of the residual block of the current block from the bitstream with the pixel values ​​of the prediction block of the current block.

[0150] The video decoding device 100 may include an image decoder (not shown), and the image decoder (not shown) may include an acquirer 105, an inter-frame predictor 110, and a reconstructor 125. (Refer to...) Figure 1e To describe the image decoder.

[0151] Figure 1b A flowchart of a video decoding method according to various embodiments is shown.

[0152] In operation S105, the video decoding device 100 can obtain motion prediction mode information about the current block in the current frame from the bitstream. The video decoding device 100 can receive a bitstream including motion prediction mode information about the current block in the current frame, and can obtain the motion prediction mode information about the current block from the received bitstream. The video decoding device 100 can obtain the motion prediction mode information about the current block from the bitstream, and can determine the prediction mode of the current block based on the motion prediction mode information. In this regard, when the prediction mode of the current block is an inter-frame prediction mode, the video decoding device 100 can obtain the motion prediction mode information about the current block.

[0153] For example, the video decoding device 100 can determine the prediction mode of the current block as an inter-frame prediction mode based on motion prediction mode information about the current block. When the prediction mode of the current block is an inter-frame prediction mode, the video decoding device 100 can obtain motion prediction mode information about the current block from the bitstream.

[0154] In operation S110, when the motion prediction mode information indicates bidirectional motion prediction mode, the video decoding device 100 can obtain from the bitstream a first motion vector indicating the first reference block in the first reference frame and a second motion vector indicating the second reference block in the second reference frame.

[0155] In other words, the video decoding device 100 can receive a bitstream including information about a first motion vector and a second motion vector, and can obtain the first motion vector and the second motion vector from the received bitstream. The video decoding device 100 can obtain a reference frame index from the bitstream, and can determine a first reference frame and a second reference frame among a plurality of previously decoded frames based on the reference frame index.

[0156] In operation S115, the video decoding device 100 can determine the displacement vector of a pixel in the current block (which includes pixels adjacent to the boundary of the current block) in the horizontal or vertical direction by using values ​​of reference pixels included in the first and second reference blocks, without using stored values ​​of pixels located outside the boundaries of the first and second reference blocks. In this regard, the stored values ​​of pixels located outside the boundaries of the first and second reference blocks, as well as the values ​​of reference pixels included in the first and second reference blocks, can be pixel values ​​of the relevant pixels, or gradient values ​​of the relevant pixels in the horizontal direction, or gradient values ​​of the relevant pixels in the vertical direction. Optionally, the stored values ​​of pixels located outside the boundaries of the first and second reference blocks, as well as the values ​​of reference pixels included in the first and second reference blocks, can be values ​​determined by using pixel values ​​of the relevant pixels or gradient values ​​of the relevant pixels.

[0157] In operation S120, the video decoding device 100 can obtain the prediction block of the current block by performing block unit motion compensation and pixel group unit motion compensation on the current block using the gradient value of the first reference pixel in the first reference block corresponding to the current pixel included in the current pixel group in the current block in the horizontal or vertical direction, the gradient value of the second corresponding reference pixel in the second reference block corresponding to the current pixel in the horizontal or vertical direction, the pixel value of the first corresponding reference pixel, the pixel value of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction.

[0158] In other words, the video decoding device 100 can generate a predicted block for the current block by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block based on a first motion vector and a second motion vector. The video decoding device 100 can perform block-unit motion compensation on the current block using pixel values ​​of a first reference block indicated by the first motion vector and pixel values ​​of a second reference block indicated by the second motion vector. Furthermore, the video decoding device 100 can perform pixel-group-unit motion compensation on the current pixel group by using gradient values ​​in the horizontal or vertical direction of at least one first corresponding reference pixel in the first reference block corresponding to at least one pixel included in the current pixel group in the current block, gradient values ​​in the horizontal or vertical direction of at least one second corresponding reference pixel in the second reference block corresponding to said at least one pixel, pixel values ​​of the first corresponding reference pixel, pixel values ​​of the second corresponding reference pixel, and the displacement vector of the current pixel in the horizontal or vertical direction.

[0159] The video decoding device 100 can obtain the predicted block of the current block by using the block-unit motion-compensated value generated by performing block-unit motion compensation on the current block and the pixel-group-unit motion-compensated value generated by performing pixel-group-unit motion compensation on the current pixel group.

[0160] During operation S125, the video decoding device 100 can obtain information about the residual block of the current block from the bit stream.

[0161] During operation S130, the video decoding device 100 can reconstruct the current block based on the prediction block and the residual block. That is, the video decoding device 100 can generate the pixel values ​​of the reconstructed block of the current block by summing the pixel values ​​of the residual block indicated by the information about the residual block of the current block and the predicted pixel values ​​of the prediction block.

[0162] Figure 1c A block diagram of a video encoding apparatus according to various embodiments is shown.

[0163] Video encoding apparatus 150 according to various embodiments includes an inter-frame predictor 155 and a bitstream generator 170.

[0164] Inter-frame predictor 155 performs inter-frame prediction on the current block by referencing various blocks based on rate-distortion cost. That is, inter-frame predictor 155 can generate the predicted pixel values ​​for the current block by using at least one of the frames that have been encoded before the current frame that includes the current block.

[0165] The inter-frame predictor 155 may include a block unit motion compensator 160 and a pixel group unit motion compensator 165.

[0166] The block unit motion compensator 160 can generate a block unit motion compensated value by performing block unit motion compensation on the current block.

[0167] The block unit motion compensator 160 can identify at least one reference frame among previously decoded frames, and can identify the reference block of the current block located in the at least one reference frame.

[0168] The block-unit motion compensator 160 can generate block-unit motion-compensated values ​​by performing block-unit motion compensation on the current block using the pixel values ​​of a reference block. Alternatively, the block-unit motion compensator 160 can generate block-unit motion-compensated values ​​by performing block-unit motion compensation on the current block using the values ​​of reference pixels in the reference block that correspond to the current pixel of the current block.

[0169] The block-unit motion compensator 160 generates a block-unit motion-compensated value by performing block-unit motion compensation on the current block using multiple reference blocks included in multiple reference frames. For example, when the motion prediction mode of the current block indicates a bidirectional prediction mode, the block-unit motion compensator 160 can identify two reference frames among the previously encoded frames, and can identify two reference blocks included in the two reference frames. In this respect, bidirectional prediction is not limited to using inter-frame prediction of frames displayed before and after the current block, and can indicate the use of inter-frame prediction of two frames encoded before the current block, regardless of the display order.

[0170] The block unit motion compensator 160 can generate a block unit motion compensated value by performing block unit motion compensation on the current block using the values ​​of two reference pixels from two reference blocks. Alternatively, the block unit motion compensator 160 can generate a block unit motion compensated value by performing block unit motion compensation on the current block using the average pixel value or a weighted sum of the two reference pixels.

[0171] The block unit motion compensator 160 can output a reference frame index indicating a reference frame in a previously encoded frame used for motion compensation of the current block.

[0172] The block unit motion compensator 160 can determine the motion vector of a reference block whose starting point is the current block and whose ending point is the current block, and can output the motion vector. The motion vector can be interpreted as a vector indicating the displacement between the reference coordinates of the current block in the current frame and the reference coordinates of the reference block in the reference frame. For example, when the top-left coordinate of the current block is (1, 1) and the top-left coordinate of the reference block in the reference frame is (3, 3), the motion vector could be (2, 2).

[0173] The reference position of the reference block can be an integer pixel position, but is not limited to this, and therefore can be a fractional pixel position. In this respect, the position of the fractional pixel can be determined in 1 / 4 pixel units or 1 / 16 pixel units. However, this disclosure is not limited to this, and therefore the position of the fractional pixel can be determined according to various fractional pixel units.

[0174] For example, when the reference position of the reference block is (1.5, 1.5) and the coordinates of the top-left corner of the current block are (1, 1), the motion vector can be (0.5, 0.5). When the motion vector is determined in 1 / 4 or 1 / 16 pixel units to indicate the reference position of the reference block (i.e., the position of a fractional pixel), an integer motion vector can be determined by scaling the motion vector, and the reference position of the reference block can be determined by using an enlarged motion vector. When the reference position of the reference block is a fractional pixel position, the position of the reference pixel of the reference block can also be a fractional pixel position. Therefore, the pixel value at the fractional pixel position in the reference block can be determined by using the pixel value of a neighboring pixel whose component in a predetermined direction is an integer.

[0175] For example, the block-unit motion compensator 160 can determine the pixel value of a reference pixel at a fractional pixel position by filtering the pixel values ​​of neighboring pixels whose components in a predetermined direction are integers using an interpolation filter, and can determine the motion-compensated value for the current block on a block-by-block basis using the pixel values ​​of the reference pixels. In this regard, the interpolation filter can be a DCT-based M-tap interpolation filter. The coefficients of the DCT-based M-tap interpolation filter can be derived from DCT and IDCT. In this regard, the coefficients of the interpolation filter can be filter coefficients scaled to integer coefficients to reduce real-number operations during filtering.

[0176] In this regard, the interpolation filter can be a 1D interpolation filter in the horizontal or vertical direction.

[0177] The block unit motion compensator 160 determines the pixel value of a reference pixel at a fractional pixel location by first filtering the neighboring integer pixels using a 1D vertical interpolation filter, and then filtering the filtered value using a 1D horizontal interpolation filter. When using scaled filter coefficients, the block unit motion compensator 160 can perform descaling on the filtered value after filtering the pixel at the integer location using a 1D vertical interpolation filter. In this regard, the descaling step may include a right bit shift by the number of descaling bits. The number of descaling bits can be determined based on the bit depth of the sample. For example, the number of descaling bits can be a value obtained by subtracting 8 from the bit depth of the sample.

[0178] Furthermore, the block unit motion compensator 160 can perform filtering on pixels where the horizontal component is an integer using a 1D vertical interpolation filter, and then perform a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the number of scaling bits with respect to the coefficients of the 1D vertical interpolation filter, the number of scaling bits with respect to the coefficients of the 1D horizontal interpolation filter, and the bit depth of the sample.

[0179] When the block unit motion compensator 160 performs a right bit shift based solely on the number of descaling bits, rounding errors may occur. Therefore, the block unit motion compensator 160 can perform filtering on pixels where the predetermined direction component is an integer using a 1D interpolation filter in the predetermined direction, add the offset value to the filtered value, and then perform descaling on the value with the added offset value. In this respect, the offset value can be 2^(number of descaling bits - 1).

[0180] The foregoing has described the process of determining the number of descaling bits based on the bit depth of the samples after filtering using a 1D vertical interpolation filter. However, this disclosure is not limited to this, and therefore the number of descaling bits can be determined not only based on the bit depth of the samples but also based on the number of bits scaled for the interpolation filter coefficients. In other words, the number of descaling bits can be determined based on both the bit depth of the samples and the number of bits scaled for the interpolation filter coefficients, without overflow, taking into account the size of the registers used during filtering and the size of the buffer storing the values ​​generated during filtering.

[0181] The pixel group unit motion compensator 165 can generate pixel group unit motion compensated values ​​by performing pixel group unit motion compensation on the current block. In this regard, when the motion prediction mode is a bidirectional motion prediction mode, the pixel group unit motion compensator 165 can generate pixel group unit motion compensated values ​​by performing pixel group unit motion compensation on the current block.

[0182] The pixel group unit motion compensator 165 can generate pixel group unit motion compensated values ​​by performing pixel group unit motion compensation on the current block using the gradient values ​​of pixels included in the reference block of the current block.

[0183] The pixel group unit motion compensator 165 can obtain the gradient value of the first pixel and the gradient value of the second pixel by applying a filter to the first neighboring region of the first pixel in the first reference block of the first reference frame and the second neighboring region of the second pixel in the second reference block of the second reference frame.

[0184] The pixel group unit motion compensator 165 can determine the pixel value and gradient value of a pixel in a first window of a predetermined size that includes the first reference pixel in a first reference frame, and can determine the pixel value and gradient value of a pixel in a second window of a predetermined size that includes the second reference pixel in a second reference frame. The pixel group unit motion compensator 165 can determine the displacement vector of the current pixel per unit time by using the pixel value and gradient value of the pixel in the first window and the pixel value and gradient value of the pixel in the second window.

[0185] The pixel group unit motion compensator 165 generates pixel group unit motion compensated values ​​by performing pixel group unit motion compensation on the current block using the displacement vector per unit time and the gradient value of the reference pixel.

[0186] The reference position of a reference pixel can be an integer pixel position, but is not limited to this, and therefore can be a fraction of a pixel position.

[0187] When the reference position of the reference block is the position of a fractional pixel, the gradient value of the reference pixel in the reference block can be determined by using the pixel value of the neighboring pixel whose predetermined direction component is an integer.

[0188] For example, the pixel group unit motion compensator 165 can determine the gradient value of the reference pixel as the result of filtering the pixel values ​​of neighboring pixels whose predetermined direction components are integers using a gradient filter. In this regard, the filter coefficients of the gradient filter can be determined by using coefficients predetermined for a DCT-based interpolation filter.

[0189] The coefficients of a gradient filter can be filter coefficients scaled to integer values ​​to reduce real-value operations when performing filtering. In this respect, a gradient filter can be a 1D horizontal gradient filter or a 1D vertical gradient filter.

[0190] The pixel group unit motion compensator 165 can determine the gradient value in the horizontal or vertical direction with respect to a reference pixel by performing filtering on neighboring pixels whose corresponding direction components are integers using a 1D horizontal gradient filter or a 1D vertical gradient filter.

[0191] For example, the pixel group unit motion compensator 165 can determine the pixel value of a pixel whose vertical component is a fraction by performing filtering on pixels whose vertical component is an integer among pixels in the vertical direction starting from an integer pixel adjacent to the reference pixel.

[0192] Regarding pixels located in another column adjacent to an integer pixel adjacent to a reference pixel, the pixel group unit motion compensator 165 can determine the pixel value at the fractional pixel position in the other column by performing filtering on the neighboring integer pixels in the vertical direction using a 1D vertical interpolation filter. In this respect, the position of the pixel in the other column can be the position of the fractional pixel in the vertical direction and the position of the integer pixel in the horizontal direction.

[0193] In other words, when the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are fractions), the pixel group unit motion compensator 165 can determine the pixel value at position (x, y+β) by performing filtering on the neighboring integer pixels in the vertical direction starting from position (x, y) using a vertical direction interpolation filter.

[0194] The pixel group unit motion compensator 165 can determine the gradient value at the (x+α, y+β) position in the horizontal direction by performing filtering on the pixel value at the (x, y+β) position and the pixel values ​​of pixels whose horizontal components are integers in the horizontal direction starting from the pixel value at the (x, y+β) position.

[0195] The order in which the 1D gradient filter and the 1D interpolation filter are used is not restricted. As described above, a vertically interpolated value can first be generated by filtering pixels at integer positions using a 1D vertical interpolation filter, and then the vertically interpolated value can be filtered using a 1D horizontal gradient filter. However, the invention is not limited to this. Therefore, a horizontally gradient-filtered value can first be generated by filtering pixels at integer positions using a 1D horizontal gradient filter, and then the horizontally gradient-filtered value can be filtered using a 1D vertical interpolation filter.

[0196] The gradient value in the horizontal direction at the position (x+α, y+β) determined by the pixel group unit motion compensator 165 has been described in detail above.

[0197] The pixel group unit motion compensator 165 can determine the gradient value in the vertical direction at the position (x+α, y+β) in a manner similar to determining the gradient value in the horizontal direction.

[0198] The pixel group unit motion compensator 165 determines the gradient value of the reference pixel in the vertical direction by performing filtering on neighboring integer pixels in the vertical direction, starting from the integer pixel adjacent to the reference pixel, using a 1D vertical gradient filter. For a pixel adjacent to the reference pixel and located in another column, the pixel group unit motion compensator 165 determines the gradient value in the vertical direction with respect to the pixel adjacent to the reference pixel and located in the other column by performing filtering on neighboring integer pixels in the vertical direction using a 1D vertical gradient filter. In this regard, the position of the pixel can be the position of a fractional pixel in the vertical direction and the position of an integer pixel in the horizontal direction.

[0199] In other words, when the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are fractions), the pixel group unit motion compensator 165 can determine the gradient value in the vertical direction at the position (x, y+β) by performing filtering on the neighboring integer pixels in the vertical direction starting from the position (x, y) using a vertical direction gradient filter.

[0200] The pixel group unit motion compensator 165 can determine the gradient value in the vertical direction at the position (x+α, y+β) by performing filtering on the gradient value at the position (x, y+β) and the gradient values ​​of the neighboring integer pixels in the horizontal direction starting from the position (x, y+β) using a horizontal interpolation filter.

[0201] The order in which the 1D gradient filter and the 1D interpolation filter are used is not restricted. As described above, a gradient-filtered value in the vertical direction can first be generated by filtering pixels at integer positions using a 1D vertical gradient filter, and then the gradient-filtered value in the vertical direction can be filtered using a 1D horizontal interpolation filter. However, this disclosure is not limited to this. Therefore, a horizontally interpolated value can first be generated by filtering pixels at integer positions using a 1D horizontal interpolation filter, and then the horizontally interpolated value can be filtered using a 1D vertical gradient filter.

[0202] The pixel group unit motion compensator 165 has been described in detail above as using gradient filters and interpolation filters to determine gradient values ​​at fractional pixel locations. However, this disclosure is not limited thereto; therefore, gradient filters and interpolation filters can be used to determine gradient values ​​at integer pixel locations.

[0203] In the case of integer pixels, although the pixel value of an integer pixel can be determined without using an interpolation filter, in order to achieve the same processing as in fractional pixels, filtering can be performed on the integer pixel and its neighboring integer pixels using an interpolation filter. For example, the interpolation filter coefficients at the integer pixel can be {0, 0, 64, 0, 0}, and since the interpolation filter coefficients related to neighboring integer pixels are 0, filtering can be performed using only the pixel value of the current integer pixel. As a result, the pixel value of the current integer pixel can be determined by performing filtering on the current integer pixel and its neighboring integer pixels using an interpolation filter.

[0204] When using scaled filter coefficients, the pixel group unit motion compensator 165 can perform filtering on pixels at integer positions using a 1D horizontal gradient filter, and then perform descaling on the filtered values. In this regard, the descaling step may include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the bit depth of the sample. For example, the number of descaling bits can be a value obtained by subtracting 8 from the bit depth of the sample.

[0205] The pixel group unit motion compensator 165 can perform descaling by filtering pixels whose vertical components are integers using a vertical interpolation filter. In this regard, the descaling step may include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the scaling bits of the 1D vertical interpolation filter, the scaling bits of the 1D horizontal gradient filter, and the bit depth of the sample.

[0206] When the pixel group unit motion compensator 165 performs a right bit shift only according to the number of descaling bits, rounding errors may occur. Therefore, after filtering using a 1D interpolation filter, the offset value can be added to the filtered value, and descaling can be performed on the value with the added offset value. In this regard, the offset value can be 2^(bit shifts - 1).

[0207] The inter-frame predictor 110 can use motion vectors while performing block-unit motion compensation and can store motion vectors. In this respect, the unit of motion vectors can be a 4×4 block. When storing motion vectors after performing block-unit motion compensation, the motion vector storage unit may not be 4×4, but may be a block of various sizes (e.g., an R×R block; where R is an integer). In this respect, the motion vector storage unit can be a block larger than 4×4. For example, the motion vector storage unit can be a 16×16 block.

[0208] When performing pixel group unit motion compensation, the size of the target block to which pixel group unit motion compensation is performed can be increased based on the size of the current block, the size of the window, and the length of the interpolation filter. The reason why the size of the target block is increased based on the size of the window compared to the size of the current block is that, for pixels located at the edges of the current block, pixel group unit motion compensation for the current block is performed based on pixels located at the edges of the current block and neighboring pixels using the window.

[0209] Therefore, in order to reduce the number of memory accesses and the number of multiplication operations, when performing pixel group unit motion compensation by using a window, the pixel group unit motion compensator 120 adjusts the position of pixels in the window that are off-center from the current block to the position of pixels adjacent to the current block. Thus, the pixel value and gradient value at the adjusted pixel position can be determined, thereby reducing the number of memory accesses and the number of multiplication operations.

[0210] The pixel group unit motion compensator 120 can determine the gradient value of a reference pixel, which is necessary for pixel group unit motion compensation, without using the pixel values ​​of pixels at integer positions. That is, the pixel group unit motion compensator 120 can calculate the gradient value of a first corresponding reference pixel in the first reference block in the horizontal or vertical direction, or the gradient value of a second corresponding reference pixel in the second reference block, by applying a horizontal gradient filter or a vertical gradient filter to the pixel values ​​of pixels at fractional positions. In this regard, the length of the gradient filter can be 5. The coefficients of the gradient filter can have coefficients symmetrical with respect to the center coefficient of the filter. The pixel at the fractional position can be a pixel having a fractional value in at least one of the horizontal and vertical components indicating the pixel's position.

[0211] The pixel values ​​of pixels at fractional positions in the first or second reference block can be calculated by applying a horizontal or vertical interpolation filter to the pixel values ​​of pixels at integer positions.

[0212] The displacement vector of the current pixel group per unit time in the horizontal or vertical direction can be the displacement vector per unit time determined by using the following: a first corresponding reference pixel in a first corresponding reference pixel group corresponding to the current pixel group in a first reference frame, a second corresponding reference pixel in a second corresponding reference pixel group corresponding to the current pixel group in a second reference frame, the pixel values ​​of the neighboring pixels of the first and second corresponding reference pixels, the gradient values ​​of the first and second corresponding reference pixels and the gradient values ​​of the neighboring pixels of the first and second corresponding reference pixels, a first POC difference between the first reference frame and the current frame, and a second POC difference between the second reference frame and the current frame.

[0213] When the first corresponding reference pixel or the second corresponding reference pixel is a boundary neighbor pixel adjacent to the boundary inside the first reference block or the second reference block, the pixel group unit motion compensator 120 can derive the pixel value of the pixel located outside the boundary of the first reference block or the second reference block by using the pixel value of the boundary neighbor pixel.

[0214] The pixel group unit motion compensator 120 can determine the displacement vector of the current block per unit time in the horizontal or vertical direction based on the pixel values ​​of boundary-adjacent pixels and the pixel values ​​of pixels located outside the boundary of the current block, wherein the pixel values ​​of pixels located outside the boundary of the current block are derived using the pixel values ​​of boundary-adjacent pixels. That is, pixels located outside the boundary may exist in the pixels included in the window about boundary-adjacent pixels, and in this respect, the pixel values ​​and gradient values ​​of pixels located outside the boundary may be pixel values ​​and gradient values ​​of pixels not included in memory, but rather pixel values ​​and gradient values ​​of pixels derived from boundary-adjacent pixels.

[0215] The pixel group unit motion compensator 120 can calculate the value of the current pixel using the pixel values ​​of the first corresponding reference pixels included in the first reference block, the pixel values ​​of the second corresponding reference pixels included in the second reference block, the gradient values ​​of the first and second corresponding reference pixels, the first POC difference between the first reference frame and the current frame, and the second POC difference between the second reference frame and the current frame. In other words, the value of the current pixel can be the result of a function based on the pixel values, gradient values, and POC differences between each reference frame and the current frame.

[0216] The pixel group unit motion compensator 120 can calculate values ​​for neighboring pixels using the pixel values ​​of the first corresponding neighboring pixels of the first corresponding reference pixel, the gradient values ​​of the first corresponding neighboring pixels, the pixel values ​​of the second corresponding neighboring pixels of the second corresponding reference pixel, the gradient values ​​of the second corresponding neighboring pixels, the first POC difference between the first reference frame and the current frame, and the second POC difference between the second reference frame and the current frame. In other words, the values ​​for neighboring pixels can be the result of a function based on the pixel values, gradient values, and POC differences between each reference frame and the current frame.

[0217] The pixel group unit motion compensator 120 can calculate a weighted average value for the displacement vector per unit time in the horizontal direction for the current pixel by using values ​​about the current pixel, values ​​about the corresponding neighboring pixels, and weights. In this regard, the weights can be determined based on the following: the distance between the current pixel and neighboring pixels, the distance between the pixel and the boundary of the block, the number of pixels located outside the boundary, or whether the pixel is located inside or outside the boundary.

[0218] The weighted average value for the current pixel can be a value calculated by applying exponential smoothing techniques in both the vertical and horizontal directions to the values ​​of pixels included in the first and second reference blocks. The value calculated for the current pixel by applying exponential smoothing techniques in both the vertical and horizontal directions to the values ​​of pixels included in the first and second reference blocks can be such that the weight for the value of the current pixel is maximized, and the weight for the values ​​of the neighboring pixels of the current pixel decreases exponentially with respect to their distance from the current pixel.

[0219] The pixel group unit motion compensator 120 can determine the displacement vector of the current pixel in the horizontal or vertical direction per unit time by using the weighted average value of the current pixel.

[0220] Inter-frame predictor 155 can generate the predicted pixel value of the current block by using the block-unit motion compensation value and the pixel-group-unit motion compensation value for the current block. For example, inter-frame predictor 155 can generate the predicted pixel value of the current block by adding the block-unit motion compensation value and the pixel-group-unit motion compensation value for the current block. Specifically, when the motion prediction mode of the current block is a bidirectional motion prediction mode, inter-frame predictor 155 can generate the predicted pixel value of the current block by using the block-unit motion compensation value and the pixel-group-unit motion compensation value for the current block.

[0221] When the motion prediction mode for the current block is unidirectional motion prediction mode, the inter-frame predictor 155 can generate the predicted pixel values ​​for the current block by using the block-unit motion-compensated values ​​and the pixel group-unit motion-compensated values ​​for the current block. In this regard, unidirectional indicates the use of a reference frame from among the previously encoded frames. The reference frame is not limited to a frame that precedes the current frame in the display order, but can be a frame that is displayed after the current frame.

[0222] Inter-frame predictor 155 can determine the motion prediction mode of the current block and output information indicating the motion prediction mode of the current block. For example, inter-frame predictor 155 can determine the motion prediction mode of the current block as a bidirectional motion prediction mode and output information indicating the bidirectional motion prediction mode. In this respect, a bidirectional motion prediction mode refers to a mode that predicts motion by using reference blocks in two decoded reference frames.

[0223] Bitstream generator 170 can generate a bitstream including motion vectors indicating reference blocks. Bitstream generator 170 can encode the motion vectors indicating reference blocks and can generate a bitstream including the encoded motion vectors. Bitstream generator 170 can encode differences in the motion vectors indicating reference blocks and can generate a bitstream including the encoded differences in the motion vectors. In this respect, the differences in motion vectors can indicate the difference between a motion vector and its prediction factor. In this respect, the differences in motion vectors can indicate the differences in motion vectors with respect to reference frames related to prediction directions including L0 and L1 directions, respectively. In this respect, the differences in motion vectors with respect to the L0 direction can represent the differences in motion vectors indicating reference blocks in reference frames included in the L0 reference frame list, and the differences in motion vectors with respect to the L1 direction can represent the differences in motion vectors indicating reference blocks in reference frames included in the L1 reference frame list.

[0224] Furthermore, the bitstream generator 170 can generate a bitstream that also includes information indicating the motion prediction mode of the current block. The bitstream generator 170 can encode a reference frame index indicating a reference frame of the current block within previously encoded frames, and can generate a bitstream including the encoded reference frame index. In this regard, the reference frame index can indicate a reference frame index for each of the prediction directions, including the L0 direction and the L1 direction. Specifically, the reference frame index for the L0 direction can indicate an index indicating a reference frame among the frames included in the L0 reference frame list, and the reference frame index for the L1 direction can indicate an index indicating a reference frame among the frames included in the L1 reference frame list.

[0225] The video encoding device 150 may include an image encoder (not shown), and the image encoder may include an inter-frame predictor 155 and a bitstream generator 170. References will follow. Figure 1f Describe the image encoder.

[0226] Figure 1d A flowchart of a video encoding method according to various embodiments is shown.

[0227] Reference Figure 1d During operation S150, the video encoding device 150 can obtain the prediction block, the first motion vector, the second motion vector, and parameters regarding the pixel group unit motion compensation by performing block unit motion compensation and pixel group unit motion compensation on the current block.

[0228] In operation S155, the video encoding device 150 can generate a bitstream including information about a first motion vector and a second motion vector, and motion prediction mode information, wherein the motion prediction mode information indicates that the motion prediction mode of the current block is a bidirectional motion prediction mode. In this respect, the first motion vector can be a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, and the second motion vector can be a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block.

[0229] The video encoding apparatus 150 can encode the residual blocks of the current block and generate a bitstream that also includes the encoded residual signal, wherein the residual blocks of the current block indicate the difference between the pixels of the predicted blocks of the current block and the original blocks of the current block. The video encoding apparatus 150 can encode information about the prediction mode of the current block and a reference frame index, and generate a bitstream that also includes the encoded prediction mode information about the current block and the encoded reference frame index. For example, the video encoding apparatus 150 can encode information indicating that the prediction mode of the current block is an inter-frame prediction mode and a reference frame index indicating at least one of the previously decoded frames, and generate a bitstream that also includes the encoded prediction mode information about the current block and the encoded reference frame index.

[0230] The video encoding device 150 can perform block-unit motion compensation and pixel-group-unit motion compensation on the current block using the following: the gradient value in the horizontal or vertical direction of a first corresponding reference pixel in a first reference block corresponding to the current pixel included in the current pixel group in the current block; the gradient value in the horizontal or vertical direction of a second corresponding reference pixel in a second reference block corresponding to the current pixel; the pixel value of the first corresponding reference pixel; the pixel value of the second corresponding reference pixel; and the displacement vector of the current pixel per unit time in the horizontal or vertical direction. The video encoding device 150 can obtain the prediction block of the current block by performing block-unit motion compensation and pixel-group-unit motion compensation on the current block.

[0231] In this regard, the displacement vector of the pixels of the current block, including pixels adjacent to the boundary of the current block, per unit time in the horizontal or vertical direction can be determined by using the values ​​of reference pixels included in the first and second reference blocks, instead of using stored values ​​of pixels located outside the boundaries of the first and second reference blocks. The values ​​of the reference pixels included in the first and second reference blocks can be pixel values ​​or gradient values ​​of the reference pixels.

[0232] Figure 1e A block diagram of an image decoder 600 according to various embodiments is shown.

[0233] Image decoder 600 according to various embodiments performs operations performed by image decoder (not shown) of video decoding device 100 to decode image data.

[0234] Reference Figure 1e The entropy decoder 615 parses the encoded image data to be decoded and the encoded information required for decoding from the bitstream 605. The encoded image data is the quantized transform coefficients, and the dequantizer 620 and the inverse transform 625 reconstruct the residual data from the quantized transform coefficients.

[0235] Intra-frame predictor 640 performs intra-frame prediction block by block. Inter-frame predictor 635 performs inter-frame prediction block by block using a reference image obtained from the reconstructed frame buffer 630. Figure 1e The inter-frame predictor 635 can be used with Figure 1a The inter-frame predictor 110 corresponds to this.

[0236] The spatial domain data of a block in the current image 605 can be reconstructed by adding the predicted data and residual data of each block generated by the intra-frame predictor 640 or the inter-frame predictor 635, and the deblocking unit 645 and the sample adaptive offset (SAO) actuator 650 can output a filtered reconstructed image 660 by performing loop filtering on the reconstructed spatial domain data. Furthermore, the reconstructed image stored in the reconstructed image buffer 630 can be output as a reference image.

[0237] In order for the decoder (not shown) of the video decoding device 100 to decode image data, step-by-step operations of the image decoder 600 according to various embodiments can be performed in blocks.

[0238] Figure 1f A block diagram of an image encoder according to various embodiments is shown.

[0239] The image encoder 700 according to various embodiments performs operations performed by the image encoder (not shown) of the video encoding device 150 to encode image data.

[0240] In other words, the intra-frame predictor 720 performs intra-frame prediction on the current image 705 block by block, and the inter-frame predictor 715 performs inter-frame prediction on the same block by using the current image 705 and a reference image obtained from the reconstructed frame buffer 710. In this respect, Figure 1e The inter-frame predictor 715 can be used with Figure 1c The inter-frame predictor 155 corresponds to this.

[0241] Residual data can be generated by subtracting the prediction data for each block output from the intra-predictor 720 or inter-predictor 715 from the data for the coded blocks of the current image 705. Transformer 725 and quantizer 730 can output block-quantized transform coefficients by performing transform and quantization on the residual data. Dequantizer 745 and inverse transformer 750 can reconstruct the spatial domain residual data by performing dequantization and inverse transform on the quantized transform coefficients. The reconstructed spatial domain residual data can be added to the prediction data for each block output from the intra-predictor 720 or inter-predictor 715 to be reconstructed as data in the spatial domain for the blocks of the current image 705. Deblocking unit 755 and SAO executor 760 generate a filtered reconstructed image by performing loop filtering on the reconstructed spatial domain data. The generated reconstructed image is stored in reconstructed frame buffer 710. The reconstructed image stored in reconstructed frame buffer 710 can be used as a reference image for inter-frame prediction of another image. The entropy encoder 735 can entropy encode the quantized transform coefficients, and the entropy-encoded coefficients can be output as a bit stream 740.

[0242] In order to apply the image encoder 700 according to various embodiments to the video encoding device 150, step-by-step operations of the image encoder 700 according to various embodiments can be performed in blocks.

[0243] Figure 2 This is a reference diagram illustrating block-based bidirectional motion prediction and compensation processing according to an embodiment.

[0244] Reference Figure 2 The video encoding device 150 performs bidirectional motion prediction, in which it searches for the region in the first reference frame 210 and the second reference frame 220 that is most similar to the current block 201 of the current frame 200 to be encoded. In this regard, it is assumed that the first reference frame 210 is a frame preceding the current frame 200, and the second reference frame 220 is a frame following the current frame 200. As a result of the bidirectional motion prediction, the video encoding device 150 determines a first corresponding region 212 from the first reference frame 210 that is most similar to the current block 201, and a second corresponding region 222 from the second reference frame 220 that is most similar to the current block 201. In this regard, the first corresponding region 212 and the second corresponding region 222 can be reference regions of the current block 201.

[0245] The video encoding device 150 can determine a first motion vector MV1 based on the position difference between the first corresponding region 212 and the block 211 of the first reference frame 210 at the same position as the current block 201, and can determine a second motion vector MV2 based on the position difference between the second corresponding region 222 and the block 221 of the second reference frame 220 at the same position as the current block 201.

[0246] The video encoding device 150 performs block-unit bidirectional motion compensation on the current block 201 by using a first motion vector MV1 and a second motion vector MV2.

[0247] For example, when the pixel value at (i, j) in the first reference frame 210 is P0(i, j) and the pixel value at (i, j) in the second reference frame 220 is P1(i, j), MV1 = (MVx1, MVy1), MV2 = (MVx2, MVy2), where i and j are integers, the block-unit bidirectional motion compensation value P_BiPredBlock(i, j) of the pixel at position (i, j) in the current block 201 can be calculated according to the following equation: P_BiPredBlock(i, j) = {P0(i+MVx1, j+MVy1) + P1(i+MVx2, j+MVy2)} / 2. In this way, the video encoding device 150 can generate a block-unit motion-compensated value by performing block-unit motion compensation on the current block 201 using the average or weighted sum of the pixels in the first corresponding region 212 indicated by the first motion vector MV1 and the second corresponding region 222 indicated by the second motion vector MV2.

[0248] Figures 3a to 3c This is a reference diagram illustrating the process of performing pixel group unit motion compensation according to an embodiment.

[0249] exist Figure 3a In the above, it is assumed that the first corresponding region 310 and the second corresponding region 320 are respectively with Figure 2 The first corresponding region 212 and the second corresponding region 222 correspond to each other and are shifted to overlap with the current block 300 by using bidirectional motion vectors MV1 and MV2.

[0250] Furthermore, P(i,j) indicates the bidirectionally predicted pixel at position (i,j) of the current block 300 (where i and j are integers), P0(i,j) indicates the first reference pixel value of the first reference frame corresponding to the bidirectionally predicted pixel P(i,j) of the current block 300, and P1(i,j) indicates the second reference pixel value of the second reference frame corresponding to the bidirectionally predicted pixel P(i,j) of the current block 300.

[0251] In other words, the first reference pixel value P0(i,j) is the pixel value of the pixel corresponding to the pixel P(i,j) of the current block 300, determined by the bidirectional motion vector MV1 indicating the first reference screen, and the second reference pixel value P1(i,j) is the pixel value of the pixel corresponding to the pixel P(i,j) of the current block 300, determined by the bidirectional motion vector MV2 indicating the second reference screen.

[0252] In addition, definition Indicates the gradient value of the first reference pixel in the horizontal direction. Indicates the gradient value of the first reference pixel in the vertical direction. Indicates the gradient value of the second reference pixel in the horizontal direction, and This indicates the gradient value of the second reference pixel in the vertical direction. Furthermore, τ0 is defined as indicating the time distance between the current frame to which the current block 300 belongs and the first reference frame to which the first corresponding region 310 belongs, and τ1 is defined as indicating the time distance between the current frame and the second reference frame to which the second corresponding region 320 belongs. In this respect, the time distance between frames can indicate the difference in POC between the frames.

[0253] When there is uniform small motion in the video sequence, the pixel in the first corresponding region 310 of the first reference frame that is most similar to the pixel P(i,j) for which bidirectional pixel group unit motion compensation is performed is not the first reference pixel P0(i,j), but rather the first displacement reference pixel PA after the first reference pixel P0(i,j) has been moved by a predetermined displacement vector. As mentioned above, assuming there is uniform motion in the video sequence, the pixel in the second corresponding region 320 of the second reference frame that is most similar to pixel P(i,j) could be the second displacement reference pixel PB after the second reference pixel P1(i,j) has been moved by a predetermined displacement vector.

[0254] The displacement vector may include a displacement vector Vx in the x-axis direction and a displacement vector Vy in the y-axis direction. Therefore, the pixel group unit motion compensator 165 calculates the displacement vectors included in the displacement vectors, namely the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction, and can perform pixel group unit motion compensation by using the displacement vectors.

[0255] Optical flow refers to a pattern of apparent motion on an object or surface caused by the relative motion between the scene and the observer (eye or video image acquisition device, such as a camera). In a video sequence, optical flow can be represented by calculating the motion between frames acquired at arbitrary times t and t+Δt. The pixel value at (x, y) in a frame at time t can be defined as I(x, y, t). That is, I(x, y, t) can be a value that varies in both time and space. I(x, y, t) can be differentiated with respect to time t according to Equation 1.

[0256] [Equation 1]

[0257]

[0258] When the pixel value changes with the motion but not with time for small movements within a block, dI / dt is 0. Furthermore, when assuming the motion of the pixel value change is constant with respect to time, dx / dt can indicate the displacement vector Vx of the pixel value I(x, y, t) in the x-axis direction, and dy / dt can indicate the displacement vector Vy of the pixel value I(x, y, t) in the y-axis direction. Therefore, Equation 1 can be expressed as Equation 2.

[0259] [Equation 2]

[0260]

[0261] In this respect, the magnitudes of the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction can be smaller than the pixel precision used in bidirectional motion prediction. For example, when the pixel precision is 1 / 4 or 1 / 16 during bidirectional motion prediction, the magnitudes of the displacement vectors Vx and Vy can be smaller than 1 / 4 or 1 / 16.

[0262] The pixel group unit motion compensator 165 calculates the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction according to Equation 2, and performs pixel group unit motion compensation by using the displacement vectors Vx and Vy. In Equation 2, since the pixel value I(x, y, t) is the value of the original signal, it may lead to high overhead during encoding when the value of the original signal is used without change. Therefore, the pixel group unit motion compensator 165 can calculate the displacement vectors Vx and Vy according to Equation 2 by using the pixels of the first reference frame and the second reference frame, which are determined as a result of performing bidirectional block unit motion compensation. That is, the pixel group unit motion compensator 165 determines the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction, where Δ is minimized in a window Ωij having a predetermined size and including neighboring pixels around the pixel P(i, j) for which bidirectional motion compensation has been performed. Δ can be 0, but for all pixels in the window Ωij, the displacement vectors Vx and Vy along the x-axis that satisfy Δ = 0 may not exist. Therefore, the displacement vectors Vx and Vy along the x-axis with the minimum Δ are determined. (Refer to...) Figure 8a Describe in detail the process of obtaining displacement vectors Vx and Vy.

[0263] To determine the predicted pixel value of the current pixel, the function P(t) with respect to t can be determined according to Equation 3.

[0264] [Equation 3]

[0265] P(t) = a3*t 3 +a2*t 2 +a1*t+a0

[0266] In this regard, it is assumed that the frame at t=0 is the current frame including the current block. Therefore, when t is 0, the predicted pixel value of the current pixel included in the current block can be defined as the value of P(t).

[0267] When the time distance between the current frame and the first reference frame (which is prior to the current frame in time) is τ0, and the time distance between the current frame and the second reference frame (which is subsequent to the current frame in time) is τ1, the reference pixel value in the first reference frame is equal to P(-τ0), and the reference pixel value in the second reference frame is equal to P(τ1). In the following text, for ease of calculation, it is assumed that both τ0 and τ1 are equal to τ.

[0268] The coefficients of each level of P(t) can be determined according to Equation 4. Here, P0(i,j) can indicate the pixel value at position (i,j) of the first reference frame, and P1(i,j) can indicate the pixel value at position (i,j) of the second reference frame.

[0269] [Equation 4]

[0270]

[0271]

[0272]

[0273]

[0274] Therefore, the predicted pixel value P(0) of the current pixel in the current block can be determined according to Equation 5.

[0275] [Equation 5]

[0276]

[0277] Considering Equation 2, Equation 5 can be expressed as Equation 6.

[0278] [Equation 6]

[0279]

[0280] Therefore, the predicted pixel value of the current pixel can be determined using displacement vectors Vx and Vy, the gradient values ​​of the first reference pixel in the horizontal and vertical directions, and the gradient values ​​of the second reference pixel in the horizontal and vertical directions. In this regard, the portion (P0(i,j)+P1(i,j)) / 2) independent of displacement vectors Vx and Vy can be a value compensated for by block-unit motion, and the portion related to displacement vectors Vx and Vy can be a value compensated for by pixel-group-unit motion. As a result, the predicted pixel value of the current pixel can be determined by adding the value compensated for by block-unit motion to the value compensated for by pixel-group-unit motion.

[0281] In the preceding text, for ease of description, the process of determining the predicted pixel value of the current pixel when the time distance between the first reference frame and the current frame and the time distance between the second reference frame and the current frame are both τ and therefore the same was described. However, the time distance between the first reference frame and the current frame can be τ0, and the time distance between the second reference frame and the current frame can be τ1. Here, the predicted pixel value P(0) of the current pixel can be determined according to Equation 7.

[0282] [Equation 7]

[0283]

[0284] Considering equation 2, equation 7 can be expressed as equation 8.

[0285] [Equation 8]

[0286]

[0287] As described above, the first reference frame is located after the current frame in time and the second reference frame is located before the current frame in time. However, both the first reference frame and the second reference frame can be located before or after the current frame in time.

[0288] For example, such as Figure 3b As shown, both the first reference screen including the first corresponding area 310 and the second reference screen including the second corresponding area 320 can be located in time before the current screen including the current block 300.

[0289] In this case, the predicted pixel value P(0) of the current pixel can be determined according to Equation 9, in which -τ1 is used instead of the reference value. Figure 3a Equation 8 shows τ1, which indicates the time distance between the second reference frame and the current frame.

[0290] [Equation 9]

[0291]

[0292] For example, such as Figure 3c As shown, both the first reference screen including the first corresponding area 310 and the second reference screen including the second corresponding area 320 can be located after the current screen including the current block 300 in time.

[0293] In this case, the predicted pixel value P(0) of the current pixel can be determined according to Equation 10, in which -τ0 is used instead of the reference value. Figure 3a Equation 8 shows τ0, which indicates the time distance between the first reference frame and the current frame.

[0294] [Equation 10]

[0295]

[0296] However, when Figure 3b and Figure 3c When both the first and second reference frames are located before or after the current frame in time, pixel-level motion compensation can be performed even if the first and second reference frames are not the same reference frame. Furthermore, in this case, pixel-level motion compensation can be performed only when both bidirectional motion vectors MV1 and MV2 have non-zero components. Additionally, in this case, pixel-level motion compensation can be performed only when the ratio of motion vectors MV1 and MV2 is the same as the ratio of the time distance between the first reference frame and the current frame to the time distance between the second reference frame and the current frame. For example, pixel-level motion compensation can be performed when the ratio of the x-component of motion vector MV1 to the x-component of motion vector MV2 is the same as the ratio of the y-component of motion vector MV1 to the y-component of motion vector MV2, and is also the same as the ratio of the time distance τ0 between the first reference frame and the current frame to the time distance τ1 between the second reference frame and the current frame.

[0297] Figure 4 This is a reference diagram used to describe the process of calculating gradient values ​​in the horizontal and vertical directions according to an embodiment.

[0298] Reference Figure 4 The gradient value of the first reference pixel P0(i,j)410 in the horizontal direction can be calculated by obtaining the changes in pixel values ​​at neighboring fractional pixel positions adjacent to the first reference pixel P0(i,j)410 in the horizontal direction and the changes in pixel values ​​at neighboring fractional pixel positions adjacent to the first reference pixel P0(i,j)410 in the vertical direction. The gradient value of the first reference pixel P0(i,j)410 in the vertical direction In other words, according to Equation 11, the gradient value in the horizontal direction can be calculated by calculating the change in pixel value of fractional pixels P0(ih,j)460 and P0(i+h,j)470 that are h away from P0(i,j) in the horizontal direction. Here, h is a fractional value less than 1, and the gradient value in the vertical direction can be calculated by measuring the changes in pixel values ​​of fractional pixels P0(i,jh)480 and P0(i,j+h)490 that are h distances from P0(i,j) in the vertical direction.

[0299] [Equation 11]

[0300]

[0301]

[0302] The pixel values ​​of fractional pixels P0(ih,j)460, P0(i+h,j)470, P0(i,jh)480, and P0(i,j+h)490 can be calculated using general interpolation. Furthermore, the gradient values ​​of the second reference pixels in another second reference frame in the horizontal and vertical directions can be calculated similarly to Equation 11.

[0303] According to an embodiment, the gradient value at the reference pixel can be calculated using a predetermined filter instead of calculating the gradient value by calculating the change in pixel value at a fractional pixel location as in Equation 11. The filter coefficients of the predetermined filter can be determined based on the coefficients of the interpolation filter used to obtain the pixel value at the fractional pixel location, depending on the linearity of the filter.

[0304] Figure 5 This is a reference diagram used to describe the process of calculating gradient values ​​in the horizontal and vertical directions according to another embodiment.

[0305] According to another embodiment, gradient values ​​can be determined by applying a predetermined filter to pixels of a reference image. (See also...) Figure 5 The video decoding device 100 can apply a predetermined filter to |M based on the reference pixel P0 500 from which the current horizontal gradient value will be obtained. min | 520 pixels on the left and M max The gradient value of reference pixel P0 500 in the horizontal direction is calculated using the right-hand pixel 510. This can be done according to... Figures 7a to 7d The indicated M is used to determine the window size. max Integer pixels and M min The value α, representing the interpolation position (fractional pel position) between integer pixels, determines the filter coefficients used here. For example, referring to... Figure 7a When M is used to determine the window sizemin and M max When the values ​​are -2 and 3 respectively, and the distance from the reference pixel P0 500 is 1 / 4, i.e., α = 1 / 4, Figure 7a The filter coefficients {4, -17, -36, 60, -15, 4} in the second row are applied to the neighboring pixel P. -2 P -1 P0, P1, P2, and P3. In this case, the gradient value of the reference pixel P0 500 in the horizontal direction can be calculated using the filter coefficients and the weighted sum of neighboring pixels. Such as equations Similarly, it can also be done by determining the interpolation position and M used to determine the window size. min and M max Will Figures 7a to 7e The filter coefficients shown are applied to neighboring pixels to calculate the gradient value in the vertical direction.

[0306] Figures 6a to 6b This is a diagram illustrating the process of determining gradient values ​​in the horizontal and vertical directions using a 1D filter according to an embodiment.

[0307] Reference Figure 6a Filtering can be performed by using multiple 1D filters for integer pixels to determine the gradient value of a reference pixel in the reference frame in the horizontal direction. Pixel group unit motion compensation is an additional motion compensation performed after block unit motion compensation is performed. Therefore, the reference position of the reference block of the current block, indicated by the motion vector during block unit motion compensation, can be a fractional pixel position, and pixel group unit motion compensation can be performed on the reference pixels at the fractional pixel positions in the reference block. Therefore, filtering can be performed taking into account the determined gradient value of the pixel at the fractional pixel position.

[0308] Reference Figure 6a First, the video decoding device 100 can perform filtering on pixels located in the horizontal or vertical direction starting from the nearest integer pixels of the reference pixel in the reference frame using a first 1D filter. Similarly, the video decoding device 100 can perform filtering on nearest integer pixels in rows or columns different from the reference pixel using the first 1D filter. The video decoding device 100 can generate the gradient value of the reference pixel in the horizontal direction by performing filtering on the values ​​generated by the filtering using a second 1D filter.

[0309] For example, when the reference pixel is located at the position of a fractional pixel at (x+α, y+β) (where x and y are integers and α and β are fractions), it can be obtained by considering the integer pixels (x, y), (x-1, y), (x+1, y) to (x+M) in the horizontal direction. min,y) and (x+M max ,y)(where M min and M max (If it is an integer) Use a 1D vertical interpolation filter and perform filtering according to Equation 12.

[0310] [Equation 12]

[0311]

[0312] Here, fracFilter β It can instruct an interpolation filter for determining the pixel value at the fractional pixel position β in the vertical direction, and fracFilter β [j'] indicates the coefficients of the interpolation filter to be applied to the pixel at position (i, j'). I[i, j'] indicates the pixel value at position (i, j').

[0313] In other words, the first 1D filter can be an interpolation filter used to determine fractional pixel values ​​in the vertical direction. `offset1` indicates the offset used to prevent rounding errors, and `shift1` indicates the number of bits to scale. `Temp[i, j+β]` indicates the pixel value at the fractional pixel position (i, j+β). `Temp[i', j+i]` (where i' is from i+M) min To i+M max The integers other than i can also be determined by replacing i with i' according to Equation 12.

[0314] Subsequently, the video decoding device 100 can perform filtering on the pixel value at fractional pixel position (i, j+) and the pixel value at fractional pixel position (i', j+β) by using a second 1D filter.

[0315] [Equation 13]

[0316]

[0317] In this regard, gradFilter α This can be a gradient filter used to determine the gradient value at a fractional pixel position α in the horizontal direction. (gradFilter) α [i'] indicates the coefficients of the gradient filter to be applied to the pixel at position (i', j+β). That is, the second 1D filter can be a gradient filter used to determine the gradient value in the horizontal direction. offset2 indicates the offset used to prevent rounding errors, and shift2 indicates the number of bits to scale.

[0318] In other words, according to Equation 13, the video decoding device 100 can achieve this by using a gradient filter (gradFilter).α Filter the pixel value (Temp[i,j+β]) at pixel position (i,j+β) and the pixel value (Temp[i',j+β]) in the vertical direction starting from pixel position (i,j+β) to determine the gradient value in the horizontal direction at (i+α,j+β).

[0319] In the foregoing, the gradient value in the horizontal direction was determined by first applying an interpolation filter and then applying a gradient filter. However, this disclosure is not limited to this, and therefore, the gradient value in the horizontal direction can also be determined by first applying a gradient filter and then applying an interpolation filter. Hereinafter, embodiments for determining the gradient value in the horizontal direction by applying a gradient filter and then applying an interpolation filter will be described.

[0320] For example, when the position of the reference pixel is the position of the fractional pixel at (x+α, y+β) (where x and y are integers and α and β are fractions), it can be determined according to Equation 14 by applying integer pixels (x, y), (x-1, y), (x+1, y) to (x+M) in the horizontal direction. min ,y) and (x+M max ,y)(where M min and M max (If it is an integer) Use the first 1D filter to perform filtering.

[0321] [Equation 14]

[0322]

[0323] In this regard, gradFilter α A gradient filter can be specified to determine the gradient value at a fractional pixel position α in the horizontal direction, and gradFilter α [i'] indicates the coefficients of the gradient filter to be applied to the pixel at position (i', j). I[i', j] indicates the pixel value at position (i', j).

[0324] In other words, the first 1D filter can be a gradient filter used to determine the gradient value of a pixel in the horizontal direction, where the horizontal component of the pixel position is a fractional position. `offset3` indicates the offset used to prevent rounding errors, and `shift3` indicates the number of bits to scale. `Temp[i+α, j]` indicates the gradient value at the pixel position (i+α, j) in the horizontal direction. `Temp[i+α, j']` (where j' is from j+M) min to j+M max (Integers other than j) can also be determined according to Equation 14 by replacing j with j'.

[0325] Then, the video decoding device 100 can perform filtering on the gradient value at the pixel position (i+α, j) in the horizontal direction and the gradient value at the pixel position (i+α, j') in the horizontal direction by using a second 1D filter according to Equation 15.

[0326] [Equation 15]

[0327]

[0328] In this regard, fracFilter β It can be an interpolation filter used to determine the pixel value at a fractional pixel position β in the vertical direction. fracFilter β [j'] indicates the coefficients of the interpolation filter to be applied to the pixel at position (i+β, j'). That is, the second 1D filter can be an interpolation filter used to determine the pixel value at fractional pixel position β in the vertical direction. offset4 indicates the offset used to prevent rounding errors, and shift4 indicates the number of bits to descale.

[0329] In other words, according to Equation 15, the video decoding device 100 can decode the video by using the interpolation filter fracFilter. β The gradient value at (i+α, j+β) is determined by filtering the gradient value (Temp[i+α, j]) at the horizontal pixel position (i+α, j) and the gradient value (Temp[i+α, j']) of the pixel in the vertical direction starting from pixel position (i+α, j).

[0330]

[0331] Reference Figure 6b Filtering can be performed by using multiple 1D filters for integer pixels to determine the gradient value of a reference pixel in the vertical direction. Pixel group unit motion compensation is an additional motion compensation performed after block unit motion compensation. Therefore, the reference position of the reference block of the current block, indicated by the motion vector during block unit motion compensation, can be a fractional pixel position, and pixel group unit motion compensation can be performed on the reference pixels at the fractional pixel positions in the reference block. Therefore, filtering can be performed taking into account the determined gradient value of the pixel at the fractional pixel position.

[0332] Reference Figure 6bFirst, the video decoding device 100 can perform filtering on pixels located in the horizontal or vertical direction, starting from the nearest integer pixels of the reference pixel in the reference frame, using a first 1D filter. Similarly, the video decoding device 100 can perform filtering on neighboring pixels in rows or columns different from the reference pixel, using the first 1D filter. The video decoding device 100 can generate the gradient value of the reference pixel in the vertical direction by performing filtering on the values ​​generated via filtering using a second 1D filter.

[0333] For example, when the position of the reference pixel is the position of a fractional pixel at (x+α, y+β) (where x and y are integers and α and β are fractions), it can be calculated according to Formula 16, by targeting integer pixels (x, y), (x-1, y-1), (x+1, y+1) to (x+M) in the horizontal direction. min y+M min ) and (x+M max y+M max (where M) min and M max (If it is an integer) Use the first 1D filter to perform filtering.

[0334] [Equation 16]

[0335]

[0336] In this regard, fracFilter α An interpolation filter can be specified to determine the pixel value at a fractional pixel position α in the horizontal direction, and fracFilter α [i'] indicates the coefficients of the interpolation filter to be applied to the pixel at position (i', j). I[i', j] indicates the pixel value at position (i', j).

[0337] In other words, the first 1D filter can be an interpolation filter used to determine the pixel value at a fractional pixel position α in the horizontal direction. offset5 can indicate the offset used to prevent rounding errors, and shift5 can indicate the number of bits to descale.

[0338] Temp[i+α, j] indicates the pixel value at the fractional pixel position (i+α, j). Temp[i+α, j'] (where j' is from j+M) min to j+M max (Integers other than j) can also be determined according to Equation 16 by replacing j with j'.

[0339] Subsequently, the video decoding device 100 can perform filtering on the pixel value at pixel position (i+α, j) and the pixel value at pixel position (i+α, j') by using a second 1D filter according to Equation 17.

[0340] [Equation 17]

[0341]

[0342] In this regard, gradFilter β This can be a gradient filter used to determine the gradient value at a fractional pixel position β in the vertical direction. (gradFilter) β [j'] indicates the coefficients of the gradient filter to be applied to the pixel at position (i+α, j'). That is, the second 1D filter can be a gradient filter used to determine the gradient value in the vertical direction at the fractional pixel position β. offset6 indicates the offset used to prevent rounding errors, and shift6 indicates the number of bits to descale.

[0343] In other words, according to Equation 17, the video decoding device 100 can achieve this by using a gradient filter (gradFilter). β Filter the pixel value (Temp[i+α,j]) at pixel position (i+α,j) and the pixel value (Temp[i+α,j']) in the vertical direction starting from pixel position (i+α,j).

[0344] As described above, the gradient value in the vertical direction is determined by first applying an interpolation filter and then applying a gradient filter. However, this disclosure is not limited to this, and therefore, the gradient value in the vertical direction can also be determined by first applying a gradient filter and then applying an interpolation filter. Hereinafter, embodiments for determining the gradient value in the vertical direction by applying a gradient filter and then applying an interpolation filter will be described.

[0345] For example, when the position of the reference pixel is the position of a fractional pixel at (x+α, y+β) (where x and y are integers and α and β are fractions), Equation 18 can be applied to integer pixels (x, y), (x, y-1), (x, y+1) to (x, y+M) in the vertical direction. min ) and (x, y+M max (where M) min and M max (If it is an integer) Use the first 1D filter to perform filtering.

[0346] [Equation 18]

[0347]

[0348] In this regard, gradFilter β A gradient filter can be specified to determine the gradient value at a fractional pixel position β in the vertical direction, and gradFilter β [j'] indicates the coefficients of the gradient filter to be applied to the pixel at position (i, j'). I[i, j'] indicates the pixel value at position (i, j').

[0349] In other words, the first 1D filter can be a gradient filter used to determine the gradient value of a pixel in the vertical direction, where the vertical component of the pixel position is a fractional position. offset7 can indicate the offset used to prevent rounding errors, and shift7 can indicate the number of bits to scale.

[0350] Temp[i, j+β] indicates the gradient value at pixel position (i, j+β) in the vertical direction. Temp[i', j+β] (where i' is the gradient value from i+M) min To i+M max The integers other than i can also be determined by replacing i with i' according to Equation 18.

[0351] Subsequently, the video decoding device 100 can perform filtering on the gradient values ​​at the pixel position (i, j+β) and the pixel position (i', j+β) in the vertical direction by using a second 1D filter according to Formula 19.

[0352] [Equation 19]

[0353]

[0354] In this regard, fracFilter α It can be an interpolation filter used to determine the pixel value at a fractional pixel position α in the horizontal direction. fracFilter α [i'] indicates the coefficients of the interpolation filter to be applied to the pixel at position (i', j+β). That is, the second 1D filter can be an interpolation filter used to determine the pixel value at a fractional pixel position α in the horizontal direction. offset8 indicates the offset used to prevent rounding errors, and shift8 indicates the number of bits to descale.

[0355] In other words, according to Equation 19, the video decoding device 100 can decode the video by using the interpolation filter fracFilter. αThe gradient value in the vertical direction at pixel position (i, j+β) is determined by filtering the gradient value (Temp[i, j+β]) at the vertical position and the gradient value (Temp[i', j+β]) of the pixels in the horizontal direction starting from pixel position (i, j+β).

[0356] According to an embodiment, in the video decoding device 100, the gradient values ​​in the horizontal and vertical directions at (i+α, j+β) can be determined based on the combination of the various filters described above. For example, to determine the gradient value in the horizontal direction, an interpolation filter for determining the pixel value in the vertical direction can be used as a first 1D filter, and a gradient filter for determining the gradient value in the horizontal direction can be used as a second 1D filter. Similarly, a gradient filter for determining the gradient value in the vertical direction can be used as a first 1D filter, and an interpolation filter for determining the pixel value in the horizontal direction can be used as a second 1D filter.

[0357] Figures 7a to 7e This is a table showing the filter coefficients of a filter used to determine the pixel value at a fractional pixel position and the gradient values ​​in the horizontal and vertical directions, according to an embodiment.

[0358] Figures 7a to 7b This is a table showing the filter coefficients of a filter used to determine the gradient value at a fractional pixel position of 1 / 4 pel unit in the horizontal or vertical direction.

[0359] As described above, 1D gradient filters and 1D interpolation filters can be used to determine gradient values ​​in the horizontal or vertical directions. (Refer to...) Figure 7a The diagram illustrates the filter coefficients of a 1D gradient filter. In this respect, a 6-tap filter can be used as a 1D gradient filter. The filter coefficients of a 1D gradient filter can be coefficients scaled by a factor of 2^4. min Indicates the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied in the negative direction of the filter based on the center integer pixel, and M max This indicates the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied to the filter in the positive direction based on the center integer pixel. For example, gradient filter coefficients used to obtain the gradient value of pixels in the horizontal direction when the fractional pixel position α is 1 / 4 in the horizontal direction can be {4, -17, -36, 60, -15, -4}. Gradient filter coefficients used to obtain the gradient value of pixels in the horizontal direction when the fractional pixel position α is 0, 1 / 2, or 3 / 4 in the horizontal direction can also be obtained by referring to... Figure 7a It has been confirmed.

[0360] Reference Figure 7b The diagram shows the filter coefficients of a 1D interpolation filter. In this respect, a 6-tap filter can be used as a 1D interpolation filter. The filter coefficients of a 1D interpolation filter can be coefficients scaled by a factor of 2^6. M min Indicates the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied in the negative direction of the filter based on the center integer pixel, and M max Indicates the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied in the positive direction of the filter based on the center integer pixel.

[0361] Figure 7c This is a table showing the filter coefficients of a 1D interpolation filter used to determine the pixel value at a fractional pixel location of 1 / 4 pel unit.

[0362] As described above, two identical 1D interpolation filters can be used in the horizontal and vertical directions to determine the pixel value at the fractional pixel location.

[0363] Reference Figure 7c The diagram shows the filter coefficients of a 1D interpolation filter. In this respect, a 6-tap filter can be used as a 1D interpolation filter. The filter coefficients of a 1D interpolation filter can be coefficients scaled by a factor of 2^6. M min Indicates the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied in the negative direction of the filter based on the center integer pixel, and M max Indicates the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied in the positive direction of the filter based on the center integer pixel.

[0364] Figure 7d This is a table showing the filter coefficients of a filter used to determine the gradient value in the horizontal or vertical direction at a fractional pixel position of 1 / 16 pel unit.

[0365] As described above, 1D gradient filters and 1D interpolation filters can be used to determine gradient values ​​in the horizontal or vertical directions. (Refer to...) Figure 7dThe diagram illustrates the filter coefficients of a 1D gradient filter. In this respect, a 6-tap filter can be used as a 1D gradient filter. The filter coefficients of a 1D gradient filter can be coefficients scaled by a factor of 2^4. For example, the gradient filter coefficients for obtaining the gradient value of a pixel in the horizontal direction when the fractional pixel position α is 1 / 16 in the horizontal direction can be {8, -32, -13, 50, -18, 5}. Gradient filter coefficients for obtaining the gradient value of a pixel in the horizontal direction when the fractional pixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, or 1 / 2 in the horizontal direction can also be obtained by referring to... Figure 7d The gradient filter coefficients used to obtain the gradient values ​​of pixels in the horizontal direction, where the fractional pixel position α is 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, or 15 / 16 in the horizontal direction, can be determined by using the symmetry of the filter coefficients based on α = 1 / 2. That is, the filter coefficients at the right fractional pixel position based on α = 1 / 2 can be determined by using, for example... Figure 7d The filter coefficients at the left fractional pixel position based on α = 1 / 2 are shown. For example, the filter coefficients at α = 15 / 16 can be determined by using the filter coefficients at α = 1 / 16 as a symmetrical position based on α = 1 / 2: {8, -32, -13, 50, -18, 5}. That is, by arranging {8, -32, -13, 50, -18, 5} in reverse order, the filter coefficients at α = 15 / 16 can be determined as {5, -18, 50, -13, -32, 8}.

[0366] Reference Figure 7e The diagram illustrates the filter coefficients of a 1D interpolation filter. In this respect, a 6-tap filter can be used as a 1D interpolation filter. The filter coefficients of a 1D interpolation filter can be coefficients scaled by a factor of 2^6. For example, the 1D interpolation filter coefficients for obtaining the pixel value in the horizontal direction when the fractional pixel position α is 1 / 16 in the horizontal direction can be {1, -3, 64, 4, -2, 0}. Interpolation filter coefficients for obtaining the pixel value in the horizontal direction when the fractional pixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, or 1 / 2 in the horizontal direction can also be obtained by referring to... Figure 7e The values ​​are determined. Furthermore, the interpolation filter coefficients used to obtain the pixel values ​​in the horizontal direction, where the fractional pixel position α is 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, or 15 / 16 in the horizontal direction, can be determined by using the symmetry of the filter coefficients based on α = 1 / 2. That is, the filter coefficients at the right fractional pixel position based on α = 1 / 2 can be determined by using, for example... Figure 7eThe filter coefficients at the left fractional pixel position based on α = 1 / 2 are shown. For example, the filter coefficients at α = 15 / 16 can be determined by using the filter coefficients at α = 1 / 16 as a symmetrical position based on α = 1 / 2: {1, -3, 64, 4, -2, 0}. That is, by arranging {1, -3, 64, 4, -2, 0} in reverse order, the filter coefficients at α = 15 / 16 can be determined as {0, -2, 4, 64, -3, 1}.

[0367] Figure 8a This is a reference diagram used to describe the process of determining the horizontal and vertical displacement vectors of a pixel according to an embodiment.

[0368] Reference Figure 8a A window Ωij 800 of a predetermined size has a size of (2M+1)*(2N+1) (where M and N are integers) for the bidirectionally predicted pixels P(i,j) of the current block.

[0369] When P(i', j') represents the pixel of the current block being bidirectionally predicted in window Ωij 800 (where iM≤i'≤i+M and jN≤j'≤j+N, (i', j')∈Ωij), P0(i', j') indicates the pixel value of the first reference pixel in the first reference screen 810 corresponding to the pixel P(i', j') of the current block being bidirectionally predicted, and P1(i', j') indicates the pixel value of the second reference pixel in the second reference screen 820 corresponding to the pixel P(i', j') of the current block being bidirectionally predicted. Indicates the gradient value of the first reference pixel in the horizontal direction. Indicates the gradient value of the first reference pixel in the vertical direction. Indicates the gradient value of the second reference pixel in the horizontal direction, and The gradient value of the second reference pixel in the vertical direction, the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB', can be determined according to Equation 20. In this respect, the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can be determined by using the first linear term of the local Taylor expansion.

[0370] [Equation 20]

[0371]

[0372]

[0373] In Equation 20, the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction can change according to the position of pixel P(i,j) (i.e., depending on (i,j)). Therefore, the displacement vectors Vx and Vy can be represented as Vx(i,j) and Vy(i,j).

[0374] The difference Δi'j' between the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can be determined according to Equation 21.

[0375] [Equation 21]

[0376]

[0377] The displacement vectors Vx in the x-axis direction and Vy in the y-axis direction that minimize the difference △i'j' between the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can be determined by using the sum of squares of the difference △i'j' as in Equation 22, Φ(Vx, Vy).

[0378] [Equation 22]

[0379]

[0380] In other words, the displacement vectors Vx and Vy can be determined by using the maximum or minimum value of Φ(Vx, Vy). Φ(Vx, Vy) refers to the displacement vectors Vx and Vy being a function of parameters, and their maximum or minimum values ​​can be determined by calculating the value of Φ(Vx, Vy) rearranged with respect to τVx and τVy according to Equation 23, making the partial derivative with respect to τVx and τVy equal to 0. In the following text, for ease of calculation, it is assumed that τ0 and τ1 are equal to τ.

[0381] [Equation 23]

[0382]

[0383] For example, the two linear equations in Equation 24, where Vx(i,j) and Vy(i,j) are variables, can be solved by using equations... Sum of equations Obtained.

[0384] [Equation 24]

[0385] TVx*s1+τVy(i,j)*s2=s3

[0386] τVx*s4+τVy(i,j)*s5=s6

[0387] In Equation 24, calculate s1 to s6 according to Equation 25.

[0388] [Equation 25]

[0389]

[0390]

[0391]

[0392]

[0393]

[0394] By solving the simultaneous equations in Equation 24, the values ​​of Vx(i,j) and Vy(i,j) can be calculated based on the Cramer formula using τ*Vx(i,j)=-det1 / det and τ*Vy(i,j)=-det2 / det. In this respect, det1=s3*s5-s2*s6, det2=s1*s6-s3*s4, and det=s1*s5-s2*s2.

[0395] A simplified solution to the above equation can be determined by first performing minimization in the horizontal direction and then minimizing in the vertical direction. That is, for example, when assuming that only the displacement vector in the horizontal direction is changed, Vy can be 0 in the first equation of Equation 24, thus the equation τVx = s3 / s1 can be determined.

[0396] Then, when the second equation of equation 24 is rearranged using the equation τVx=s3 / s1, the equation τVy=(s6-τVx*S2) / s5 can be determined.

[0397] In this respect, gradient value and It can be scaled without changing the resulting values ​​Vx(i,j) and Vy(i,j). However, this is contingent on no overflow and no rounding errors occurring.

[0398] When calculating Vx(i,j) and Vy(i,j), adjustment parameters r and m can be introduced to prevent division operations by 0 or very small values ​​from being performed.

[0399] For convenience, assume that Vx(i,j) and Vy(i,j) are the same as... Figure 3a The directions shown are opposite. For example, aside from the symbols, based on Figure 3a The directions of Vx(i,j) and Vy(i,j) shown are derived from Equation 24. Vx(i,j) and Vy(i,j) can have the same direction as those determined by Equation 24. Figure 3a The opposite directions of Vx(i,j) and Vy(i,j) have the same value.

[0400] The first displacement-corresponding pixel PA' and the second displacement-corresponding pixel PB' can be determined according to Equation 26. In this respect, the first displacement-corresponding pixel PA' and the second displacement-corresponding pixel PB' can be determined by using the first linear term of the local Taylor expansion.

[0401] [Equation 26]

[0402]

[0403]

[0404] The difference △i'j' between the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can be determined according to Equation 27.

[0405] [Equation 27]

[0406]

[0407]

[0408] The displacement vectors Vx and Vy in the x-axis direction that minimize the difference Δi'j' between the first displacement-corresponding pixel PA' and the second displacement-corresponding pixel PB can be determined by using the sum of squares of the differences Δ as in Equation 28, Φ(Vx, Vy). In other words, the displacement vectors Vx and Vy can be determined when Φ(Vx, Vy) is minimized as in Equation 29, and this can be achieved by using the maximum or minimum value of Φ(Vx, Vy).

[0409] [Equation 28]

[0410]

[0411] [Equation 29]

[0412] (Vx, Vy) = argmin Vx，Vy Φ(Vx, Vy)

[0413] Φ(Vx, Vy) is a function of displacement vectors Vx and Vy as parameters, and the maximum or minimum value can be determined by calculating the value that makes Φ(Vx, Vy) equal to 0 by partial differentiation with respect to displacement vectors Vx and Vy, as in Equation 30.

[0414] [Equation 30]

[0415]

[0416] That is, the displacement vectors Vx and Vy that minimize the value of Φ(Vx, Vy) can be determined. To solve the optimization problem, minimization can be first performed in the vertical direction and then in the horizontal direction. According to the minimization, the displacement vector Vx can be determined according to Equation 31.

[0417] [Equation 31]

[0418]

[0419] In this regard, the function clip3(x, y, z) is a function such that when z < x, the function outputs x, when z > y, the function outputs y, and when x < z < y, the function outputs z. According to Equation 31, when s1 + r > m, the displacement vector Vx can be clip3(-thBIO, thBIO, -s3 / (s1 + r)), and when s1 + r ≤ m, the displacement vector Vx can be 0.

[0420] According to the minimization, the displacement vector Vy can be determined according to Equation 32.

[0421] [Equation 32]

[0422]

[0423] In this regard, the function clip3(x, y, z) is a function such that when z < x, the function outputs x, when z > y, the function outputs y, and when x < z < y, the function outputs z. According to Equation 32, when s5 + r > m, the displacement vector Vy can be clip3(-thBIO, thBIO, -(s6 - Vx * s2 / 2 / (s5 + r)), and when s5 + r ≤ m, the displacement vector Vy can be 0.

[0424] In this regard, s1, s2, s3, s5, and s6 can be determined according to Equation 33. s4 can have the same value as the value of s2.

[0425] [Equation 33]

[0426]

[0427]

[0428]

[0429]

[0430]

[0431] As mentioned above, r and m can be adjustment parameters introduced to avoid the result of division operations being 0 or less than 0, and can be determined based on the internal bit depth d of the input video according to Equation 34. That is, the adjustment parameter m can be the minimum allowed denominator, and the adjustment parameter r can be an adjustment parameter introduced to prevent division by using 0 as the denominator when the gradient value is 0.

[0432] [Equation 34]

[0433] r = 500 * 4 d-8

[0434] m = 700 * 4 d-8

[0435] The displacement vectors Vx and Vy can have upper and lower limits ±thBIO. Because pixel-group unit motion compensation may be unreliable due to noise or irregular motion, the displacement vectors Vx and Vy can be limited by a specific threshold thBIO. The adjustment parameter thBIO can be determined based on whether the orientations of all reference frames are the same. For example, when the orientations of all reference frames are the same, the adjustment parameter thBIO can be determined as 12^(d-8-1) or 12*2^(14-d). When the orientations of all reference frames are different, the adjustment parameter thBIO can be determined as 12^(d-8-1) / 2 or 12*2^(13-d).

[0436] However, this disclosure is not limited thereto, and the values ​​of the adjustment parameters r, m, and thBIO can be determined based on information about the adjustment parameters obtained from the bitstream. In this regard, information about the adjustment parameters can be included in the strip header, the picture parameter set, the sequence parameter set, or various high-level syntax carriers.

[0437] Furthermore, the adjustment parameter can be determined based on whether bidirectional forecasting is available at different temporal intervals. For example, the adjustment parameter thBIO could be used when bidirectional forecasting is available at different temporal intervals. diff The adjustment parameter thBIO is greater than the one used when bidirectional prediction is unavailable at the same temporal level. same And adjust the parameter thBIO diff The value can be adjusted by the parameter thBIO. same Twice the value.

[0438] Figure 8b This is a reference diagram used to describe the process of determining the horizontal and vertical displacement vectors of a pixel group according to an embodiment.

[0439] Reference Figure 8bA window Ωij 810 of a predetermined size has a size of (2M+K+1)*(2N+K+1) (where M and N are integers) for a pixel group 820 of size K×K that includes multiple pixels in the current block (instead of for pixels that are bidirectionally predicted).

[0440] In this respect, with Figure 8a The difference is that the window size is increased, and apart from that difference, it can be customized according to... Figure 8a The same method is used to determine the horizontal and vertical displacement vectors for a group of pixels.

[0441] Figure 8c This is a reference diagram used to describe the process of determining the horizontal and vertical displacement vectors of a pixel according to an embodiment.

[0442] Reference Figure 8c The video decoding device 100 can determine the horizontal and vertical displacement vectors for each pixel 830. Therefore, the displacement vector 835 for each pixel 830 per unit time can be determined. In this regard, the horizontal displacement vector Vx[i,j] and the vertical displacement vector Vy[i,j] for each pixel 830 per unit time can be determined according to Equation 35. In this regard, i and j can indicate the x and y components of the pixel's coordinates. Furthermore, σ1[i,j], σ2[i,j], σ3[i,j], σ5[i,j], and σ6[i,j] can be s1, s2, s3, s5, and s6 of Equation 33, respectively.

[0443] [Equation 35]

[0444]

[0445]

[0446] Figure 8d This is a reference diagram used to describe the process of determining the horizontal and vertical displacement vectors of a pixel group according to an embodiment.

[0447] Reference Figure 8d The video decoding device 100 can determine σ1[i,j], σ2[i,j], σ3[i,j], σ5[i,j] and σ6[i,j] for each pixel included in each pixel group 840 according to the above equation 35.

[0448] The video decoding device 100 can determine the horizontal displacement vector Vx[i,j] with respect to each pixel group 840 by using σ1[i,j] and σ3[i,j] of the pixels according to Equation 36. In this respect, i and j can indicate the x and y components of the upper left coordinate of the pixel group.

[0449] [Equation 36]

[0450]

[0451] The video decoding device 100 can calculate the vertical displacement vector of a pixel group using a method similar to that used to calculate the horizontal displacement vector. That is, the vertical displacement vector of a pixel group can be calculated by adding variables to the denominator and by adding variables to the numerator, where the variables added to the denominator are used by the denominator for each pixel to calculate the vertical displacement vector of the pixel group, and the variables added to the numerator are used by the numerator for each pixel to calculate the vertical displacement vector of the pixel group.

[0452] The video decoding device 100 can determine the vertical displacement vector Vy[i,j] for each pixel group 840 using σ2[i,j], σ5[i,j], σ6[i,j], and Vx[i,j] of the pixels according to Equation 37. In this respect, the horizontal displacement vector Vx[i,j] can be determined according to Equation 36.

[0453] [Equation 37]

[0454]

[0455] like Figure 8c As shown, when determining the horizontal and vertical displacement vectors for each pixel, two division operations must be performed for each pixel. However, as... Figure 8d As shown, when determining the horizontal and vertical displacement vectors for each group, a 2 / (L^2) division operation must be performed on each pixel when the size of the pixel group is L×L (where L is an integer). Therefore, determining the horizontal and vertical displacement vectors for each group reduces the number of required division operations, thereby reducing hardware complexity and computation time.

[0456] In the above text, refer to Figure 8d The pixel group is described in detail as having a size of 2×2, but this disclosure is not limited thereto, and therefore, the pixel group may have a size of L×L (where L is an integer).

[0457] In this regard, the size L of the pixel group can be determined according to Equation 38. W and H can refer to the width and height of the current block, respectively.

[0458] [Equation 38]

[0459]

[0460] When the video decoding device 100 performs pixel group unit motion compensation together with block enlargement by the window size, and when the video decoding device 100 performs only pixel group unit motion compensation, the video decoding device 100 can perform as many memory access operations and multiplication operations as there are for each case according to Table 1 below. In this regard, assume that the length T of the signal filter is 8, the length T of the gradient filter is 6, the size of the pixel group is L×L, the size of the block is N×N, and the window size 2M+1 per pixel is 5.

[0461] [Table 1]

[0462]

[0463] In other words, when the video decoding device 100 performs pixel group unit motion compensation, the video decoding device 100 can perform the division operation by reducing the number of division operations by dividing the value corresponding to the size of the pixel group by L^2.

[0464] Figure 9a This is a diagram illustrating the process of adding an offset value after performing filtering and determining the gradient value in the horizontal or vertical direction by performing descaling, according to an embodiment.

[0465] Reference Figure 9a The video decoding device 100 can determine gradient values ​​in the horizontal or vertical direction by performing filtering on pixels whose predetermined directional components are at integer positions using a first 1D filter and a second 1D filter. However, the values ​​obtained by performing filtering on pixels whose predetermined directional components are at integer positions using the first or second 1D filter may exceed a predetermined range. This phenomenon is called overflow. The coefficients of the 1D filter can be determined to be integers for performing integer operations (rather than inaccurate and complex fractional operations). The coefficients of the 1D filter can be scaled to be determined as integers. When filtering is performed using the scaled coefficients of the 1D filter, integer operations can be performed, but the filtered values ​​will be larger than when filtering is performed using the unscaled coefficients of the 1D filter, which may cause overflow. Therefore, to prevent overflow, descaling can be performed after filtering is performed using the 1D filter. In this regard, the descaling step may include right bit shifting according to the number of descaling bits. While maximizing the accuracy of the calculation, the number of descaling bits can be determined based on the maximum number of bits in the register used for the filtering operation and the maximum number of bits in the temporary buffer storing the result of the filtering operation. Specifically, the number of descaling bits can be determined based on the internal bit depth, the number of scaling bits of the interpolation filter, and the number of scaling bits of the gradient filter.

[0466] The following describes the descaling process performed during the process of generating the interpolated values ​​in the vertical direction by first filtering the pixels at integer positions using a vertical interpolation filter to determine the gradient values ​​in the horizontal direction, and then filtering the interpolated values ​​in the vertical direction using a horizontal gradient filter.

[0467] According to Equation 12 above, the video decoding device 100 can first perform filtering on pixels at integer positions using a vertical interpolation filter to determine the gradient value in the horizontal direction. In this regard, shift1 can be determined as b-8. In this regard, b can indicate the internal bit depth of the input image. Hereinafter, referring to Table 2, the bit depth of the register (Reg Bitdepth) and the bit depth of the temporary buffer (Temp Bitdepth) when descaling is actually performed based on shift1 will now be described.

[0468] [Table 2]

[0469]

[0470]

[0471] In this regard, the values ​​of the variables in Table 2 can be determined according to Equation 39.

[0472] [Equation 39]

[0473] RegMin=Min(I)*FilterSumPos+Max(I)*FilterSumNeg

[0474] RegMax=Max(I)*FilterSumPos+Min(I)*FilterSumNeg

[0475] Reg BitDepth=ceiling(log2(RegMax-RegMin)+1)

[0476] TempMin=(RegMin+offset1)>>shift1

[0477] TempMax=(RegMax+offset1)>>shift1

[0478] Temp BitDepth=ceiling(log2(TempMax-TempMin)+1)

[0479] In this regard, Min(I) indicates the minimum value of pixel value I determined by the internal bit depth, and Max(I) indicates the maximum value of pixel value I determined by the internal bit depth. FilterSumPos indicates the maximum value of the sum of positive filter coefficients, and FilterSumNeg indicates the minimum value of the sum of negative filter coefficients.

[0480] For example, when assuming the use of Figure 7c When using a gradient filter FracFilter with units of 1 / 4 pel, FilterSumPos can be 88 and FilterSumNeg can be -24.

[0481] The function Ceiling(x) can be a function of the smallest integer among real numbers x whose output is equal to or greater than x. offset1 is an offset value added to the value that has been filtered to prevent rounding errors that may occur when descaling is performed using shift1, and offset1 can be determined as 2^(shift1-1).

[0482] Referring to Table 2, when the internal bit depth b is 8, the register bit depth can be 16; when the internal bit depth b is 9, the register bit depth can be 17; and when the internal bit depth b is 10, 11, 12, or 16, the register bit depth can be 18, 19, 20, or 24, respectively. When the register used for filtering is a 32-bit register, overflow will not occur because the bit depth of all registers in Table 2 does not exceed 32.

[0483] Similarly, when the internal bit depth b is 8, 9, 10, 11, 12, and 16, the bit depth (TempBitDepth) of the temporary buffer is 16. When the temporary buffer used to store the value that has been filtered and then descaled is a 16-bit buffer, no overflow will occur because the bit depth of all temporary buffers in Table 2 is 16 and therefore does not exceed 16.

[0484] According to Equation 12, the video decoding device 100 can generate vertically interpolated values ​​to determine the gradient values ​​in the horizontal direction by first performing filtering on pixels at integer positions using a vertical interpolation filter, and then performing filtering on the vertically interpolated values ​​using a horizontal gradient filter according to Equation 13. In this respect, shift2 can be determined as p + q - shift1. In this respect, p can indicate values ​​for including... Figure 7c The interpolation filter coefficients shown are scaled by the number of bits, and q can indicate the number of bits for the filter coefficients including... Figure 7aThe gradient filter coefficients shown are scaled by the number of bits. For example, p can be 6 and q can be 4, therefore shift2 = 18 - b.

[0485] The reason shift2 is determined as described above is that shift1 + shift2 (i.e., the sum of the number of bits to be scaled) should be equal to the sum of the number of bits amplified for the filter (p + q), so that the final filtering result value is the same whether the filter coefficients are amplified or not.

[0486] In the following text, the register bit depth and temporary buffer bit depth will now be described with reference to Table 3 when descaling is actually performed based on shift2.

[0487] [Table 3]

[0488]

[0489] In this respect, the values ​​of the variables in Table 3 can be determined according to Equation 40.

[0490] [Equation 40]

[0491] RegMin=TempMin*FilterSumPos+TempMax*FilterSumNeg

[0492] RegMax=TempMax*FilterSumPos+TempMin*FilterSumNeg

[0493] Reg BitDepth=ceiling(log2(RegMax-RegMin)+1)

[0494] TempMin=(RegMin+offset2)>>shift2

[0495] TempMax=(RegMax+offset2)>>shift2

[0496] Temp BitDepth=ceiling(log2(TempMax-TempMin)+1)

[0497] In this regard, TempMax indicates the TempMax in Table 3, and TempMin indicates the TempMin in Table 3. FilterSumPos indicates the maximum sum of the positive filter coefficients, and FilterSumNeg indicates the minimum sum of the negative filter coefficients. For example, when assuming the use of... Figure 7c When using the gradient filter gradFilter shown in 1 / 4pel units, FilterSumPos can be 68 and FilterSumNeg can be -68.

[0498] offset2 is an offset value added to the value that has been filtered to prevent rounding errors that may occur when descaling is performed using shift2, and offset2 can be determined as 2^(shift2-1).

[0499] shift1 and shift2 can be determined as described above, but this disclosure is not limited thereto, and therefore, shift1 and shift2 can be determined differently in such a way that the sum of shift1 and shift2 can be equal to the sum of the scaling bits for the filter. In this respect, the values ​​of shift1 and shift2 can be determined based on the premise that no overflow occurs. shift1 and shift2 can be determined based on the internal bit depth of the input image and the scaling bits for the filter.

[0500] However, shift1 and shift2 do not necessarily need to be determined such that their sum equals the sum of the scaling bits for the filter. For example, shift1 could be determined to be d-8, and shift2 could be determined to be a fixed number.

[0501] When shift1 is the same as before and shift2 is a fixed number of 7, the OutMax, OutMin, and Temp Bitdepth described in Table 3 can be changed. The Temp Bitdepth of the temporary buffer will now be described in Table 4 below.

[0502] [Table 4]

[0503] b OutMax OutMin Temp Bitdepth 8 15173 -15174 16 9 15203 -15204 16 10 15218 -15219 16 11 15225 -15226 16 12 15229 -15230 16 16 15233 -15234 16

[0504] Unlike Table 3, in Table 4, the temporary bit depth is equal to 16 for all b values. Furthermore, when assuming the result data is stored using a 16-bit temporary buffer, the temporary bit depth is less than 16, preventing overflow for all input images. On the other hand, referring to Table 3, when the input image's internal bit depth is 12 and 16, assuming the result data is stored using a 16-bit temporary buffer, the temporary bit depth is greater than 16, potentially leading to overflow.

[0505] When shift2 is a fixed number, the result obtained by performing filtering without using the scaled filter coefficients may differ from the result obtained by performing filtering with the scaled filter coefficients followed by descaling. In this case, it will be apparent to those skilled in the art that descaling is required separately.

[0506] The descaling process described above is performed during the following process: first, the pixels at integer positions are filtered using a vertical interpolation filter to produce interpolated values ​​in the vertical direction in order to determine the gradient values ​​in the horizontal direction; then, the interpolated values ​​in the vertical direction are filtered using a gradient filter in the horizontal direction. However, it will be apparent to those skilled in the art that when filtering is performed on pixels where the predetermined direction component is an integer, descaling can be performed in a similar manner to determine the gradient values ​​in the horizontal and vertical directions via a combination of various 1D filters.

[0507] Figure 9b This is a diagram illustrating the process of adding an offset value after performing filtering according to another embodiment, and the process of determining the gradient value in the horizontal or vertical direction by performing descaling.

[0508] Reference Figure 9b The video decoding device 100 can perform filtering based on the input of fractional pixels and integer pixels of the reference pixel. In this regard, it is assumed that the fractional pixels of the reference image have been determined by applying 1D filters in the horizontal and vertical directions to the integer pixels of the reference pixel.

[0509] Video decoding device 100 can determine gradient values ​​in the horizontal or vertical direction by performing filtering on pixels whose predetermined directional position components are fractions and integers using a 1D filter in the horizontal or vertical direction. However, the values ​​obtained by performing filtering on pixels whose predetermined directional position components are fractions and integers using a first 1D filter may exceed a predetermined range. This phenomenon is called overflow. The coefficients of the 1D filter can be determined to be integers for performing integer operations (rather than inaccurate and complex fractional operations). The coefficients of the 1D filter can be scaled to be determined as integers. When filtering is performed using the scaled coefficients of the 1D filter, integer operations can be performed, but the filtered values ​​will be larger than when filtering is performed using the unscaled coefficients of the 1D filter, which may cause overflow. Therefore, to prevent overflow, descaling can be performed after filtering is performed using the 1D filter. In this regard, the descaling step may include shifting bits to the right by the number of descaling bits (shift1). The number of descaling bits can be determined based on the maximum number of bits in the register used for the filtering operation and the maximum number of bits in the temporary buffer storing the results of the filtering operation while maximizing the accuracy of the calculation. Specifically, the number of descaling bits can be determined based on the internal bit depth and the number of scaling bits of the gradient filter.

[0510] Figure 9c This is a diagram used to describe the necessary range for determining the horizontal and vertical displacement vectors during the process of performing pixel-unit motion compensation on the current block.

[0511] Reference Figure 9c During the processing of pixel-unit motion compensation for a reference block 910 corresponding to the current block, the video decoding device 100 can determine the horizontal displacement vector and the vertical displacement vector per unit time at pixel 915 by using a window 920 surrounding pixel 915 located in the upper left of the reference block 910. In this regard, the horizontal or vertical displacement vector per unit time can be determined by using the pixel values ​​and gradient values ​​of pixels located outside the range of the reference block 910. Similarly, during the processing of determining the horizontal and vertical displacement vectors for pixels located at the boundaries of the reference block 910, the video decoding device 100 can determine the pixel values ​​and gradient values ​​of pixels located outside the range of the reference block 910. Therefore, the video decoding device 100 can determine the horizontal and vertical displacement vectors by using a block 925 with a range larger than the reference block 910. For example, when the size of the current block is A×B and the window size of each pixel is (2M+1)×(2N+1), the size of the range used to determine the horizontal displacement vector and the vertical displacement vector can be (A+2M)×(B+2N).

[0512] Figures 9d to 9e This is an illustration of the range of regions used in the process of performing pixel-unit motion compensation, according to various embodiments.

[0513] Reference Figure 9d During the pixel-unit motion compensation process, the video decoding device 100 can determine the horizontal and vertical displacement vectors for each pixel included in the reference block 935 based on block 935, where block 935 has a range of windows whose size has been expanded to include pixels located at the boundaries of the reference block 930. However, when determining the horizontal and vertical displacement vectors, the video decoding device 100 requires the pixel values ​​and gradient values ​​of the pixels located in block 935, and to obtain these pixel values ​​and gradient values, the video decoding device 100 can use an interpolation filter or a gradient filter. During the processing of applying the interpolation filter or gradient filter to the boundary pixels of block 935, the pixel values ​​of neighboring pixels can be used, and therefore, pixels located outside the block boundaries can be used. Therefore, pixel-unit motion compensation can be performed using block 940, where the range of block 940 is further expanded based on a value obtained by subtracting 1 from the number of taps of the interpolation filter or gradient filter. Therefore, when the block size is N×N, the window size of each pixel is (2M+1)×(2M+1), and the length of the interpolation filter or gradient filter is T, the size of block 940 can be (N+2M+T-1)×(N+2M+T-1).

[0514] Reference Figure 9e During pixel-unit motion compensation processing, the video decoding device 100 can determine the horizontal and vertical displacement vectors for each pixel by using the pixel values ​​and gradient values ​​of the pixels located in the reference block 945, without expanding the reference block 945 based on the size of the window of pixels located at the boundary of the reference block 945. Figure 9e This describes a specific process performed by the video decoding device 100 to determine the displacement vector per unit time in the horizontal direction and the displacement vector per unit time in the vertical direction without enlarging the reference block. In this regard, an interpolation filter or a gradient filter can be used for the reference block 945 to obtain the pixel values ​​and gradient values ​​of the pixels, and pixel-unit motion compensation can be performed using an enlarged block 950. Therefore, with a block size of N×N, a window size of (2M+1)×(2M+1) for each pixel, and a length of T for the interpolation filter or gradient filter, the size of block 950 can be (N+T-1)×(N+T-1).

[0515] Figure 9fThis is a diagram illustrating the process of determining the horizontal and vertical displacement vectors without enlarging the reference block.

[0516] Reference Figure 9f For a pixel located outside the boundary of reference block 955, video decoding device 100 can adjust the position of the pixel to the position of the available pixel among the pixels closest to the pixel and located within the boundary of reference block 955, and can determine the pixel value and gradient value of the pixel located outside the boundary as the pixel value and gradient value of the available pixel at the nearest position. In this regard, video decoding device 100 can adjust the position of the pixel located outside the boundary of reference block 955 to the position of the available pixel at the nearest position according to the equations i′=i′<0? 0: i′; i′>H-1? H-1: i′ and j′=j′<0? 0: j′; j′>W-1? W-1: j′.

[0517] In this respect, i′ can indicate the x-coordinate value of the pixel, j′ can indicate the y-coordinate value of the pixel, and H and W can indicate the height and width of the reference block. In this respect, it is assumed that the top-left position in the reference block is (0, 0). When the top-left position in the reference block is (xP, yP), the final pixel position can be (i′+xP, j′+yP).

[0518] Refer to the return Figure 9d In block 935, where the window size of each pixel has been enlarged, the position of a pixel located outside the boundary of reference block 930 can be adjusted to the position of an adjacent pixel inside the boundary of reference block 930. Therefore, as Figure 9e As shown, the video decoding device 100 can determine the horizontal and vertical displacement vectors about each pixel in the reference block 945 by using the pixel values ​​and gradient values ​​of the reference block 945.

[0519] Therefore, the video decoding device 100 performs pixel-unit motion compensation without expanding the window size of the reference block 945 for each pixel, which reduces the number of memory accesses for reference pixel values ​​and the number of multiplication operations, thereby reducing computational complexity.

[0520] When the video decoding device 100 performs block-unit motion compensation (as in operation according to the High Efficiency Video Coding (HEVC) standard), performs pixel-unit motion compensation using block enlargement by window size, and performs pixel-unit motion compensation without block enlargement, the video decoding device 100 can perform as many memory access operations and multiplication operations as there are for each case according to Table 5 below. In this regard, it is assumed that the length of the signal (interpolation) filter is 8, the length of the gradient filter is 6, the block size is N×N, and the window size 2M+1 per pixel is 5.

[0521] [Table 5]

[0522]

[0523] In block-unit motion compensation according to the HEVC standard, an 8-tap interpolation filter is used for one sample, therefore, 8 neighboring samples are required. Thus, when the reference block size is N×N, 8-tap interpolation requires (N+7)×(N+7) reference samples, and because bidirectional motion prediction compensation will be performed, two reference blocks will be used. Therefore, in block-unit motion compensation according to the HEVC standard, 2*(N+7)×(N+7) memory accesses are required, as shown in Table 5. When performing pixel-unit motion compensation using block enlargement, at M=2, when performing pixel-unit motion compensation on an enlarged block of size (N+4)×(N+4) using an 8-tap interpolation filter or gradient filter, (N+4+7)×(N+4+7) reference samples are required. Since bidirectional motion prediction compensation will be performed, two reference blocks will be used. Therefore, in pixel-unit motion compensation performed using block enlargement, as shown in Table 5, 2×(N+4+7)×(N+4+7) memory accesses are required.

[0524] Without block scaling, pixel-unit motion compensation requires (N+7)×(N+7) reference samples, as is the case in unit motion compensation according to the HEVC standard, since there is no block scaling. Furthermore, since bidirectional motion prediction compensation will be performed, two reference blocks will be used. Therefore, pixel-unit motion compensation without block scaling requires 2*(N+7)×(N+7) memory accesses, as shown in Table 5.

[0525] In block-unit motion compensation according to the HEVC standard, an 8-tap interpolation filter is used for one sample. Therefore, the number of samples required to perform the first interpolation in the horizontal direction is (N+7)×N samples. The number of samples required to perform the second interpolation in the vertical direction is N×N samples. The number of multiplication operations required for each 8-tap interpolation filter is 8, and because two reference blocks will be used for bidirectional motion prediction compensation, 2*8*{(N+7)*N+N*N)} multiplication operations are required in block-unit motion compensation according to the HEVC standard, as shown in Table 5.

[0526] When performing pixel-unit motion compensation using block enlargement, the block size is enlarged to perform pixel-unit motion compensation. Therefore, in order to determine the pixel value at the fractional pixel position using an 8-tap interpolation filter for the enlarged block of size (N+4)×(N+4), as shown in Table 5, a total of 2*8*{(N+4+7)×(N+4)+(N+4)×(N+4)} multiplication operations are required.

[0527] When performing pixel-unit motion compensation using block enlargement, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine the gradient values ​​in the horizontal or vertical directions. Because the block size has been enlarged, as shown in Table 5, a total of 2*6*{(N+4+5)×(N+4)+(N+4)×(N+4)}*2 multiplication operations are required to determine the gradient values ​​for the enlarged block of size (N+4)×(N+4) using the 6-tap gradient filter and the 6-tap interpolation filter.

[0528] Without block scaling, pixel-unit motion compensation requires (N+7)×(N+7) reference samples, as is the case in block-unit motion compensation according to the HEVC standard. Since bidirectional motion prediction compensation will be performed, two reference blocks will be used. Therefore, pixel-unit motion compensation without block scaling requires a total of 2*8*{(N+7)×N+N×N} multiplication operations to determine the pixel value at the fractional pixel position for a block of size N×N using an 8-tap interpolation filter.

[0529] When performing pixel-unit motion compensation without block enlargement, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine the gradient values ​​in the horizontal or vertical directions. To determine the gradient values ​​for an N×N block using the 6-tap gradient filter and the 6-tap interpolation filter, as shown in Table 5, a total of 2*6*{(N+5)×N+N×N}*2 multiplication operations are required.

[0530] Figures 9g to 9iThis is a diagram illustrating the process of determining the horizontal and vertical displacement vectors without enlarging the reference block, according to other embodiments.

[0531] As referred above Figure 9d During the pixel-unit motion compensation process, the video decoding device 100 can determine the horizontal and vertical displacement vectors of each pixel included in the reference block 930 based on block 935, wherein the scope of block 935 is expanded to include the window size of pixels located at the boundary of the reference block 930. For example, when the window size is (2M+1)×(2M+1), and the window is applied to pixels located at the boundary of the reference block 930, the video decoding device 100 can determine the horizontal and vertical displacement vectors of each pixel by referring to the pixel values ​​and gradient values ​​of pixels extending up to M beyond the reference block 930.

[0532] Hereinafter, a method according to another embodiment, performed by a video decoding device 100, will now be described, which determines the values ​​of the horizontal and vertical displacement vectors for each pixel (s1 to s6 of Equation 33) by using only the pixel values ​​and gradient values ​​of a reference block corresponding to the current pixel, without referring to the pixel values ​​and gradient values ​​of pixels outside the reference block. Here, it is assumed that the window size is 5×5. For convenience, only pixels with the current pixel located in the horizontal direction will be described now. It will be apparent to those skilled in the art that the weights can be determined in the same way for pixels with the current pixel located in the vertical direction.

[0533] The video decoding device 100 performs calculations on functions relating to the following: pixel values ​​(P0(i', j') and P1(i', j')), gradient values ​​in the horizontal or vertical direction (...). and ) and the difference (τ0, τ1) between the current frame's POC and the reference frame's POC for each pixel, wherein each pixel is included in the window of the current pixel in which the horizontal displacement vector or the vertical displacement vector will be determined, and the video decoding device 100 determines the values ​​for determining the horizontal displacement vector and the vertical displacement vector for each pixel by adding the results of the calculations performed on each pixel (s1 to s6 of Equation 33).

[0534] In this respect, the video decoding device 100 can determine the values ​​for determining the horizontal and vertical displacement vectors with respect to each pixel by assigning the same weight to each pixel, multiplying the weight by each result value of the calculation performed on each pixel, and then summing the multiplied values ​​(s1 to s6 of Equation 33).

[0535] Reference Figure 9g During the process of determining the horizontal and vertical displacement vectors of the current pixel 961 in the current block 960, the video decoding device 100 can determine that pixels in the window have the same weight 1. The video decoding device 100 can determine the values ​​used to determine the horizontal and vertical displacement vectors of the current pixel by multiplying the weights determined for each pixel by the calculated values ​​for each pixel, and then summing the calculated values.

[0536] Reference Figure 9h When pixel 971 is immediately adjacent to the boundary of block 970 within block 970, the video decoding device 100 may assign a weight of 3 to the pixel, which is located inside block 970 and adjacent to its boundary. The video decoding device 100 may assign a weight of 1 to other pixels 973.

[0537] Reference Figure 9i When the current pixel 981 is close to the boundary of the current block 980 (when the current pixel 981 is one pixel away from the boundary), the video decoding device 100 may determine the weight for the pixel 982 located outside the boundary of the current block 980 to be 0, and may determine the weight for the pixel 983 adjacent to the boundary of the current block 980 to be 2. The video decoding device 100 may determine the weight for other pixels 984 to be 1.

[0538] For reference Figures 9g to 9i The video decoding device 100 can determine the values ​​of the horizontal and vertical displacement vectors for each pixel (s1 to s6 of Equation 33) by assigning different weights to pixels in the window according to the position of the current pixel, using the pixel values ​​and gradient values ​​of pixels located in a reference block corresponding to the current block, without referring to the pixel values ​​and gradient values ​​of pixels outside the reference block.

[0539] Figure 9j This is a diagram illustrating the process of determining the horizontal and vertical displacement vectors for each pixel by applying exponential smoothing techniques in the vertical and horizontal directions, without expanding the block, by referring to the pixel values ​​and gradient values ​​of a reference block.

[0540] Reference Figure 9j The video decoding device 100 can perform calculations on functions relating to: the pixel values ​​(P0(i', j') and P1(i', j')) of the corresponding reference pixels included in the corresponding reference block, and the gradient values ​​of the corresponding reference pixels included in the pixels of the corresponding reference block in the horizontal or vertical direction. And the difference (τ0, τ1) between the POC of the reference frame and the POC of the current frame for each pixel included in the current block 990.

[0541] The video decoding device 100 determines the values ​​(s1 to s6; σ) used to determine the horizontal and vertical displacement vectors with respect to the current pixel by summing the result of a function calculation on the current pixel and the result of a function calculation on the neighboring pixels. k (k = 1, 2, 3, 4, 5, 6)). That is, the values ​​(s1 to s6) used to determine the horizontal and vertical displacement vectors with respect to the current pixel can be expressed as a weighted average of the values ​​calculated with respect to the current pixel and its neighboring pixels according to Equation 41 below. In this respect, the position coordinates of the pixels included in the window Ω are (i', j'). Furthermore, W[i', j'] can indicate the weights used for the pixels included in the window Ω. In this respect, the size of the window Ω can be (2M+1)×(2M+1) (where M is an integer). Furthermore, the function A k The value of [i', j'] can be the value of a function of the following: the pixel value of the corresponding reference pixel (P0(i', j') and P1(i', j')) for the pixel at position (i', j') included in the window Ω, and the gradient value of the corresponding reference pixel included in the corresponding reference block in the horizontal or vertical direction (i', j') Ix,y[i',j'](0,1)) and the difference between the POC of the reference frame and the POC of the current frame (τ0,τ1).

[0542] [Equation 41]

[0543]

[0544] For example, the video decoding device 100 can determine the values ​​(s1 to s6) for determining the horizontal and vertical displacement vectors of the current pixel by summing the result of a function calculation on the current pixel and the result of a function calculation on the pixels in the window of the current pixel. When assuming the window size is 5×5, the values ​​required to determine the horizontal and vertical displacement vectors of the pixel 991 located at the boundary of the current block 990 need to be referenced to the result of a function calculation on the pixels in the window 992 for pixel 991. Therefore, the result of a function calculation on the pixels of block 996, which is expanded by 2 pixels horizontally and vertically compared to the current block 990, needs to be referenced. The function A can be determined as shown in Table 6 below. k The values ​​of [i', j'].

[0545] [Table 6]

[0546]

[0547] In this regard, Ix (0) [i', j'] and Ix (1) [i', j'] can be respectively and Iy (0) [i', j'] and Iy (1) [i', j'] can be respectively and And I (0) [i', j'] and I (1) [i', j'] can be P0(i', j') and P1(i', j') respectively.

[0548] Video decoding device 100 can apply exponential smoothing technology to A in both the vertical and horizontal directions. k [i', j'] are used to perform an average to determine the value of A. k The weighted average of [i', j'].

[0549] Reference Figure 9j The video decoding device 100 can apply exponential smoothing technology to A in the horizontal direction. k [i', j']. The following will now describe in detail the process performed by the video decoding device 100 by applying exponential smoothing technology to A in the horizontal direction. k [i', j'] determine about A k Processing of the weighted average of [i', j'].

[0550] First, the video decoding device 100 will now be described as performing an average in the right direction by applying exponential smoothing technology. The video decoding device 100 can perform an average in the right direction by using pseudocode 1 as follows: k [i', j'] performs averaging. In this regard, H can indicate the height of the current block, W can indicate the width of the current block, and Stride can indicate the distance between one row and the next row in the 1D array. That is, a two-dimensional (2D) array A[i, j] can be represented as A[i+j*Stride] of a 1D array.

[0551] [Pseudocode 1]

[0552]

[0553] For example, when the size of the current block is 4×4, and A[0], A[1], A[2] and A[3] are a, b, c and d respectively, the result of processing using pseudocode 1 can be: A′[0] can be 2a, A′[1] can be A[0] / 2+b=a+b, A′[2] can be A[1] / 2+c=a / 2+b / 2+c, and A′[3] can be A[2] / 2+d=a / 4+b / 4+c / 2+d.

[0554] In the following text, the video decoding device 100 will now be described as performing an average in the left direction by applying exponential smoothing technology. The video decoding device 100 can perform an average in the left direction by using the following pseudocode 2. k [i′, j′] is averaged. In this respect, H can indicate the height of the current block, W can indicate the width of the current block, and Stride can indicate the distance between one row and the next row in the 1D array. That is, the 2D array A[i, j] can be represented as A[i+j*Stride] of the 1D array.

[0555] [Pseudocode 2]

[0556]

[0557] For example, when the current block size is 4×4, and the averaged A′[0], A′[1], A′[2] and A′[3] in the right direction are 2a, a+b, a / 2+b / 2+c and a / 4+b / 4+c / 2+d, the result of processing using pseudocode 2 can be: A[3] can be 2*A′[3]=a / 2+b / 2+c+2d, A[2] can be A′[2]+A[3] / 2=a / 2+b / 2+c+a / 4+b / 4+c / 2+d=3a / 4+3b / 4+3c / 2+d, A[1] can be A′[1]+A[2] / 2..., A[0] can be A′[0]+A[1] / 2.

[0558] In the following text, the video decoding device 100 will now be described as performing an average in the downward direction by applying an exponential smoothing technique. The video decoding device 100 can perform an average on Ak[i′, j′] in the downward direction using the following pseudocode 3. In this regard, H can indicate the height of the current block, W can indicate the width of the current block, and Stride can indicate the distance between one row and the next row in the 1D array. That is, the 2D array A[i, j] can be represented as A[i+j*Stride] of the 1D array.

[0559] [Pseudocode 3]

[0560]

[0561] In the following text, the video decoding device 100 will now be described as performing an average in the upward direction by applying exponential smoothing technology. The video decoding device 100 can perform an average in the upward direction by using the following pseudocode 4. k [i′, j′] is averaged. In this respect, H can indicate the height of the current block, W can indicate the width of the current block, and Stride can indicate the distance between one row and the next row in the 1D array. That is, the 2D array A[i, j] can be represented as A[i+j*Stride] of the 1D array.

[0562] [Pseudocode 4]

[0563]

[0564] In other words, the video decoding device 100 can use exponential smoothing technology to smooth A in both the vertical and horizontal directions. k The values ​​of [i′, j′] are averaged, and the resulting value for each pixel determined by averaging can be transformed into σ. k [i, j], where σ k [i, j] are the values ​​needed to determine the horizontal or vertical displacement vector for each pixel. In this respect, when averaging is performed using an exponential smoothing technique, the current pixel has the highest weight regarding its position, and the weight of that pixel decreases exponentially as it moves further away from the current pixel.

[0565] Therefore, the video decoding device 100 can determine σ by performing an average of the current block 990 in both the vertical and horizontal directions using exponential smoothing techniques. k [i, j], where σ k [i, j] is the weighted average value required to determine the horizontal or vertical displacement vector for each pixel. That is, the video decoding device 100 can determine the horizontal or vertical displacement vector for each pixel by referring only to the pixel values ​​and gradient values ​​of the reference block corresponding to the current block 990, without referring to the pixel values ​​and gradient values ​​of the reference block corresponding to the block 996 expanded from the current block 990.

[0566] Figure 9k This is a diagram illustrating the process of determining the pixel values ​​and gradient values ​​of reference pixels in a reference block in the horizontal and vertical directions using a filter, according to an embodiment, to perform motion compensation on the current block.

[0567] Reference Figure 9kThe video decoding device 100 can perform pixel-level motion compensation and block-level motion compensation on the current block by using the pixel values ​​and gradient values ​​of reference pixels in the reference block corresponding to the current block. Therefore, in order to perform pixel-level motion compensation and block-level motion compensation on the current block, the pixel values ​​and gradient values ​​of reference pixels in the reference block corresponding to the current block must be determined. In this regard, the unit of the pixel values ​​and gradient values ​​of the reference pixels in the reference block can be fractional pixel units. For example, the unit of the pixel values ​​and gradient values ​​of the reference pixels in the reference block can be 1 / 16 pel units.

[0568] The video decoding device 100 can perform filtering on the pixel values ​​of integer pixels in the reference block to determine the pixel value and gradient value of the reference pixel in the reference block in fractional pixel units.

[0569] First, the video decoding device 100 can determine the pixel value of the reference pixel in the reference block by applying a horizontal 8-tap signal filter (also known as an interpolation filter) and a vertical 8-tap signal filter to the pixel value of the integer pixel in the reference block.

[0570] The video decoding device 100 can determine the pixel value of a reference pixel having a position component with fractional pixel units in the horizontal direction by applying a horizontal 8-tap signal filter to the pixel values ​​of integer pixels in the reference block, and can store the determined pixel value in a buffer. The video decoding device 100 can also determine the pixel value of a reference pixel having a position component with fractional pixel units in the vertical direction by applying a vertical 8-tap signal filter to the pixel values ​​of reference pixels having a position component with integer units in the vertical direction.

[0571] The video decoding device 100 can perform filtering to determine the gradient value of the reference pixel in the horizontal direction by first applying a vertical 6-tap signal filter to the pixel value of the integer pixel in the reference block and then applying a horizontal 6-tap gradient filter to it.

[0572] In other words, the video decoding device 100 can perform filtering by applying a vertical 6-tap signal filter to the pixel values ​​of integer pixels in the reference block to determine the pixel values ​​of reference pixels with a position component in the vertical direction that has fractional pixel units, and can store the determined pixel values ​​in a buffer.

[0573] The video decoding device 100 can determine the gradient value of the reference pixel with a position component in the horizontal direction by applying a horizontal 6-tap gradient filter to the pixel value of the reference pixel with an integer position component in the horizontal direction.

[0574] The video decoding device 100 can perform filtering to determine the gradient value of the reference pixel in the reference block in the horizontal direction by first applying a vertical 6-tap signal filter to the pixel value of the integer pixel in the reference, and then applying a horizontal 6-tap gradient filter to it.

[0575] In other words, the video decoding device 100 can perform filtering by applying a vertical 6-tap signal filter to the pixel values ​​of integer pixels in the reference block to determine the gradient value of the reference pixel with a position component in the vertical direction that has fractional pixel units, and can store the determined gradient value in a buffer.

[0576] The video decoding device 100 can determine the gradient value of the reference pixel with a position component in the horizontal direction in the vertical direction by applying a horizontal 6-tap gradient filter to the gradient value of the reference pixel with an integer position component in the horizontal direction in the vertical direction.

[0577] In other words, the video decoding device 100 can apply two 1D filters to determine the pixel value of a reference pixel in the reference block, the gradient value of the reference pixel in the reference block in the horizontal direction, and the gradient value of the reference pixel in the reference block in the vertical direction, respectively. In this regard, multiplication operations can be performed on the coefficients of each filter and the value of the corresponding pixel. For example, to determine the gradient value of a reference pixel in the reference block in the horizontal direction, a 6-tap signal filter and a 6-tap gradient filter can be used, and 12 multiplication operations can be performed for each pixel. Similarly, to determine the gradient value of a reference pixel in the reference block in the vertical direction, a 6-tap signal filter and a 6-tap gradient filter can be used, and 12 multiplication operations can be performed for each pixel.

[0578] Figure 9l This is a diagram illustrating a process, according to another embodiment, of determining the pixel values ​​and gradient values ​​of reference pixels in a reference block in the horizontal and vertical directions using a filter to perform motion compensation on the current block.

[0579] Reference Figure 9l The video decoding device 100 can perform pixel-level motion compensation and block-level motion compensation on the current block by using the pixel values ​​and gradient values ​​of reference pixels in the reference block corresponding to the current block. Therefore, in order to perform pixel-level motion compensation and block-level motion compensation on the current block, the pixel values ​​and gradient values ​​of reference pixels in the reference block corresponding to the current block must be determined. In this regard, the unit of the pixel values ​​and gradient values ​​of the reference pixels in the reference block can be fractional pixel units. For example, the unit of the pixel values ​​and gradient values ​​of the reference pixels in the reference block can be 1 / 16 pel units.

[0580] The video decoding device 100 can perform filtering on the pixel values ​​of integer pixels in the reference block to determine the pixel value and gradient value of the reference pixel in the reference block in fractional pixel units.

[0581] Regarding Figure 9k Unlike the descriptions, the video decoding device 100 can determine the gradient value of a reference pixel in the reference block in the horizontal direction by first applying a horizontal 8-tap signal filter and a vertical 8-tap signal filter to the pixel values ​​of integer pixels in the reference block, and then applying a horizontal 5-tap gradient filter to the pixel values ​​of reference pixels in the reference block. Furthermore, the video decoding device 100 can determine the gradient value of a reference pixel in the vertical direction by applying a vertical 5-tap gradient filter to the pixel values ​​of reference pixels in the reference block.

[0582] The video decoding device 100 can determine the pixel value of a reference pixel at a position with fractional pixel units by applying two 1D signal (interpolation) filters, and then determine the gradient value of the reference pixel in the reference block in the horizontal or vertical direction by applying two 1D gradient filters in parallel to the pixel value of the reference pixel at the position with fractional pixel units.

[0583] The video decoding device can determine the gradient value Ix(k) of the reference pixel in the horizontal direction by applying a horizontal 5-tap gradient filter (the filter coefficients are {9, -48, 0, 48, 9}; but the filter coefficients are not limited thereto) to the pixel value of the reference pixel in the reference block according to Equation 42 below. In this respect, k can have a value of one of 0 and 1, where 0 and 1 indicate reference frame 0 and 1, respectively. I(k)[i,j] can be the pixel value at position (i,j) of the reference pixel in the reference block. i can indicate the position component of the pixel in the horizontal direction, and j can indicate the position component of the pixel in the vertical direction, where the units of i and j are fractional pixel units.

[0584] [Equation 42]

[0585] Ix (k) [i, j] = -9*(I (k) [i+2, j]-I (k) [i-2,j])+48*(I (k) [i+1, j]-I (k) [i-1, j])

[0586] Therefore, the video decoding device 100 can determine the gradient value Ix(k) of the reference pixel in the reference block in the horizontal direction by performing only two multiplication operations on each sample.

[0587] Furthermore, the video decoding device 100 can determine the gradient value Iy(k) of the reference pixel in the vertical direction by applying a vertical 5-tap gradient filter (where the filter coefficients are {9, -48, 0, 48, 9}; but not limited thereto) to the pixel value of the reference pixel in the reference block. In this respect, k can have a value of either 0 or 1, where 0 and 1 indicate reference frame 0 and 1, respectively. I(k)[i,j] can be the pixel value at position (i,j) of the reference pixel in the reference block. i can indicate the position component of the pixel in the horizontal direction, and j can indicate the position component of the pixel in the vertical direction, where the units of i and j are fractional pixel units.

[0588] [Equation 43]

[0589] Iy (k) [i, j] = --9*(I (k) [i, j+2]-I (k) [i, j-2])+48*(I (k) [i, j+1]-I (k) [i, j-1])

[0590] Therefore, the video decoding device 100 can determine the gradient value Iy(k) of the reference pixel in the vertical direction in the reference block by performing only two multiplication operations on each sample point.

[0591] When the video decoding device 100 performs pixel-unit motion compensation using block enlargement based on the window size and pixel-unit motion compensation by reducing the length of the gradient filter, the video decoding device 100 can perform as many memory access operations and multiplication operations as there are for each case according to Table 7 below. In this regard, assume the signal filter length T is 8, the gradient filter length T is 6, the reduced gradient filter length T is 5, the block size is N×N, and the window size per pixel is 2M+1, which is 5.

[0592] [Table 7]

[0593]

[0594] In other words, because the video decoding device 100 performs two multiplication operations for each 1D gradient filter according to equations 41 and 42, the gradient filter is applied to two reference blocks, and the gradient filter is applied to a reference block of size (N+4)×(N+4) based on the window size being enlarged, a total of 2*2*{(N+4)×(N+4)}*2 multiplication operations can be performed to determine the gradient values ​​of the reference pixels in the horizontal and vertical directions.

[0595] When the video decoding device 100 performs pixel-unit motion compensation using block enlargement based on the window size, and performs pixel-group-unit motion compensation by reducing the length of the gradient filter without block enlargement, the video decoding device 100 can perform as many memory access operations and multiplication operations as there are for each case according to Table 8 below. In this regard, it is assumed that the length T of the signal filter is 8, the length T of the gradient filter is 6, the length T of the reduced gradient filter is 5, the size of the pixel group is L×L, the size of the block is N×N, and the window size 2M+1 per pixel is 5.

[0596] [Table 8]

[0597]

[0598] Therefore, compared with the number of memory accesses, multiplications, and divisions when performing pixel-unit motion compensation using block expansion, the number of memory accesses, multiplications, and divisions when the video decoding device 100 performs pixel-group-unit motion compensation by reducing the length of the gradient filter without using block expansion can be significantly reduced.

[0599] In the following text, reference will be made to Figures 10 to 23 A method for determining data units that can be used when decoding an image using a video decoding apparatus 100 according to an embodiment is described. The operation of the video encoding apparatus 150 may be similar to or opposite to various embodiments of the operation of the video decoding apparatus 100 described below.

[0600] Figure 10 The illustration shows a process performed by a video decoding device 100 according to an embodiment to determine at least one coding unit by dividing the current coding unit.

[0601] According to an embodiment, the video decoding device 100 can determine the shape of the coding unit by using block shape information, and can determine the coding unit partitioning method by using partition shape information. That is, the coding unit partitioning method indicated by the partition shape information can be determined based on the block shape indicated by the block shape information used by the video decoding device 100.

[0602] According to an embodiment, the video decoding device 100 can use block shape information indicating that the current coding unit has a square shape. For example, the video decoding device 100 can determine whether to not divide the square coding unit, whether to divide the square coding unit vertically, whether to divide the square coding unit horizontally, or whether to divide the square coding unit into four coding units based on the division shape information. (See also...) Figure 10When the block shape information of the current coding unit 1000 indicates a square shape, the video decoding device 100 can determine that the coding unit 1010a with the same size as the current coding unit 1000 will not be divided based on the division shape information indicating that division will not be performed, or it can determine the coding units 1010b, 1010c or 1010d that are divided based on the division shape information indicating a predetermined division method.

[0603] Reference Figure 10 According to an embodiment, the video decoding device 100 can determine two coding units 1010b obtained by dividing the current coding unit 1000 in a vertical direction based on the division shape information indicating that division is performed in a vertical direction. The video decoding device 100 can determine two coding units 1010c obtained by dividing the current coding unit 1000 in a horizontal direction based on the division shape information indicating that division is performed in a horizontal direction. The video decoding device 100 can determine four coding units 1010d obtained by dividing the current coding unit 1000 in both vertical and horizontal directions based on the division shape information indicating that division is performed in both vertical and horizontal directions. However, the method for dividing square coding units is not limited to the above methods, and the division shape information can indicate various methods. A predetermined division method for dividing square coding units will be described in detail below with reference to various embodiments.

[0604] Figure 11 The illustration shows a process performed by a video decoding device 100 according to an embodiment to determine at least one coding unit by dividing a non-square coding unit.

[0605] According to an embodiment, the video decoding device 100 can use block shape information indicating that the current coding unit has a non-square shape. The video decoding device 100 can determine, based on the partitioning shape information, whether not to partition the non-square current coding unit or whether to partition the non-square current coding unit using a predetermined partitioning method. (Refer to...) Figure 11 When the block shape information of the current coding unit 1100 or 1150 indicates a non-square shape, the video decoding device 100 can determine, based on the partitioning shape information indicating that partitioning is not performed, that coding units 1110 or 1160 with the same size as the current coding unit 1100 or 1150 will not be partitioned, or determine that coding units 1120a and 1120b, 1130a to 1130c, 1170a and 1170b, or 1180a to 1180c, which are partitioned based on the partitioning shape information indicating a predetermined partitioning method, will be partitioned. The predetermined partitioning method for partitioning non-square coding units will be described in detail below with reference to various embodiments.

[0606] According to an embodiment, the video decoding device 100 can determine the partitioning method of the coding units by using partition shape information, and in this case, the partition shape information can indicate the number of one or more coding units generated by partitioning the coding units. (See also...) Figure 11 When the partition shape information indicates that the current coding unit 1100 or 1150 is divided into two coding units, the video decoding device 100 can determine the two coding units 1120a and 1120b, or 1170a and 1170b, included in the current coding unit 1100 or 1150 by partitioning the current coding unit 1100 or 1150 based on the partition shape information.

[0607] According to an embodiment, when the video decoding device 100 divides a non-square current coding unit 1100 or 1150 based on the division shape information, the position of the long side of the non-square current coding unit 1100 or 1150 can be taken into consideration. For example, considering the shape of the current coding unit 1100 or 1150, the video decoding device 100 can determine multiple coding units by dividing the long side of the current coding unit 1100 or 1150.

[0608] According to an embodiment, when the partitioning shape information indicates that the coding unit is divided into an odd number of blocks, the video decoding device 100 can determine an odd number of coding units included in the current coding unit 1100 or 1150. For example, when the partitioning shape information indicates that the current coding unit 1100 or 1150 is divided into three coding units, the video decoding device 100 can divide the current coding unit 1100 or 1150 into three coding units 1130a, 1130b, and 1130c, or 1180a, 1180b, and 1180c. According to an embodiment, the video decoding device 100 can determine an odd number of coding units included in the current coding unit 1100 or 1150, and not all determined coding units may have the same size. For example, a predetermined coding unit 1130b or 1180b among the determined odd-numbered coding units 1130a, 1130b, and 1130c, or 1180a, 1180b, and 1180c, may have a different size than the other coding units 1130a and 1130c or 1180a and 1180c. That is, the coding units determined by dividing the current coding unit 1100 or 1150 may have multiple sizes, and in some cases, all the odd-numbered coding units 1130a, 1130b, and 1130c, or 1180a, 1180b, and 1180c, may have different sizes.

[0609] According to an embodiment, when the shape information indicates that the coding unit is divided into an odd number of blocks, the video decoding device 100 can determine the odd number of coding units included in the current coding unit 1100 or 1150, and can set a predetermined limit on at least one coding unit among the odd number of coding units generated by dividing the current coding unit 1100 or 1150. (Refer to...) Figure 11 The video decoding device 100 may allow the decoding method of encoding unit 1130b or 1180b to differ from the decoding methods of other encoding units 1130a and 1130c or 1180a and 1180c, wherein encoding unit 1130b or 1180b is located at the center of the three encoding units 1130a, 1130b, and 1130c or 1180a, 1180b, and 1180c generated by dividing the current encoding unit 1100 or 1150. For example, unlike other encoding units 1130a and 1130c or 1180a and 1180c, the video decoding device 100 may restrict the encoding unit 1130b or 1180b at the center position to either not be divided or to be divided only a predetermined number of times.

[0610] Figure 12 The illustration shows a process performed by a video decoding device 100 according to an embodiment, which divides the coding unit based on at least one of block shape information and partition shape information.

[0611] According to an embodiment, the video decoding device 100 can determine whether to divide the square first coding unit 1200 into multiple coding units based on at least one of block shape information and partition shape information. According to an embodiment, when the partition shape information indicates that the first coding unit 1200 is partitioned along a horizontal direction, the video decoding device 100 can determine the second coding unit 1210 by partitioning the first coding unit 1200 along a horizontal direction. The terms first coding unit, second coding unit, and third coding unit used in the embodiment are terms used to understand the relationship before and after the partitioning of the coding units. For example, the second coding unit can be determined by partitioning the first coding unit, and the third coding unit can be determined by partitioning the second coding unit. It will be understood that the structures of the first coding unit, second coding unit, and third coding unit follow the above description.

[0612] According to an embodiment, the video decoding device 100 can determine whether to divide the determined second coding unit 1210 into multiple coding units based on at least one of the block shape information and the partition shape information. (Refer to...) Figure 12The video decoding device 100 may or may not divide the non-square second coding unit 1210, determined by dividing the first coding unit 1200, into one or more third coding units 1220a or third coding units 1220b, 1220c, and 1220d based on at least one of the block shape information and the division shape information. The video decoding device 100 can obtain at least one of the block shape information and the division shape information, and determine multiple second coding units (e.g., 1210) of different shapes by dividing the first coding unit 1200 based on the at least one of the obtained block shape information and the division shape information, and the second coding unit 1210 can be divided based on at least one of the block shape information and the division shape information using the division method of the first coding unit 1200. According to an embodiment, when the first coding unit 1200 is divided into a second coding unit 1210 based on at least one of the block shape information and the partition shape information of the first coding unit 1200, the second coding unit 1210 can also be divided into a third coding unit 1220a or third coding units 1220b, 1220c, and 1220d based on at least one of the block shape information and the partition shape information of the second coding unit 1210. That is, the coding units can be recursively divided based on at least one of the block shape information and the partition shape information of each coding unit. Therefore, square coding units can be determined by dividing non-square coding units, and non-square coding units can be determined by recursively dividing square coding units. (Refer to...) Figure 12 The predetermined coding units (e.g., coding units at the center position or square coding units) among the odd number of third coding units 1220b, 1220c, and 1220d, determined by dividing the non-square second coding unit 1210, can be recursively divided. According to an embodiment, the square third coding unit 1220c among the odd number of third coding units 1220b, 1220c, and 1220d can be divided into a plurality of fourth coding units along the horizontal direction. The non-square fourth coding unit 1240 among the plurality of fourth coding units can be divided into a plurality of coding units. For example, the non-square fourth coding unit 1240 can be divided into an odd number of coding units 1250a, 1250b, and 1250c.

[0613] The following describes methods for recursively dividing coding units in conjunction with various embodiments.

[0614] According to an embodiment, the video decoding device 100 can determine, based on at least one of block shape information and partition shape information, whether to divide the third coding unit 1220a or each of the third coding units 1220b, 1220c, and 1220d into multiple coding units or not to divide the second coding unit 1210. According to an embodiment, the video decoding device 100 can divide the non-square second coding unit 1210 into an odd number of third coding units 1220b, 1220c, and 1220d. The video decoding device 100 can set predetermined restrictions on predetermined third coding units among the odd number of third coding units 1220b, 1220c, and 1220d. For example, the video decoding device 100 can restrict the third coding unit 1220c at the center position among the odd number of third coding units 1220b, 1220c, and 1220d to not be divided or to be divided a set number of times. (See also...) Figure 12 The video decoding device 100 may restrict the third coding unit 1220c located at the center position among the odd number of third coding units 1220b, 1220c, and 1220d included in the non-square second coding unit 1210 to be no longer divided, to be divided using a predetermined division method (e.g., divided only into four coding units or divided using the division method of the second coding unit 1210), or to be divided only a predetermined number of times (e.g., divided only n times (where n>0)). However, the restriction on the third coding unit 1220c located at the center position is not limited to the above examples and may include various restrictions for decoding the third coding unit 1220c located at the center position differently from the other third coding units 1220b and 1220d.

[0615] According to an embodiment, the video decoding device 100 can obtain at least one piece of information, namely block shape information and partition shape information, from a predetermined position in the current encoding unit for dividing the current encoding unit.

[0616] Figure 13 A method for determining a predetermined coding unit from an odd number of coding units, performed by a video decoding device 100 according to an embodiment, is shown. (Refer to...) Figure 13 At least one piece of information, either block shape information or partition shape information, of the current encoding unit 1300 can be obtained from a sample at a predetermined position among a plurality of samples included in the current encoding unit 1300 (e.g., sample 1340 at the center position). However, the predetermined position in the current encoding unit 1300 from which at least one piece of information, either block shape information or partition shape information, can be obtained is not limited to... Figure 13The center position in the code can include various positions included in the current coding unit 1300 (e.g., top, bottom, left, right, upper left, lower left, upper right, and lower right positions). The video decoding device 100 can obtain at least one of the block shape information and the division shape information from the predetermined position, and determine whether to divide the current coding unit into coding units of various shapes and sizes or not to divide the current coding unit.

[0617] According to an embodiment, when the current coding unit is divided into a predetermined number of coding units, the video decoding device 100 may select one coding unit from the coding units. As will be described below in conjunction with various embodiments, various methods can be used to select one coding unit from a plurality of coding units.

[0618] According to an embodiment, the video decoding device 100 can divide the current encoding unit into multiple encoding units and determine the encoding unit at a predetermined position.

[0619] Figure 13 This illustrates a method performed by a video decoding device 100 according to an embodiment to determine a coding unit at a predetermined position from an odd number of coding units.

[0620] According to an embodiment, the video decoding device 100 can use information indicating the positions of an odd number of coding units to determine the coding unit at the center position among the odd number of coding units. (See also...) Figure 13 The video decoding device 100 can determine an odd number of coding units 1320a, 1320b, and 1320c by dividing the current coding unit 1300. The video decoding device 100 can determine the coding unit 1320b at the center position by using information about the positions of the odd number of coding units 1320a to 1320c. For example, the video decoding device 100 can determine the positions of coding units 1320a, 1320b, and 1320c based on information indicating the positions of predetermined samples included in coding units 1320a, 1320b, and 1320c to determine the coding unit 1320b at the center position. Specifically, the video decoding device 100 can determine the positions of coding units 1320a, 1320b, and 1320c based on information indicating the positions of samples 1330a, 1330b, and 1330c located to the upper left of coding units 1320a, 1320b, and 1320c to determine the coding unit 1320b at the center position.

[0621] According to an embodiment, the information indicating the positions of the upper left sample points 1330a, 1330b, and 1330c respectively included in encoding units 1320a, 1320b, and 1320c may include information about the positions or coordinates of the encoding units 1320a, 1320b, and 1320c in the frame. According to an embodiment, the information indicating the positions of the upper left sample points 1330a, 1330b, and 1330c respectively included in encoding units 1320a, 1320b, and 1320c may include information indicating the width or height of the encoding units 1320a, 1320b, and 1320c included in the current encoding unit 1300, and the width or height may correspond to information indicating the difference between the coordinates of the encoding units 1320a, 1320b, and 1320c in the frame. In other words, the video decoding device 100 can determine the encoding unit 1320b at the center position by directly using information about the position or coordinates of the encoding units 1320a, 1320b and 1320c in the frame, or by using information about the height or width corresponding to the difference between the encoding units and their coordinates.

[0622] According to an embodiment, the information indicating the position of the upper left sample 1330a of the upper encoding unit 1320a may include coordinates (xa, ya), the information indicating the position of the upper left sample 1330b of the middle encoding unit 1320b may include coordinates (xb, yb), and the information indicating the position of the upper left sample 1330c of the lower encoding unit 1320c may include coordinates (xc, yc). The video decoding device 100 can determine the middle encoding unit 1320b by using the coordinates of the upper left samples 1330a, 1330b, and 1330c respectively included in the encoding units 1320a, 1320b, and 1320c. For example, when the coordinates of the upper left sample points 1330a, 1330b, and 1330c are sorted in ascending or descending order, the coding unit 1320b at the center position, which includes the coordinates (xb, yb) of sample point 1330b, can be determined as the coding unit at the center position among the coding units 1320a, 1320b, and 1320c determined by dividing the current coding unit 1300. However, the coordinates indicating the positions of the upper left samples 1330a, 1330b, and 1330c may include coordinates indicating absolute positions within the frame, or coordinates (dxb, dyb) indicating the relative positions of the upper left sample 1330b of the middle encoding unit 1320b relative to the upper left sample 1330a of the upper encoding unit 1320a, and coordinates (dxc, dyc) indicating the relative positions of the upper left sample 1330c of the lower encoding unit 1320c relative to the upper left sample 1330a of the upper encoding unit 1320a. The method for determining the encoding unit at a predetermined position by using the coordinates of samples included in the encoding unit as information indicating the position of the samples is not limited to the methods described above, and may include various arithmetic methods capable of using the coordinates of the samples.

[0623] According to an embodiment, the video decoding device 100 can divide the current encoding unit 1300 into a plurality of encoding units 1320a, 1320b, and 1320c, and select one of the encoding units 1320a, 1320b, and 1320c based on a predetermined criterion. For example, the video decoding device 100 can select encoding unit 1320b, which has a different size from the other encoding units, from among the encoding units 1320a, 1320b, and 1320c.

[0624] According to an embodiment, the video decoding device 100 can determine the width or height of encoding units 1320a, 1320b, and 1320c by using coordinates (xa, ya) indicating the position of the upper left sample 1330a of the upper encoding unit 1320a, coordinates (xb, yb) indicating the position of the upper left sample 1330b of the middle encoding unit 1320b, and coordinates (xc, yc) indicating the position of the upper left sample 1330c of the lower encoding unit 1320c. The video decoding device 100 can also determine the individual dimensions of encoding units 1320a, 1320b, and 1320c by using coordinates (xa, ya), (xb, yb), and (xc, yc) indicating the positions of the encoding units 1320a, 1320b, and 1320c.

[0625] According to an embodiment, the video decoding device 100 can determine the width of the upper encoding unit 1320a as xb-xa and the height of the upper encoding unit 1320a as yb-ya. According to an embodiment, the video decoding device 100 can determine the width of the middle encoding unit 1320b as xc-xb and the height of the middle encoding unit 1320b as yc-yb. According to an embodiment, the video decoding device 100 can determine the width or height of the lower encoding unit 1320c by using the width or height of the current encoding unit 1300 or the width or height of the upper encoding unit 1320a and the middle encoding unit 1320b. The video decoding device 100 can determine encoding units with dimensions different from the dimensions of other encoding units based on the determined widths and heights of the encoding units 1320a to 1320c. (Refer to...) Figure 13 The video decoding device 100 can determine an intermediate encoding unit 1320b, which has a size different from that of the upper encoding unit 1320a and the lower encoding unit 1320c, as an encoding unit at a predetermined position. However, the method described above by the video decoding device 100 for determining an encoding unit with a size different from that of the other encoding units corresponds only to an example of determining an encoding unit at a predetermined position by using the size of the encoding unit determined based on the coordinates of the sample points. Therefore, various methods for determining an encoding unit at a predetermined position by comparing the size of the encoding unit determined based on the coordinates of the predetermined sample points can be used.

[0626] However, the position of the sample points considered to be used to determine the position of the coding unit is not limited to the aforementioned upper left position, and information about any position of the sample points included in the coding unit can be used.

[0627] According to an embodiment, taking into account the shape of the current coding unit, the video decoding device 100 can select a coding unit at a predetermined position from an odd number of coding units determined by dividing the current coding unit. For example, when the current coding unit has a non-square shape with a width longer than its height, the video decoding device 100 can determine a coding unit at a predetermined position in the horizontal direction. That is, the video decoding device 100 can determine one of the coding units at different positions in the horizontal direction and set a restriction on that coding unit. When the current coding unit has a non-square shape with a height longer than its width, the video decoding device 100 can determine a coding unit at a predetermined position in the vertical direction. That is, the video decoding device 100 can determine one of the coding units at different positions in the vertical direction and set a restriction on that coding unit.

[0628] According to an embodiment, the video decoding device 100 can use information indicating the positions of an even number of coding units to determine a coding unit at a predetermined position among the even number of coding units. The video decoding device 100 can determine the even number of coding units by dividing the current coding unit, and determine the coding unit at the predetermined position by using information about the positions of the even number of coding units. The associated operations can correspond to those already combined above. Figure 13 The operation of determining a coding unit at a predetermined position (e.g., the center position) among an odd number of coding units is described in detail, therefore, its detailed description is not provided here.

[0629] According to an embodiment, when a non-square current coding unit is divided into multiple coding units, predetermined information about the coding unit at a predetermined position can be used in the division operation to determine the coding unit at the predetermined position among the multiple coding units. For example, the video decoding device 100 can use at least one piece of information, namely block shape information and division shape information, stored in the sample points included in the coding unit at the center position, in the division operation to determine the coding unit at the center position among the multiple coding units determined by dividing the current coding unit.

[0630] Reference Figure 13The video decoding device 100 can divide the current coding unit 1300 into multiple coding units 1320a, 1320b, and 1320c based on at least one of the block shape information and the partition shape information, and determine the coding unit 1320b at the center position among the multiple coding units 1320a, 1320b, and 1320c. Furthermore, considering the position of obtaining at least one of the block shape information and the partition shape information, the video decoding device 100 can determine the coding unit 1320b at the center position. That is, at least one of the block shape information and the partition shape information of the current coding unit 1300 can be obtained from the sample point 1340 at the center position of the current coding unit 1300, and when the current coding unit 1300 is divided into multiple coding units 1320a, 1320b, and 1320c based on at least one of the block shape information and the partition shape information, the coding unit 1320b including the sample point 1340 can be determined as the coding unit at the center position. However, the information used to determine the coding unit at the center location is not limited to at least one of the block shape information and the partition shape information, and various types of information can be used to determine the coding unit at the center location.

[0631] According to an embodiment, predetermined information for identifying the coding unit at a predetermined location can be obtained from predetermined samples included in the coding unit to be determined. (Refer to...) Figure 13 The video decoding device 100 can use at least one piece of information, either block shape information or partition shape information, obtained from samples at predetermined positions in the current encoding unit 1300 (e.g., samples at the center of the current encoding unit 1300), to determine the encoding unit at a predetermined position (e.g., the encoding unit at the center of the multiple divided encoding units) among the plurality of encoding units 1320a, 1320b, and 1320c determined by dividing the current encoding unit 1300. That is, the video decoding device 100 can determine the samples at predetermined positions by considering the block shape of the current encoding unit 1300, determine the encoding unit 1320b among the plurality of encoding units 1320a, 1320b, and 1320c determined by dividing the current encoding unit 1300 that includes samples from which predetermined information (e.g., at least one piece of information, either block shape information or partition shape information) can be obtained, and set predetermined limitations on the encoding unit 1320b. (Refer to...) Figure 13 According to an embodiment, during the decoding operation, the video decoding device 100 may determine a sample 1340 at the center position of the current encoding unit 1300 as a sample from which predetermined information can be obtained, and set a predetermined restriction on the encoding unit 1320b including the sample 1340. However, the position of the sample from which the predetermined information can be obtained is not limited to the above-mentioned position, and may include any position of the sample included in the encoding unit 1320b to be determined for restriction.

[0632] According to an embodiment, the location of a sample from which predetermined information can be obtained can be determined based on the shape of the current encoding unit 1300. According to an embodiment, block shape information can indicate whether the current encoding unit has a square shape or a non-square shape, and the location of a sample from which predetermined information can be obtained can be determined based on this shape. For example, the video decoding device 100 can determine a sample located on a boundary that divides at least one of the width and height of the current encoding unit as a sample from which predetermined information can be obtained by using at least one of information about the width and information about the height of the current encoding unit. As another example, when the block shape information of the current encoding unit indicates a non-square shape, the video decoding device 100 can determine one of the samples adjacent to the boundary that divides the long side of the current encoding unit in half as a sample from which predetermined information can be obtained.

[0633] According to an embodiment, when a current coding unit is divided into multiple coding units, the video decoding device 100 can use at least one of block shape information and partition shape information to determine the coding unit at a predetermined position among the multiple coding units. According to an embodiment, the video decoding device 100 can obtain at least one of block shape information and partition shape information from samples at predetermined positions in the coding unit, and divide the multiple coding units by using at least one of partition shape information and block shape information obtained from samples at predetermined positions in each of the multiple coding units generated by dividing the current coding unit. That is, the coding unit can be recursively divided based on at least one of block shape information and partition shape information obtained from samples at predetermined positions in each coding unit. The above has already been combined with... Figure 12 The operation of recursively dividing the coding unit is described, therefore, its detailed description will not be provided here.

[0634] According to an embodiment, the video decoding device 100 can determine one or more coding units by dividing the current coding unit, and determine the order in which the one or more coding units are decoded based on a predetermined block (e.g., the current coding unit).

[0635] Figure 14 The following illustration shows the order in which the video decoding device 100 processes multiple coding units when it determines multiple coding units by dividing the current coding unit, according to an embodiment.

[0636] According to an embodiment, the video decoding device 100 may determine the second coding units 1410a and 1410b by dividing the first coding unit 1400 along the vertical direction based on block shape information and division shape information, determine the second coding units 1430a and 1430b by dividing the first coding unit 1400 along the horizontal direction, or determine the second coding units 1450a to 1450d by dividing the first coding unit 1400 along both the vertical and horizontal directions.

[0637] Reference Figure 14 The video decoding device 100 can determine to process second coding units 1410a and 1410b, determined by dividing the first coding unit 1400 along the vertical direction, in a horizontal direction sequence 1410c. The video decoding device 100 can determine to process second coding units 1430a and 1430b, determined by dividing the first coding unit 1400 along the horizontal direction, in a vertical direction sequence 1430c. The video decoding device 100 can determine to process second coding units 1450a to 1450d, determined by dividing the first coding unit 1400 along both the vertical and horizontal directions, in a predetermined order (e.g., raster scan order or Z-scan order 1450e) for processing coding units in one row and then processing coding units in the next row.

[0638] According to an embodiment, the video decoding device 100 can recursively divide the encoding units. (Refer to...) Figure 14 The video decoding device 100 can determine multiple coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d by dividing the first coding unit 1400, and recursively divide each of the determined multiple coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d. The method for dividing the multiple coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d can correspond to the method for dividing the first coding unit 1400. Thus, each of the multiple coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b, 1450c, and 1450d can be independently divided into multiple coding units. (See reference...) Figure 14 The video decoding device 100 can determine the second coding units 1410a and 1410b by dividing the first coding unit 1400 along the vertical direction, and can determine whether to divide each of the second coding units 1410a and 1410b independently or not to divide each of the second coding units 1410a and 1410b.

[0639] According to an embodiment, the video decoding device 100 can determine the third coding units 1420a and 1420b by dividing the second coding unit 1410a on the left side along the horizontal direction, and may not divide the second coding unit 1410b on the right side.

[0640] According to an embodiment, the processing order of coding units can be determined based on the operation of dividing coding units. In other words, the processing order of the divided coding units can be determined based on the processing order of the coding units immediately preceding the division. The video decoding device 100 can independently determine the processing order of the third coding units 1420a and 1420b, which are determined by dividing the second coding unit 1410a on the left, with respect to the second coding unit 1410b on the right. Because the third coding units 1420a and 1420b are determined by dividing the second coding unit 1410a on the left in a horizontal direction, the third coding units 1420a and 1420b can be processed in a vertical order 1420c. Because the second coding unit 1410a on the left and the second coding unit 1410b on the right are processed in a horizontal order 1410c, the second coding unit 1410b on the right can be processed after the third coding units 1420a and 1420b, which are included in the second coding unit 1410a on the left, are processed in a vertical order 1420c. The operation of determining the processing order of coding units based on the coding units before they are divided is not limited to the examples above, and various methods can be used to independently process coding units that have been divided and determined to be of various shapes in a predetermined order.

[0641] Figure 15 The illustration shows a process performed by a video decoding device 100 according to an embodiment, in which the current encoding unit is determined to be divided into an odd number of encoding units when the encoding units cannot be processed in a predetermined order.

[0642] According to an embodiment, the video decoding device 100 can determine whether the current coding unit is divided into an odd number of coding units based on the obtained block shape information and partition shape information. (See also...) Figure 15 The square first coding unit 1500 can be divided into non-square second coding units 1510a and 1510b, and the second coding units 1510a and 1510b can be independently divided into third coding units 1520a and 1520b and third coding units 1520c to 1520e. According to an embodiment, the video decoding device 100 can determine a plurality of third coding units 1520a and 1520b by dividing the second coding unit 1510a on the left side along the horizontal direction, and dividing the second coding unit 1510b on the right side into an odd number of third coding units 1520c to 1520e.

[0643] According to an embodiment, the video decoding device 100 can determine whether to divide any coding unit into an odd number of coding units by judging whether the third coding units 1520a and 1520b and the third coding units 1520c to 1520e can be processed in a predetermined order. (See also...) Figure 15 The video decoding device 100 can determine the third coding units 1520a and 1520b, and the third coding units 1520c to 1520e, by recursively dividing the first coding unit 1500. The video decoding device 100 can determine whether any one of the first coding unit 1500, the second coding units 1510a and 1510b, the third coding units 1520a and 1520b, and 1520c, 1520d, and 1520e is divided into an odd number of coding units based on at least one of the block shape information and the division shape information. For example, the second coding unit 1510b on the right can be divided into an odd number of third coding units 1520c, 1520d, and 1520e. The processing order of the plurality of coding units included in the first coding unit 1500 may be a predetermined order (e.g., zigzag scanning order 1530), and the video decoding device 100 may determine whether the third coding units 1520c, 1520d and 1520e, which are determined by dividing the second coding unit 1510b on the right into an odd number of coding units, meet the condition of being processed in a predetermined order.

[0644] According to an embodiment, the video decoding device 100 can determine whether the third encoding units 1520a and 1520b, and the third encoding units 1520c, 1520d, and 1520e, included in the first encoding unit 1500, satisfy a condition for processing in a predetermined order. This condition relates to whether at least one of the width and height of the second encoding units 1510a and 1510b is divided in half along the boundary of the third encoding units 1520a and 1520b, and the third encoding units 1520c, 1520d, and 1520e. For example, the third encoding units 1520a and 1520b, determined by dividing the height of the non-square left-side second encoding unit 1510a in half, satisfy the condition. However, since the boundaries of the third coding units 1520c, 1520d, and 1520e, determined by dividing the second coding unit 1510b on the right into three coding units, do not divide the width or height of the second coding unit 1510b on the right into two halves, it can be determined that the third coding units 1520c, 1520d, and 1520e do not satisfy the aforementioned condition. When the aforementioned condition is not satisfied as described above, the video decoding device 100 can determine the discontinuity of the scanning order and, based on the determination result, determine that the second coding unit 1510b on the right is divided into an odd number of coding units. According to an embodiment, when a coding unit is divided into an odd number of coding units, the video decoding device 100 can set a predetermined restriction on the coding units at predetermined positions among the divided coding units. The restriction or predetermined position has been described above in conjunction with various embodiments, and therefore its detailed description will not be provided here.

[0645] Figure 16 The illustration shows a process performed by a video decoding device 100 according to an embodiment, involving the division of a first coding unit 1600 to determine at least one coding unit. According to the embodiment, the video decoding device 100 may divide the first coding unit 1600 based on at least one of block shape information and division shape information obtained by the acquirer 105. A square first coding unit 1600 may be divided into four square coding units, or it may be divided into multiple non-square coding units. For example, refer to… Figure 16When the block shape information indicates that the first coding unit 1600 has a square shape and the division shape information indicates that the first coding unit 1600 is divided into non-square coding units, the video decoding device 100 can divide the first coding unit 1600 into a plurality of non-square coding units. Specifically, when the division shape information indicates that the first coding unit 1600 is divided along a horizontal or vertical direction to determine an odd number of coding units, the video decoding device 100 can divide the square first coding unit 1600 into an odd number of coding units, for example, second coding units 1610a, 1610b, and 1610c determined by dividing the square first coding unit 1600 along a vertical direction, or second coding units 1620a, 1620b, and 1620c determined by dividing the square first coding unit 1600 along a horizontal direction.

[0646] According to an embodiment, the video decoding device 100 can determine whether the second encoding units 1610a, 1610b, 1610c, 1620a, 1620b, and 1620c included in the first encoding unit 1600 meet the condition of being processed in a predetermined order, and the condition relates to whether at least one of the width and height of the first encoding unit 1600 is divided into two halves along the boundary of the second encoding units 1610a, 1610b, 1610c, 1620a, 1620b, and 1620c. (Refer to...) Figure 16 Because the boundaries of the second coding units 1610a, 1610b, and 1610c, determined by dividing the square first coding unit 1600 vertically, do not divide the width of the first coding unit 1600 in half, it can be determined that the first coding unit 1600 does not meet the conditions for processing in a predetermined order. Furthermore, because the boundaries of the second coding units 1620a, 1620b, and 1620c, determined by dividing the square first coding unit 1600 horizontally, do not divide the height of the first coding unit 1600 in half, it can be determined that the first coding unit 1600 does not meet the conditions sufficient for processing in a predetermined order. When the conditions are not met as described above, the video decoding device 100 can determine a discontinuity in the scanning order and, based on the determination result, determine that the first coding unit 1600 is divided into an odd number of coding units. According to an embodiment, when the coding unit is divided into an odd number of coding units, the video decoding device 100 can set a predetermined restriction on the coding units at predetermined positions among the divided coding units. The limitations or predetermined locations have been described above in conjunction with various embodiments, so their detailed description will not be provided here.

[0647] According to an embodiment, the video decoding device 100 can determine coding units of various shapes by dividing the first coding unit.

[0648] Reference Figure 16 The video decoding device 100 can divide the square first coding unit 1600 or the non-square first coding unit 1630 or 1650 into coding units of various shapes.

[0649] Figure 17 The image illustrates a shape limitation that the second coding unit can be divided by the video decoding device 100 when a second coding unit having a non-square shape, determined by dividing the first coding unit 1700, satisfies predetermined conditions, according to an embodiment.

[0650] According to an embodiment, the video decoding device 100 can determine, based on at least one of the block shape information and partition shape information obtained by the acquirer 105, whether to divide the square first coding unit 1700 into non-square second coding units 1710a, 1710b, 1720a, and 1720b. The second coding units 1710a, 1710b, 1720a, and 1720b can be divided independently. Thus, the video decoding device 100 can determine, based on at least one of the block shape information and partition shape information for each of the second coding units 1710a, 1710b, 1720a, and 1720b, whether to divide each of the second coding units 1710a, 1710b, 1720a, and 1720b into multiple coding units. According to an embodiment, the video decoding device 100 can determine third coding units 1712a and 1712b by dividing the left side of the second coding unit 1710a, which is a non-square determined by dividing the first coding unit 1700 along the vertical direction, along the horizontal direction. However, when the left side of the second coding unit 1710a is divided along the horizontal direction, the video decoding device 100 can restrict the right side of the second coding unit 1710b to be divided without following the horizontal direction in which the left side of the second coding unit 1710a is divided. When the third coding units 1714a and 1714b are determined by dividing the right side of the second coding unit 1710b along the horizontal direction, the third coding units 1712a, 1712b, 1714a, and 1714b can be determined because the left side of the second coding unit 1710a and the right side of the second coding unit 1710b are divided independently along the horizontal direction. However, the same applies to the case where the video decoding device 100 divides the first coding unit 1700 into four squares, namely, second coding units 1730a, 1730b, 1730c and 1730d, based on at least one of the block shape information and the partition shape information, but this may be inefficient in terms of image decoding.

[0651] According to an embodiment, the video decoding device 100 can determine third coding units 1722a, 1722b, 1724a, and 1724b by dividing a non-square second coding unit 1720a or 1720b, which is determined by dividing the first coding unit 1700 horizontally, along the vertical direction. However, when the second coding unit (e.g., the upper second coding unit 1720a) is divided vertically, for the reasons described above, the video decoding device 100 can restrict another second coding unit (e.g., the lower second coding unit 1720b) ​​to be divided not along the vertical direction in which the upper second coding unit 1720a is divided.

[0652] Figure 18 The illustration shows a process performed by a video decoding device 100 according to an embodiment, in which a square coding unit is divided when the division shape information indicates that the square coding unit will not be divided into four square coding units.

[0653] According to an embodiment, the video decoding device 100 can determine second coding units 1810a, 1810b, 1820a, 1820b, etc., by dividing the first coding unit 1800 based on at least one of block shape information and partition shape information. The partition shape information may include information about various methods of dividing the coding unit, but may not include information about coding units used to divide the coding unit into four squares. Based on this partition shape information, the video decoding device 100 may not divide the square first coding unit 1800 into four square second coding units 1830a, 1830b, 1830c, and 1830d. ​​The video decoding device 100 can also determine non-square second coding units 1810a, 1810b, 1820a, 1820b, etc., based on the partition shape information.

[0654] According to an embodiment, the video decoding device 100 can independently divide non-square second coding units 1810a, 1810b, 1820a, 1820b, etc. Each of the second coding units 1810a, 1810b, 1820a, 1820b, etc., can be recursively divided in a predetermined order, and this division method can correspond to a method of dividing the first coding unit 1800 based on at least one of block shape information and division shape information.

[0655] For example, the video decoding device 100 can determine the square third coding units 1812a and 1812b by dividing the second coding unit 1810a on the left side horizontally, and can determine the square third coding units 1814a and 1814b by dividing the second coding unit 1810b on the right side horizontally. Furthermore, the video decoding device 100 can determine the square third coding units 1816a, 1816b, 1816c, and 1816d by dividing both the second coding unit 1810a on the left side and the second coding unit 1810b on the right side horizontally. In this case, coding units with the same shape as the four squares of second coding units 1830a, 1830b, 1830c, and 1830d divided from the first coding unit 1800 can be determined.

[0656] As another example, the video decoding device 100 can determine square third coding units 1822a and 1822b by dividing the upper second coding unit 1820a vertically, and can determine square third coding units 1824a and 1824b by dividing the lower second coding unit 1820b vertically. Furthermore, the video decoding device 100 can determine square third coding units 1826a, 1826b, 1826c, and 1826d by dividing both the upper and lower second coding units 1820a and 1820b vertically. In this case, coding units with the same shape as the four squares of second coding units 1830a, 1830b, 1830c, and 1830d divided from the first coding unit 1800 can be determined.

[0657] Figure 19 It is shown that the processing order among multiple coding units according to the embodiment can be changed based on the processing of dividing the coding units.

[0658] According to an embodiment, the video decoding device 100 can divide the first coding unit 1900 based on block shape information and partition shape information. When the block shape information indicates a square shape and the partition shape information indicates that the first coding unit 1900 is divided along at least one direction, either horizontal or vertical, the video decoding device 100 can determine second coding units 1910a, 1910b, 1920a, and 1920b by dividing the first coding unit 1900. (Refer to...) Figure 19The non-square second coding units 1910a, 1910b, 1920a, and 1920b, determined by dividing the first coding unit 1900 only along the horizontal or vertical direction, can be independently divided based on the block shape information and partition shape information of each coding unit. For example, the video decoding device 100 can determine the third coding units 1916a, 1916b, 1916c, and 1916d by dividing the second coding units 1910a and 1910b generated by dividing the first coding unit 1900 along the vertical direction along the horizontal direction, and determine the third coding units 1926a, 1926b, 1926c, and 1926d by dividing the second coding units 1920a and 1920b generated by dividing the first coding unit 1900 along the horizontal direction along the vertical direction. The above has already been combined with... Figure 17 The operation of dividing the second coding units 1910a, 1910b, 1920a and 1920b is described, therefore, its detailed description will not be provided here.

[0659] According to an embodiment, the video decoding device 100 can process the encoding units in a predetermined order. (The above has already been combined with...) Figure 14 The operation of processing encoded units in a predetermined order is described; therefore, its detailed description will not be provided here. (See reference...) Figure 19 The video decoding device 100 can determine the third encoding units 1916a, 1916b, 1916c, and 1916d, and the third encoding units 1926a, 1926b, 1926c, and 1926d of the four squares by dividing the first encoding unit 1900 of the square. According to an embodiment, the video decoding device 100 can determine the processing order of the third encoding units 1916a, 1916b, 1916c, and 1916d, and the third encoding units 1926a, 1926b, 1926c, and 1926d, based on the division method of the first encoding unit 1900.

[0660] According to an embodiment, the video decoding device 100 can determine third coding units 1916a, 1916b, 1916c, and 1916d by dividing second coding units 1910a and 1910b, which are generated by dividing the first coding unit 1900 in the vertical direction, in the horizontal direction. The third coding units 1916a, 1916b, 1916c, and 1916d can be processed in the following processing order 1917: initially, the third coding units 1916a and 1916c included in the second coding unit 1910a on the left side are processed in the vertical direction, and then the third coding units 1916b and 1916d included in the second coding unit 1910b on the right side are processed in the vertical direction.

[0661] According to an embodiment, the video decoding device 100 can determine third coding units 1926a, 1926b, 1926c, and 1926d by dividing second coding units 1920a and 1920b, which are generated by dividing the first coding unit 1900 in the horizontal direction, in the vertical direction. The third coding units 1926a, 1926b, 1926c, and 1926d can be processed in the following processing order 1927: initially processing the third coding units 1926a and 1926b included in the upper second coding unit 1920a in the horizontal direction, and then processing the third coding units 1926c and 1926d included in the lower second coding unit 1920b in the horizontal direction.

[0662] Reference Figure 19 The third coding units 1916a, 1916b, 1916c and 1916d of the square, as well as the third coding units 1926a, 1926b, 1926c and 1926d, can be determined by dividing the second coding units 1910a, 1910b, 1920a and 1920b respectively. Although the second coding units 1910a and 1910b are determined by dividing the first coding unit 1900 vertically, unlike the second coding units 1920a and 1920b which are determined by dividing the first coding unit 1900 horizontally, the third coding units 1916a, 1916b, 1916c, and 1916d, as well as the third coding units 1926a, 1926b, 1926c, and 1926d derived from the second coding units 1910a and 1910 and the second coding units 1920a and 1920b, ultimately appear as coding units of the same shape derived from the first coding unit 1900. Thus, by recursively dividing the coding units in different ways based on at least one of the block shape information and the division shape information, the video decoding device 100 can process multiple coding units in different orders, even when multiple coding units are ultimately determined to have the same shape.

[0663] Figure 20 The illustration shows a process for determining the depth of a coding unit as the shape and size of the coding unit change when recursively dividing the coding unit to determine multiple coding units, according to an embodiment.

[0664] According to an embodiment, the video decoding device 100 can determine the depth of the coding unit based on a predetermined criterion. For example, the predetermined criterion could be the length of the long side of the coding unit. When the length of the long side of the coding unit before it was divided is 2n (n>0) times the length of the long side of the current coding unit, the video decoding device 100 can determine that the depth of the current coding unit increases by n from the depth of the coding unit before it was divided. In the following description, the coding unit with the increased depth is referred to as a deeper coding unit.

[0665] Reference Figure 20 According to an embodiment, the video decoding device 100 can determine a deeper second coding unit 2002 and a third coding unit 2004 by dividing a square first coding unit 2000 based on block shape information indicating the shape of the square (e.g., the block shape information can be represented as "0: SQUARE"). Assuming the size of the square first coding unit 2000 is 2N×2N, then by dividing the width and height of the first coding unit 2000 into 1 / 2... 1 The determined second coding unit 2002 may have a size of N×N. Furthermore, the determined third coding unit 2004, by reducing the width and height of the second coding unit 2002 to half, may have a size of N / 2×N / 2. In this case, the width and height of the third coding unit 2004 are half the width and height of the first coding unit 2000. 2 The depth of the first coding unit 2000 is D, and the depth of the second coding unit 2002 can be D+1, and the depth of the third coding unit 2004 can be D+2, wherein the width and height of the second coding unit 2002 are half the width and height of the first coding unit 2000. 1 The width and height of the third coding unit 2004 are half the width and height of the first coding unit 2000. 2 times.

[0666] According to an embodiment, the video decoding device 100 can determine a deeper second coding unit 2012 or 2022 and a third coding unit 2014 or 2024 by dividing the non-square first coding unit 2010 or 2020 based on block shape information indicating a non-square shape (for example, the block shape information can be represented as "1: NS_VER" indicating a non-square shape with a height longer than its width, or as "2: NS_HOR" indicating a non-square shape with a width longer than its height).

[0667] The video decoding device 100 can determine the second encoding unit 2002, 2012, or 2022 by dividing at least one of the width and height of the first encoding unit 2010, which has a size of N×2N. That is, the video decoding device 100 can determine the second encoding unit 2002, which has a size of N×N, or the second encoding unit 2022, which has a size of N×N / 2, by dividing the first encoding unit 2010 along the horizontal direction, or by dividing the first encoding unit 2010 along both the horizontal and vertical directions.

[0668] According to an embodiment, the video decoding device 100 can determine a second encoding unit 2002, 2012, or 2022 by dividing at least one of the width and height of a first encoding unit 2020 with a size of 2N×N. That is, the video decoding device 100 can determine a second encoding unit 2002 with a size of N×N or a second encoding unit 2012 with a size of N / 2×N by dividing the first encoding unit 2020 along the vertical direction, or it can determine a second encoding unit 2022 with a size of N×N / 2 by dividing the first encoding unit 2020 along both the horizontal and vertical directions.

[0669] According to an embodiment, the video decoding device 100 can determine a third encoding unit 2004, 2014, or 2024 by dividing at least one of the width and height of a second encoding unit 2002 with dimensions N×N. That is, the video decoding device 100 can determine a third encoding unit 2004 with dimensions N / 2×N / 2 by dividing the second encoding unit 2002 along the vertical and horizontal directions. 2 The third encoding unit 2014 of size N / 2 × N / 2, or size N / 2 × N / 2 2 The third coding unit 2024.

[0670] According to an embodiment, the video decoding device 100 can determine a third encoding unit 2004, 2014, or 2024 by dividing at least one of the width and height of a second encoding unit 2012 with dimensions of N / 2 × N. That is, the video decoding device 100 can determine a third encoding unit 2004 with dimensions of N / 2 × N / 2 by dividing the second encoding unit 2012 horizontally. 2 The third coding unit 2024, or the second coding unit 2012, is divided into units of size N / 2 along the vertical and horizontal directions. 2 The third coding unit 2014 is ×N / 2.

[0671] According to an embodiment, the video decoding device 100 can determine a third encoding unit 2004, 2014, or 2024 by dividing at least one of the width and height of a second encoding unit 2022 with a size of N×N / 2. That is, the video decoding device 100 can determine a third encoding unit 2004 with a size of N / 2×N / 2 by dividing the second encoding unit 2022 along the vertical direction. 2 The third coding unit 2014 of size N / 2 can be determined either by dividing the second coding unit 2022 along the vertical and horizontal directions. 2 The third coding unit 2024.

[0672] According to an embodiment, the video decoding device 100 can divide the square encoding units 2000, 2002, or 2004 along a horizontal or vertical direction. For example, the video decoding device 100 can determine a first encoding unit 2010 of size N×2N by dividing the first encoding unit 2000 of size 2N×2N along a vertical direction, or determine a first encoding unit 2020 of size 2N×N by dividing the first encoding unit 2000 along a horizontal direction. According to an embodiment, when the depth is determined based on the length of the longest side of the encoding unit, the depth of the encoding unit determined by dividing the first encoding unit 2000, 2002, or 2004 of size 2N×2N along a horizontal or vertical direction can be the same as the depth of the first encoding unit 2000, 2002, or 2004.

[0673] According to an embodiment, the width and height of the third encoding unit 2014 or 2024 can be half the width and height of the first encoding unit 2010 or 2020. 2 The depth of the first coding unit 2010 or 2020 is D. When the depth of the first coding unit 2010 or 2020 is D, the depth of the second coding unit 2012 or 2022 can be D+1, and the depth of the third coding unit 2014 or 2024 can be D+2. The width and height of the second coding unit 2012 or 2022 are half the width and height of the first coding unit 2010 or 2020, and the width and height of the third coding unit 2014 or 2024 are half the width and height of the first coding unit 2010 or 2020. 2 times.

[0674] Figure 21 The diagram illustrates a depth that can be determined based on the shape and size of the coding unit, according to an embodiment, and a partial index (PID) used to distinguish the coding unit.

[0675] According to an embodiment, the video decoding device 100 can determine second coding units of various shapes by dividing a square first coding unit 2100. (See also...) Figure 21 The video decoding device 100 can determine the second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d by dividing the first coding unit 2100 along at least one of the vertical and horizontal directions based on the division shape information. In other words, the video decoding device 100 can determine the second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d based on the division shape information of the first coding unit 2100.

[0676] According to an embodiment, the depths of the second coding units 2102a, 2102b, 2104a, 2104b, 2106a, 2106b, 2106c, and 2106d, determined based on the division shape information of the square first coding unit 2100, can be determined based on the length of their longer sides. For example, since the side length of the square first coding unit 2100 is equal to the length of the longer side of the non-square second coding units 2102a, 2102b, 2104a, and 2104b, the first coding unit 2100 and the non-square second coding units 2102a, 2102b, 2104a, and 2104b can have the same depth, for example, D. However, when the video decoding device 100 divides the first coding unit 2100 into four square second coding units 2106a, 2106b, 2106c and 2106d based on the division shape information, since the side length of the square second coding units 2106a, 2106b, 2106c and 2106d is 1 / 2 times the side length of the first coding unit 2100, the depth of the second coding units 2106a, 2106b, 2106c and 2106d can be D+1 deeper than the depth D of the first coding unit 2100.

[0677] According to an embodiment, the video decoding device 100 can determine a plurality of second coding units 2112a, 2112b, 2114a, 2114b, and 2114c by dividing a first coding unit 2110, whose height is longer than its width, along the horizontal direction based on division shape information. According to an embodiment, the video decoding device 100 can determine a plurality of second coding units 2122a, 2122b, 2124a, 2124b, and 2124c by dividing a first coding unit 2120, whose width is longer than its height, along the vertical direction based on division shape information.

[0678] According to an embodiment, the depths of the second coding units 2112a, 2112b, 2114a, 2114b, 2114c, 2122a, 2122b, 2124a, 2124b, and 2124c, determined based on the division shape information of the non-square first coding unit 2110 or 2120, can be determined based on the length of their longer sides. For example, since the side length of the square second coding units 2112a and 2112b is half the length of the longer side of the non-square first coding unit 2110, which has a height longer than its width, the depths of the square second coding units 2112a and 2112b are D+1 deeper than the depth D of the non-square first coding unit 2110.

[0679] Furthermore, the video decoding device 100 can divide the non-square first coding unit 2110 into an odd number of second coding units 2114a, 2114b, and 2114c based on the division shape information. The odd number of second coding units 2114a, 2114b, and 2114c may include the non-square second coding units 2114a and 2114c as well as the square second coding unit 2114b. In this case, since the length of the longer side of the non-square second coding units 2114a and 2114c and the side length of the square second coding unit 2114b are 1 / 2 times the length of the longer side of the first coding unit 2110, the depth of the second coding units 2114a, 2114b, and 2114c can be D+1, which is 1 deeper than the depth D of the non-square first coding unit 2110. The video decoding device 100 can determine the depth of the coding unit divided from the first coding unit 2120, which has a non-square shape with a width longer than its height, by using the method described above for determining the depth of the coding unit divided from the first coding unit 2110.

[0680] According to an embodiment, when an odd number of the divided coding units do not have equal sizes, the video decoding device 100 can determine a PID for identifying the divided coding units based on the size ratio between the coding units. (See also...) Figure 21The central coding unit 2114b among the odd-numbered divided coding units 2114a, 2114b, and 2114c may have a width equal to that of the other coding units 2114a and 2114c, and a height twice that of the other coding units 2114a and 2114c. That is, in this case, the central coding unit 2114b may include two other coding units 2114a or 2114c. Therefore, when the PID of the central coding unit 2114b is 1 based on the scan order, the PID of the coding unit 2114c located next to 2114b may be increased by 2, and thus may be 3. That is, there may be discontinuities in the PID values. According to an embodiment, the video decoding device 100 may determine whether the odd-numbered divided coding units do not have equal sizes based on whether there are discontinuities in the PIDs used to identify the divided coding unit...

Claims

1. A video decoding device, comprising: At least one processor is configured as follows: Obtain a first extended reference block in a first reference frame and a second extended reference block in a second reference frame, wherein the first extended reference block includes a first reference block and a first portion extended from the first reference block, and the second extended reference block includes a second reference block and a second portion extended from the second reference block. The displacement vector of a pixel group in the current block is determined by using the gradient values ​​of at least one reference pixel in a first extended reference block and at least one reference pixel in a second extended reference block, wherein the pixel group includes at least one pixel adjacent to the inner side of the boundary of the current block, the first portion of the first extended reference block is used to calculate the gradient value of at least one reference pixel in the first reference block, and the second portion of the second extended reference block is used to calculate the gradient value of at least one reference pixel in the second reference block. The predicted pixel value of the current block is obtained by performing optical flow-based compensation on the current block using the gradient values ​​of at least one reference pixel in the first reference block, the gradient values ​​of at least one reference pixel in the second reference block, and the displacement vector of the pixel group. Obtain the residual pixel value of the current block. The current block is reconstructed based on the predicted pixel values ​​and the residual pixel values. The pixel group comprises K×K blocks, where K is an integer greater than 1.

2. A video encoding device, comprising: At least one processor is configured as follows: Predicted pixel values ​​for the current block are obtained by performing optical flow-based compensation on the current block using the gradient values ​​of at least one reference pixel in a first reference block, the gradient values ​​of at least one reference pixel in a second reference block, and the displacement vector of a pixel group in the current block. The pixel group includes at least one pixel adjacent to the inner side of the boundary of the current block. The residual pixel value is obtained based on the predicted pixel value and the pixel value of the current block. Specifically, a first extended reference block in the first reference frame and a second extended reference block in the second reference frame are obtained. The first extended reference block includes a first reference block and a first portion extending from the first reference block, and the second extended reference block includes a second reference block and a second portion extending from the second reference block. The displacement vector of the pixel group in the current block is determined using the gradient values ​​of at least one reference pixel in a first extended reference block and at least one reference pixel in a second extended reference block. The first portion of the first extended reference block is used to calculate the gradient value of at least one reference pixel in the first reference block, and the second portion of the second extended reference block is used to calculate the gradient value of at least one reference pixel in the second reference block. The pixel group comprises K×K blocks, where K is an integer greater than 1.

3. A method for transmitting a bit stream, comprising: Execute the video encoding method to generate a bitstream; And sending the bit stream, The video encoding method includes: Predicted pixel values ​​for the current block are obtained by performing optical flow-based compensation on the current block using the gradient values ​​of at least one reference pixel in a first reference block, the gradient values ​​of at least one reference pixel in a second reference block, and the displacement vector of a pixel group in the current block. The pixel group includes at least one pixel adjacent to the inner side of the boundary of the current block. The residual pixel value is obtained based on the predicted pixel value and the pixel value of the current block. Specifically, a first extended reference block in the first reference frame and a second extended reference block in the second reference frame are obtained. The first extended reference block includes a first reference block and a first portion extending from the first reference block, and the second extended reference block includes a second reference block and a second portion extending from the second reference block. The displacement vector of the pixel group in the current block is determined using the gradient values ​​of at least one reference pixel in a first extended reference block and at least one reference pixel in a second extended reference block. The first portion of the first extended reference block is used to calculate the gradient value of at least one reference pixel in the first reference block, and the second portion of the second extended reference block is used to calculate the gradient value of at least one reference pixel in the second reference block. The pixel group comprises K×K blocks, where K is an integer greater than 1.

Images