Devices for decoding video and devices for encoding video

Abstract

Provided is a video decoding method, comprising: obtaining motion prediction mode information about a current block in a current picture; when a bidirectional motion prediction mode is indicated, obtaining a first motion vector and a second motion vector respectively indicating a first reference block of the current block in a first reference picture and a second reference block of the current block in a second reference picture; generating a pixel value of a first pixel of the first reference block indicated by the first motion vector and a pixel value of a second pixel of the second reference block indicated by the second motion vector by applying an interpolation filter to a first neighboring area of ​​the first pixel and a second neighboring area of ​​the second pixel; generating gradient values ​​of the first pixel and the second pixel by applying a filter to the first neighboring area and the second neighboring area; and generating a predicted pixel value of the current block by using the pixel values ​​and the gradient values ​​of the first pixel and the second pixel.

Description

[0001] This application is a divisional application of a patent application with application date of November 11, 2016, application number 201680079512.9, and titled “Method and device for decoding video and method and device for encoding video”. Technical Field

[0002] The present disclosure relates to a video decoding method and a video encoding method. More specifically, the present disclosure relates to video decoding and video encoding that perform inter-frame prediction in a bidirectional motion prediction mode. Background Art

[0003] As hardware for reproducing and storing high-resolution or high-quality video content is being developed and supplied, the need for a video codec for efficiently encoding or decoding high-resolution or high-quality video content has increased. In conventional video codecs, video is encoded according to a limited encoding method based on a tree-structured coding unit.

[0004] Image data in the spatial domain is transformed into coefficients in the frequency domain via frequency transformation. According to the video codec, the image is divided into blocks of a predetermined size, discrete cosine transform (DCT) is performed on each block, and the frequency coefficients are encoded in block units for fast calculation of frequency transformation. Compared with image data in the spatial domain, coefficients in the frequency domain are easily compressed. In particular, since the image pixel value in the spatial domain is expressed based on the prediction error of inter-frame prediction or intra-frame prediction via the video codec, a large amount of data can be transformed into 0 when the prediction error is subjected to frequency transformation. According to the video codec, the amount of data can be reduced by replacing the data that is continuously and repeatedly generated with small data. Summary of the invention

[0005] Technical issues

[0006] According to various embodiments, a predicted pixel value of a current block can be generated in a bidirectional motion prediction mode by using not only a pixel value of a first reference block of a first reference picture and a pixel value of a second reference block of a second reference picture but also a first gradient value of the first reference block and a second gradient value of the second reference block. Therefore, encoding and decoding efficiency can be improved because a predicted block similar to an original block can be generated.

[0007] Provided is a computer-readable recording medium having recorded thereon a program for executing the method according to various embodiments.

[0008] Aspects of the various embodiments herein are not limited thereto, and additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0009] Solution to the problem

[0010] Aspects of the disclosure are not limited thereto, and additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0011] According to one aspect of the present disclosure, a video decoding method includes: obtaining motion prediction mode information about a current block in a current picture from a bitstream; when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtaining a first motion vector indicating a first reference block of the current block in a first reference picture and a second motion vector indicating a second reference block of the current block in a second reference picture from the bitstream; generating a pixel value of the first pixel and a pixel value of the second pixel by applying an interpolation filter to a first neighboring area of ​​the first pixel and a second neighboring area of ​​the second pixel, wherein the first pixel comes from pixels of the first reference block indicated by the first motion vector, and the second pixel comes from pixels of the second reference block indicated by the second motion vector; generating a gradient value of the first pixel and a gradient value of the second pixel by applying a filter to the first neighboring area and the second neighboring area; and generating a predicted pixel value of the current block by using the pixel value of the first pixel, the pixel value of the second pixel, the gradient value of the first pixel, and the gradient value of the second pixel.

[0012] The generating of the predicted pixel values ​​of the current block may include: determining a displacement vector per unit time in the horizontal direction and the vertical direction by using a gradient value of a pixel in a first window having a specific size and including the first pixel, a gradient value of a pixel in a second window having a specific size and including the second pixel, a pixel value of the pixel in the first window, and a pixel value of the pixel in the second window; and generating the predicted pixel value of the current block by further using the displacement vector per unit time in the horizontal direction and the vertical direction, a first temporal distance between the current picture and the first reference picture, and a second temporal distance between the current picture and the second reference picture.

[0013] The first motion vector may be a vector indicating the first pixel at a fractional pixel position (i+α, j+β), wherein i and j are each integers and α and β are each fractions, and generating the gradient value of the first pixel and the gradient value of the second pixel may include: generating the gradient value at the pixel position (i, j+β) by applying a gradient filter to the pixel value at the pixel position (i, j) and pixel values ​​of integer pixels located in the vertical direction from the pixel at the pixel position (i, j); and generating the gradient value of the first pixel at the fractional pixel position (i+α, j+β) in the vertical direction by applying an interpolation filter to the gradient value at the pixel position (i, j+β) and gradient values ​​of pixels whose positions of horizontal components are integer pixel positions, wherein the pixels whose positions of horizontal components are integer pixel positions are from among the pixels located in the horizontal direction from the pixel position (i, j+β).

[0014] The first motion vector may be a vector indicating the first pixel at a fractional pixel position (i+α, j+β), wherein i and j are each integers and α and β are each fractions, and generating the gradient value of the first pixel and the gradient value of the second pixel may include: generating the pixel value at the pixel position (i, j+β) by applying an interpolation filter to the pixel value at the pixel position (i, j) and pixel values ​​of integer pixels located in the vertical direction from the pixel at the pixel position (i, j); and generating the gradient value of the first pixel at the fractional pixel position (i+α, j+β) in the horizontal direction by applying a gradient filter to the pixel value at the pixel position (i, j+β) and pixel values ​​of pixels whose positions of horizontal components are integer pixel positions, wherein the pixels whose positions of horizontal components are integer pixel positions are from among the pixels located in the horizontal direction from the pixel position (i, j+β).

[0015] Generating the gradient value at the pixel position (i, j+β) may include: generating the gradient value at the pixel position (i, j+β) by applying an M-tap gradient filter to the pixel at the pixel position (i, j) and M-1 neighboring integer pixels located in the vertical direction from the pixel at the pixel position (i, j), wherein M is an integer, and generating the gradient value of the first pixel at the fractional pixel position (i+α, j+β) in the vertical direction may include generating the gradient value of the first pixel at the fractional pixel position (i+α, j+β) in the vertical direction by applying an N-tap interpolation filter to the gradient value at the pixel position (i, j+β) and gradient values ​​of N-1 pixels whose positions of horizontal components are integer pixel positions, wherein N is an integer, and the N-1 pixels whose positions of horizontal components are integer pixel positions are from among the pixels located in the horizontal direction from the pixel position (i, j+β).

[0016] Generating the pixel value at the pixel position (i, j+β) may include: generating the pixel value at the pixel position (i, j+β) by applying an N-tap interpolation filter to the pixel value at the pixel position (i, j+β) and pixel values ​​of N-1 neighboring pixels whose components in a predetermined direction are integers, and the N-1 neighboring pixels are located in the vertical direction starting from the pixel position (i, j+β), wherein N is an integer, and generating the gradient value of the first pixel at the fractional pixel position (i+α, j+β) in the horizontal direction may include generating the gradient value of the first pixel at the fractional pixel position (i+α, j+β) in the horizontal direction by applying an M-tap gradient filter to the pixel value at the pixel position (i, j+β) and pixel values ​​of M-1 pixels whose positions of the horizontal component are integer pixel positions, wherein M is an integer, and the M-1 pixels whose positions of the horizontal component are integer pixel positions are from among the pixels located in the horizontal direction starting from the pixel position (i, j+β).

[0017] The N-tap interpolation filter may be a discrete cosine transform-based interpolation filter for pixel values ​​at fractional pixel positions in a horizontal direction, and N may be 6.

[0018] The M-tap gradient filter may be an interpolation filter for gradient values ​​at fractional pixel positions in a vertical direction, wherein filter coefficients are predetermined by using a discrete cosine transform-based interpolation filter in the vertical direction, and M may be 6.

[0019] Generating the gradient value at the pixel position (i, j+β) by applying the M-tap gradient filter may include generating the gradient value at the pixel position (i, j+β) by performing descaling on the value to which the M-tap gradient filter is applied based on a descaling bit number, wherein the descaling bit number may be based on a bit depth of a sampling point, and the descaling may include shifting the descaling bit number to the right.

[0020] When the filter coefficients of the M-tap gradient filter are scaled by 2^x and the filter coefficients of the N-tap interpolation filter are scaled by 2^y, where x and y are each integers, generating the gradient value of the first pixel at the fractional pixel position (i+α, j+β) in the vertical direction by applying the N-tap interpolation filter may include generating the gradient value of the first pixel at the fractional pixel position (i+α, j+β) in the vertical direction by descaling the value to which the N-tap interpolation filter is applied based on a descaling bit number, the descaling bit number may be based on the bit depth of the sampling point, x and y, and the descaling may include shifting the descaling bit number rightward.

[0021] According to another aspect of the present disclosure, a video decoding device includes: an obtainer, the obtainer being configured to obtain motion prediction mode information about a current block in a current picture from a bitstream, and when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtain from the bitstream a first motion vector indicating a first reference block of the current block in a first reference picture and a second motion vector indicating a second reference block of the current block in a second reference picture; and an inter-frame predictor, the inter-frame predictor being configured to generate a pixel value of the first pixel and a pixel value of the second pixel by applying an interpolation filter to a first neighboring area of ​​the first pixel and a second neighboring area of ​​the second pixel, generate a gradient value of the first pixel and a gradient value of the second pixel by applying a filter to the first neighboring area and the second neighboring area, and generate a predicted pixel value of the current block by using the pixel value of the first pixel, the pixel value of the second pixel, the gradient value of the first pixel, and the gradient value of the second pixel, wherein the first pixel is from among the pixels of the first reference block indicated by the first motion vector, and the second pixel is from among the pixels of the second reference block indicated by the second motion vector.

[0022] According to another aspect of the present disclosure, a video encoding method includes: determining a first reference block of a first reference picture corresponding to a current block in a current picture and a second reference block of a second reference picture corresponding to the current block, and determining a first motion vector indicating the first reference block and a second motion vector indicating the second reference block; generating a pixel value of the first pixel and a pixel value of the second pixel by applying an interpolation filter to a first neighboring area of ​​a first pixel from among pixels of the first reference block and a second neighboring area of ​​a second pixel from among pixels of the second reference block; generating a gradient value of the first pixel and a gradient value of the second pixel by applying a filter to the first neighboring area of ​​the first pixel and the second neighboring area of ​​the second pixel, and generating a predicted pixel value of the current block by using the pixel value of the first pixel, the pixel value of the second pixel, the gradient value of the first pixel and the gradient value of the second pixel; and generating a bit stream including information about the first motion vector and the second motion vector and motion prediction mode information indicating that the motion prediction mode of the current block is a bidirectional motion prediction mode.

[0023] According to another aspect of the present disclosure, a video encoding device includes: an inter-frame predictor, which is configured to determine a first reference block of a first reference picture corresponding to a current block in a current picture and a second reference block of a second reference picture corresponding to the current block, determine a first motion vector indicating the first reference block and a second motion vector indicating the second reference block, generate a pixel value of the first pixel and a pixel value of the second pixel by applying an interpolation filter to a first neighboring area of ​​a first pixel from among pixels of the first reference block and a second neighboring area of ​​a second pixel from among pixels of the second reference block, generate a gradient value of the first pixel and a gradient value of the second pixel by applying a filter to the first neighboring area of ​​the first pixel and the second neighboring area of ​​the second pixel, and generate a predicted pixel value of the current block by using the pixel value of the first pixel, the pixel value of the second pixel, the gradient value of the first pixel, and the gradient value of the second pixel; and a bitstream generator, which is configured to generate a bitstream including information about the first motion vector and the second motion vector and motion prediction mode information indicating that the motion prediction mode of the current block is a bidirectional motion prediction mode.

[0024] According to another aspect of the present disclosure, a computer-readable recording medium has recorded thereon a program for executing the method according to various embodiments.

[0025] Disclosure of beneficial effects

[0026] According to various embodiments, encoding and decoding efficiency may be improved by performing inter prediction on a current block in a bidirectional motion prediction mode by using a gradient value of a reference block of a reference picture to predict a value similar to that of an original block of the current block. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1a is a block diagram of a video decoding device according to various embodiments.

[0028] Figure 1b is a flowchart of a video decoding method according to various embodiments.

[0029] Figure 1c is a block diagram of a video encoding apparatus according to various embodiments.

[0030] Figure 1d is a flowchart of a video encoding method according to various embodiments.

[0031] Figure 1e is a block diagram of an image decoder according to various embodiments.

[0032] Figure 1f is a block diagram of an image encoder according to various embodiments.

[0033] Figure 2 is a reference diagram for describing a block-based bidirectional motion prediction and compensation process according to an embodiment.

[0034] Figure 3a to Figure 3c is a reference diagram for describing a process of performing pixel-unit motion compensation according to an embodiment.

[0035] Figure 4 is a reference diagram for describing a process of calculating gradient values ​​in horizontal and vertical directions according to an embodiment.

[0036] Figure 5 is a reference diagram for describing a process of calculating gradient values ​​in horizontal and vertical directions according to another embodiment.

[0037] Figure 6a and Figure 6b is a diagram for describing a process of determining gradient values ​​in horizontal and vertical directions by using a one-dimensional (1D) filter according to an embodiment.

[0038] Figures 7a to 7e is a table showing filter coefficients of a filter for determining pixel values ​​at fractional pixel positions of fractional pixel units and gradient values ​​in horizontal and vertical directions according to an embodiment.

[0039] Figure 8is a reference diagram for describing a process of determining a horizontal direction displacement vector and a vertical direction displacement vector according to an embodiment.

[0040] Fig. 9 is a diagram for describing a process of adding an offset value after performing filtering and determining a gradient value in a horizontal or vertical direction by performing inverse scaling according to an embodiment.

[0041] Fig.10 A process of determining at least one coding unit as a current coding unit is split according to an embodiment is illustrated.

[0042] Fig.11 A process of determining at least one coding unit when a coding unit having a non-square shape is split according to an embodiment is illustrated.

[0043] Fig.12 A process of splitting a coding unit based on at least one of block shape information and partition shape information according to an embodiment is illustrated.

[0044] Fig.13 A method of determining a specific coding unit from among an odd number of coding units according to an embodiment is illustrated.

[0045] Fig.14 An order in which a plurality of coding units are processed when a plurality of coding units are determined when a current coding unit is split according to an embodiment is illustrated.

[0046] Fig.15 A process of determining that a current coding unit is split into an odd number of coding units when coding units cannot be processed in a specific order according to an embodiment is illustrated.

[0047] Fig.16 A process of determining at least one coding unit when a first coding unit is split according to an embodiment is illustrated.

[0048] Fig.17 It is illustrated that shapes into which the second coding unit may be split are restricted when a second coding unit having a non-square shape determined when a first coding unit is split satisfies a specific condition according to an embodiment.

[0049] Fig.18 A process of splitting a coding unit having a square shape when split shape information may not indicate that the coding unit is split into four square shapes according to an embodiment is illustrated.

[0050] Fig.19 It is illustrated that an order of processing a plurality of coding units may be changed according to a process of splitting the coding units according to an embodiment.

[0051] Fig. 20A process of determining a depth of a coding unit as a shape and a size of a coding unit are changed when a plurality of coding units are determined when the coding unit is recursively split according to an embodiment is illustrated.

[0052] Fig.21 An example of distinguishing a depth and a partial index (PID) of a coding unit, which may be determined according to a shape and a size of a coding unit, according to an embodiment is illustrated.

[0053] Fig. 22 It is illustrated that a plurality of coding units are determined according to a plurality of specific data units included in a picture according to an embodiment.

[0054] Fig.23 A processing block serving as a criterion for determining an order of determining reference coding units included in a picture according to an embodiment is illustrated.

[0055] Best Mode

[0056] According to one aspect of the present disclosure, a video decoding method includes: obtaining motion prediction mode information about a current block in a current picture from a bitstream; when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtaining from the bitstream a first motion vector indicating a first reference block of the current block in a first reference picture and a second motion vector indicating a second reference block of the current block in a second reference picture; generating a pixel value of a first pixel from among pixels of the first reference block indicated by the first motion vector and a pixel value of a second pixel from among pixels of the second reference block indicated by the second motion vector by applying an interpolation filter to a first neighboring area of ​​the first pixel and a second neighboring area of ​​the second pixel; generating a gradient value of the first pixel and a gradient value of the second pixel by applying a filter to the first neighboring area and the second neighboring area; and generating a predicted pixel value of the current block by using the pixel value of the first pixel, the pixel value of the second pixel, the gradient value of the first pixel, and the gradient value of the second pixel.

[0057] According to another aspect of the present disclosure, a video decoding device includes: an obtainer, which is configured to obtain motion prediction mode information about a current block in a current picture from a bitstream, and when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtain a first motion vector indicating a first reference block of the current block in a first reference picture and a second motion vector indicating a second reference block of the current block in a second reference picture from the bitstream; and an inter-frame predictor, which is configured to generate a pixel value of a first pixel from among pixels of the first reference block indicated by the first motion vector and a pixel value of a second pixel from among pixels of the second reference block indicated by the second motion vector by applying an interpolation filter to a first neighboring area of ​​the first pixel and a second neighboring area of ​​the second pixel, generate a gradient value of the first pixel and a gradient value of the second pixel by applying a filter to the first neighboring area and the second neighboring area, and generate a predicted pixel value of the current block by using the pixel value of the first pixel, the pixel value of the second pixel, the gradient value of the first pixel, and the gradient value of the second pixel.

[0058] According to another aspect of the present disclosure, a video encoding method includes: determining a first reference block of a first reference picture corresponding to a current block in a current picture and a second reference block of a second reference picture corresponding to the current block, and determining a first motion vector indicating the first reference block and a second motion vector indicating the second reference block; generating a pixel value of a first pixel from among pixels of the first reference block and a pixel value of a second pixel from among pixels of the second reference block by applying an interpolation filter to a first neighboring area of ​​the first pixel and a second neighboring area of ​​the second pixel; generating a gradient value of the first pixel and a gradient value of the second pixel by applying a filter to a first neighboring area of ​​the first pixel and a second neighboring area of ​​the second pixel, and generating a predicted pixel value of the current block by using the pixel value of the first pixel, the pixel value of the second pixel, the gradient value of the first pixel and the gradient value of the second pixel; and generating a bit stream including information about the first motion vector and the second motion vector and motion prediction mode information indicating that the motion prediction mode of the current block is a bidirectional motion prediction mode.

[0059] According to another aspect of the present disclosure, a video encoding device includes: an inter-frame predictor, which is configured to determine a first reference block of a first reference picture corresponding to a current block in a current picture and a second reference block of a second reference picture corresponding to the current block, determine a first motion vector indicating the first reference block and a second motion vector indicating the second reference block, generate a pixel value of a first pixel from among pixels of the first reference block and a pixel value of a second pixel from among pixels of the second reference block by applying an interpolation filter to a first neighboring area of ​​the first pixel and a second neighboring area of ​​the second pixel, generate a gradient value of the first pixel and a gradient value of the second pixel by applying a filter to the first neighboring area of ​​the first pixel and the second neighboring area of ​​the second pixel, and generate a predicted pixel value of the current block by using the pixel value of the first pixel, the pixel value of the second pixel, the gradient value of the first pixel and the gradient value of the second pixel; and a bitstream generator, which is configured to generate a bitstream including information about the first motion vector and the second motion vector and motion prediction mode information indicating that the motion prediction mode of the current block is a bidirectional motion prediction mode.

[0060] According to another aspect of the present disclosure, a computer-readable recording medium has recorded thereon a program for executing the method according to various embodiments. DETAILED DESCRIPTION

[0061] Hereinafter, an “image” may mean a still image or a moving image of a video, that is, the video itself.

[0062] Hereinafter, a "sampling point" refers to a sampling position assigned to an image and data to be processed. For example, a pixel in an image in a spatial domain may be a sampling point.

[0063] Hereinafter, a 'current block' may mean a block of an image to be encoded or decoded.

[0064] Figure 1a is a block diagram of a video decoding device according to various embodiments.

[0065] The video decoding apparatus 100 according to various embodiments includes an obtainer 105 and an inter-frame predictor 110 .

[0066] The obtainer 105 receives a bitstream including information about a prediction mode of a current block, information indicating a motion prediction mode of the current block, and information about a motion vector.

[0067] The obtainer 105 may obtain information about the prediction mode of the current block, information indicating the motion prediction mode of the current block, and information about the motion vector from the received bitstream. In addition, the obtainer 105 may obtain a reference picture index indicating a reference picture from among previously decoded pictures from the bitstream.

[0068] When the prediction mode of the current block is the inter prediction mode, the inter predictor 110 performs inter prediction on the current block. In other words, the inter predictor 110 can generate a predicted pixel value of the current block by using at least one of the pictures decoded before the current picture including the current block. For example, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the inter predictor 110 can generate a predicted pixel value of the current block by using two pictures decoded before the current picture. In other words, when the information about the motion prediction mode obtained from the bitstream indicates a bidirectional motion prediction mode, the inter predictor 110 can generate a predicted pixel value of the current block by using two pictures decoded before the current picture.

[0069] The inter predictor 110 may include a block-unit motion compensator 115 and a pixel-unit motion compensator 120 .

[0070] The block unit motion compensator 115 may perform motion compensation on the current block in block units. The block unit motion compensator 115 may determine at least one reference picture from a previously decoded picture by using a reference picture index obtained from a bitstream. Here, the reference picture index may represent a reference picture index for each of prediction directions including an L0 direction and an L1 direction. Here, the reference picture index for the L0 direction may represent an index indicating a reference picture among pictures included in the L0 reference picture list, and the reference picture index for the L1 direction may represent an index indicating a reference picture among pictures included in the L1 reference picture list.

[0071] The block unit motion compensator 115 may determine a reference block of a current block, the reference block being located in at least one reference picture, by using information about a motion vector received from a bitstream. Here, a corresponding block corresponding to the current block may be a reference block.

[0072] In other words, the block-unit motion compensator 115 may determine a reference block of the current block by using a motion vector indicating a reference block from the current block.

[0073] Here, the motion vector refers to a vector indicating the displacement of the reference coordinates of the current block in the current picture and the reference coordinates of the reference block in the reference picture. For example, when the upper left coordinates of the current block are (1,1) and the upper left coordinates of the reference block in the reference picture are (3,3), the motion vector may be (2,2).

[0074] Here, the information about the motion vector may include a differential value of the motion vector, and the block unit motion compensator 115 may reconstruct the motion vector by using a predictor of the motion vector and the differential value of the motion vector obtained from the bitstream, and determine a reference block of the current block located in at least one reference picture by using the reconstructed motion vector. Here, the differential value of the motion vector may represent a differential value of a motion vector of a reference picture related to each of prediction directions including an L0 direction and an L1 direction. Here, the differential value of the motion vector of the L0 direction may represent a differential value of a motion vector indicating a reference block in a reference picture included in the L0 reference picture list, and the differential value of the motion vector of the L1 direction may represent a differential value of a motion vector indicating a reference block in a reference picture included in the L1 reference picture list.

[0075] The block unit motion compensator 115 may perform motion compensation on the current block in block units by using pixel values ​​of the reference block. The block unit motion compensator 115 may perform motion compensation on the current block in block units by using pixel values ​​of reference pixels in the reference block corresponding to the current pixel in the current block. Here, the reference pixel may be a pixel included in the reference block, and a corresponding pixel corresponding to the current pixel in the current block may be the reference pixel.

[0076] The block unit motion compensator 115 may perform motion compensation on the current block in block units by using a plurality of reference blocks respectively included in a plurality of reference pictures. For example, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the block unit motion compensator 115 may determine two reference pictures from among previously encoded pictures, and determine two reference blocks included in the two reference pictures.

[0077] The block unit motion compensator 115 may perform motion compensation on the current block in block units by using the pixel values ​​of two reference pixels in the two reference blocks. The block unit motion compensator 115 may perform motion compensation on the current block in block units by using the average value or weighted sum of the pixel values ​​of the two reference pixels, thereby generating a motion compensation value in block units.

[0078] The reference position of the reference block may be the position of an integer pixel, but is not limited thereto, and may be the position of a fractional pixel. Here, an integer pixel may represent a pixel whose position component is an integer, and may be a pixel at an integer pixel position. A fractional pixel may represent a pixel whose position component is a fraction, and may be a pixel at a fractional pixel position.

[0079] For example, when the upper left coordinate of the current block is (1, 1) and the motion vector is (2.5, 2.5), the upper left coordinate of the reference block in the reference picture may be (3.5, 3.5). Here, the position of the fractional pixel may be determined in units of 1 / 4 pel or 1 / 16 pel, where pel represents a pixel element. Alternatively, the position of the fractional pixel may be determined in units of various fractional pels.

[0080] When the reference position of the reference block is a fractional pixel position, the block unit motion compensator 115 can generate a pixel value of a first pixel from among the pixels of the first reference block indicated by a first motion vector and a pixel value of a second pixel from among the pixels of the second reference block indicated by a second motion vector by applying an interpolation filter to a first neighboring area including the first pixel and a second neighboring area including the second pixel.

[0081] In other words, the pixel value of the reference pixel in the reference block can be determined by using the pixel values ​​of neighboring pixels whose components in a specific direction are integers. Here, the specific direction can be a horizontal direction or a vertical direction.

[0082] For example, the block unit motion compensator 115 can determine the value obtained by filtering the pixel value of the pixel whose component in a specific direction is an integer as the pixel value of the reference pixel by using an interpolation filter, and determine the motion compensation value of the current block by using the pixel value of the reference pixel. The motion compensation value by block unit is determined by using the average value or weighted sum of the reference pixels. Here, the interpolation filter can be an M-tap interpolation filter based on discrete cosine transform (DCT). The coefficients of the M-tap interpolation filter based on DCT can be summarized from DCT and inverse DCT (IDCT). Here, the coefficients of the interpolation filter can be filter coefficients scaled to integer coefficients to reduce real number operations during filtering. Here, the interpolation filter can be a one-dimensional (1D) interpolation filter in the horizontal or vertical direction. For example, when the position of the pixel is expressed by x, y orthogonal coordinate components, the horizontal direction can be a direction parallel to the x-axis. The vertical direction can be a direction parallel to the y-axis.

[0083] The block-unit motion compensator 115 may first perform filtering on pixel values ​​of pixels at integer positions by using a 1D interpolation filter in a vertical direction, and then perform filtering on the values ​​generated by filtering by using a 1D interpolation filter in a horizontal direction to determine pixel values ​​of reference pixels at fractional pixel positions.

[0084] Meanwhile, a value generated via filtering when the scaled filter coefficient is used may be greater than a value generated via filtering when the unscaled filter coefficient is used. Therefore, the block-unit motion compensator 115 may perform de-scaling on the value generated via filtering.

[0085] The block unit motion compensator 115 may perform descaling after filtering the pixel values ​​of the pixels at the integer positions by using a 1D interpolation filter in the vertical direction. Here, descaling may include right bit shifting the descaling bit number. The descaling bit number may be determined based on the bit depth of the sampling point of the input image. For example, the descaling bit number may be a value obtained by subtracting 8 from the bit depth of the sampling point.

[0086] In addition, the block unit motion compensator 115 may perform filtering for the pixel values ​​of the pixels at the integer positions by using a 1D interpolation filter in the vertical direction, and perform filtering for the values ​​generated by filtering by using a 1D interpolation filter in the horizontal direction, and then perform descaling. Here, descaling may include shifting the descaling bit number to the right. The descaling bit number may be determined based on the scaling bit number of the 1D interpolation filter in the vertical direction, the scaling bit number of the 1D interpolation filter in the horizontal direction, and the bit depth of the sampling point. For example, when the scaling bit number p of the 1D interpolation filter in the vertical direction is 6, the scaling bit number q of the 1D interpolation filter in the horizontal direction is 6, and the bit depth of the sampling point is b, the descaling bit number may be p+q+8-b, i.e., 20-b.

[0087] When the block unit motion compensator 115 performs only right shifting the number of descaling bits after performing filtering for pixels whose components in a specific direction are integers by using a 1D interpolation filter, a rounding error may be generated, and thus the block unit motion compensator 115 may perform descaling after performing filtering for pixels whose components in a specific direction are integers by using a 1D interpolation filter and then adding an offset value. Here, the offset value may be 2^(number of descaling bits-1).

[0088] The pixel unit motion compensator 120 may generate motion compensation values ​​by pixel unit by performing motion compensation on the current block by pixel unit. When the motion prediction mode of the current block is a bidirectional motion prediction mode, the pixel unit motion compensator 120 may generate motion compensation values ​​by pixel unit by performing motion compensation on the current block by pixel unit.

[0089] The pixel-unit motion compensator 120 may generate a motion compensation value by pixel unit by performing motion compensation on the current block by pixel unit based on the optical flows of the pixels of the first reference picture and the second reference picture. Figure 3a Describe optical flow.

[0090] The pixel-unit motion compensator 120 may generate a motion compensation value by pixel unit by performing motion compensation by pixel unit with respect to pixels included in a reference block of the current block.

[0091] The pixel-unit motion compensator 120 may determine a reference pixel in a reference block corresponding to a current pixel of a current block, and determine a gradient value of the reference pixel.

[0092] The pixel-unit motion compensator 120 may perform motion compensation on a current block on a pixel-by-pixel basis by using gradient values ​​of reference pixels, thereby generating motion compensation values ​​on a pixel-by-pixel basis.

[0093] The pixel unit motion compensator 120 can generate a gradient value of a first pixel from among pixels of a first reference block indicated by a first motion vector and a gradient value of a second pixel from among pixels of a second reference block indicated by a second motion vector by applying a filter to a first neighboring area of ​​the first pixel including the first pixel and a second neighboring area of ​​the second pixel including the second pixel.

[0094] The pixel-unit motion compensator 120 may determine pixel values ​​and gradient values ​​of pixels in a first window having a specific size around a first pixel in a first reference picture and including the first pixel, and determine pixel values ​​and gradient values ​​of pixels in a second window having a specific size around a second pixel in a second reference picture and including the second pixel.

[0095] The pixel-unit motion compensator 120 may determine a displacement vector per unit time with respect to the current pixel by using pixel values ​​and gradient values ​​of pixels in the first window and pixel values ​​and gradient values ​​of pixels in the second window.

[0096] The pixel-unit motion compensator 120 may perform motion compensation on the current block by pixel unit by using a displacement vector per unit time with respect to the current pixel and a gradient value of a reference pixel.

[0097] The reference position of the reference block may be an integer pixel position, but may alternatively be a fractional pixel position.

[0098] When the reference position of the reference block is a fractional pixel position, the gradient value of the reference pixel in the reference block may be determined by using pixel values ​​of neighboring pixels whose components in a specific direction are integers.

[0099] For example, the pixel unit motion compensator 120 may determine the result value obtained by filtering the pixel values ​​of neighboring pixels whose components in a specific direction are integers using a gradient filter as the gradient value of the reference pixel. Here, the filter coefficients of the gradient filter may be determined by using coefficients predetermined for a DCT-based interpolation filter. The filter coefficients of the gradient filter may be filter coefficients scaled to integer coefficients in order to reduce real number operations during filtering.

[0100] Here, the gradient filter may be a 1D gradient filter in the horizontal or vertical direction.

[0101] The pixel-unit motion compensator 120 may perform filtering on neighboring pixels whose components in a corresponding direction are integers by using a 1D gradient filter in a horizontal or vertical direction so as to determine a gradient value of a reference pixel in a horizontal or vertical direction.

[0102] For example, the pixel-unit motion compensator 120 may determine a gradient value of a reference pixel in the horizontal direction by filtering such pixels from among pixels adjacent to the reference pixel using a 1D gradient filter in the horizontal direction, wherein the pixels are positioned in the horizontal direction from the pixels whose horizontal direction components are integers.

[0103] When the position of the reference pixel is (x+α, y+β) (where x and y are each integers and α and β are each fractions), the pixel unit motion compensator 120 can determine the pixel value at the pixel position (x, y+β) as a result value obtained by filtering the pixel at the (x, y) position and the pixels whose vertical components are integers using a 1D interpolation filter, wherein the pixels whose vertical components are integers are from among the pixels positioned in the horizontal direction from the pixel at the (x, y) position.

[0104] The pixel-unit motion compensator 120 may determine a result value obtained by filtering the pixel values ​​of the pixel at the (x, y+β) position and the pixels whose horizontal components are integers using a 1D gradient filter in the horizontal direction as a gradient value at the (x+α, y+β) position in the horizontal direction, wherein the pixels whose horizontal components are integers are from among the pixels positioned in the horizontal direction from the pixel at the (x, y+β) position.

[0105] The order of using the 1D gradient filter and the 1D interpolation filter is not limited. In the above description, the pixels at the integer positions are first filtered by using the 1D interpolation filter in the vertical direction to generate the interpolation filter values ​​in the vertical direction, and then the interpolation filter values ​​in the vertical direction are filtered by using the 1D gradient filter in the horizontal direction, but alternatively, the pixels at the integer positions are first filtered by using the 1D gradient filter in the horizontal direction to generate the interpolation filter values ​​in the horizontal direction, and then the interpolation filter values ​​in the horizontal direction can be filtered by using the 1D interpolation filter in the vertical direction.

[0106] In the above, it has been described in detail that the pixel-unit motion compensator 120 determines the gradient value in the horizontal direction at the position (x+α, y+β). Since the pixel-unit motion compensator 120 determines the gradient value in the vertical direction at the position (x+α, y+β) in a manner similar to determining the gradient value in the horizontal direction, the details thereof are not provided.

[0107] In the above, it has been described in detail that the pixel unit motion compensator 120 uses a 1D gradient filter and a 1D interpolation filter to determine the gradient value at the fractional pixel position. However, alternatively, the gradient filter and the interpolation filter may be used to determine the gradient value at the integer pixel position. However, in the case of integer pixels, the pixel value may be determined without using an interpolation filter, but for a process consistent with the process in the fractional pixel, the pixel value of the integer pixel may be determined by using an interpolation filter to perform filtering on the integer pixel and the neighboring pixels whose components in a specific direction are integers. For example, the interpolation filter coefficient in the integer pixel may be {0, 0, 64, 0, 0}, and because the interpolation filter coefficient related to the neighboring integer pixel is 0, filtering may be performed by using only the pixel value of the current integer pixel, and as a result, the pixel value of the current integer pixel may be determined by using an interpolation filter to perform filtering on the current integer pixel and the neighboring integer pixels.

[0108] The pixel unit motion compensator 120 may perform descaling after filtering the pixels at the integer positions by using a 1D interpolation filter in the vertical direction. Here, descaling may include shifting the descaling bit number to the right. The descaling bit number may be determined based on the bit depth of the sampling point. For example, the descaling bit number may be a value obtained by subtracting 8 from the bit depth of the sampling point.

[0109] The pixel unit motion compensator 120 may perform descaling after filtering the values ​​generated by performing descaling using a gradient filter in the horizontal direction. Here, descaling may include shifting the descaling bit number to the right. The descaling bit number may be determined based on the scaling bit number of the 1D interpolation filter in the vertical direction, the scaling bit number of the 1D gradient filter in the horizontal direction, and the bit depth of the sampling point. For example, when the scaling bit number p of the 1D interpolation filter in the vertical direction is 6, the scaling bit number q of the 1D gradient filter in the horizontal direction is 4, and the bit depth of the sampling point is b, the descaling bit number may be p+q+8-b, i.e., 18-b.

[0110] When the pixel-unit motion compensator 120 performs only right shifting by the number of descaling bits on the value generated via filtering after performing filtering, a rounding error may be generated, and thus the pixel-unit motion compensator 120 may perform descaling after adding an offset value to the value generated via filtering. Here, the offset value may be 2^(number of descaling bits-1).

[0111] The inter-frame predictor 110 may generate a predicted pixel value of the current block by using a motion compensation value in block units and a motion compensation value in pixel units with respect to the current block. For example, the inter-frame predictor 110 may generate a predicted pixel value of the current block by adding a motion compensation value in block units and a motion compensation value in pixel units with respect to the current block. Here, the motion compensation value in block units may represent a value generated by performing motion compensation in block units, and the motion compensation value in pixel units represents a value generated by performing motion compensation in pixel units, wherein the motion compensation value in block units may be an average value or a weighted sum of reference pixels, and the motion compensation value in pixel units may be a value determined based on a displacement vector related to the current pixel and a gradient value of the reference pixel.

[0112] The video decoding apparatus 100 may obtain a residual block of a current block from a bitstream, and reconstruct the current block by using the residual block of the current block and the predicted pixel value. For example, the video decoding apparatus 100 may determine the pixel value of the reconstructed block from the bitstream by adding the pixel value of the residual block of the current block and the predicted pixel value of the current block.

[0113] The video decoding apparatus 100 may include an image decoder (not shown), wherein the image decoder may include an obtainer 105 and an inter-frame predictor 110. Figure 1e Describes an image decoder.

[0114] Figure 1b is a flowchart of a video decoding method according to various embodiments.

[0115] In operation S105, the video decoding apparatus 100 may obtain motion prediction mode information about the current block in the current picture from the bitstream. The video decoding apparatus 100 may receive a bitstream including motion prediction mode information about the current block in the current picture, and obtain motion prediction mode information about the current block from the received bitstream.

[0116] The video decoding apparatus 100 may obtain information about the prediction mode of the current block from the bitstream, and determine the prediction mode of the current block based on the information about the prediction mode of the current block. Here, when the prediction mode of the current block is an inter-frame prediction mode, the video decoding apparatus 100 may obtain motion prediction mode information about the current block.

[0117] For example, the video decoding apparatus 100 may determine the prediction mode of the current block as the inter prediction mode based on the information about the prediction mode of the current block. When the prediction mode of the current block is the inter prediction mode, the video decoding apparatus 100 may obtain motion prediction mode information about the current block from the bitstream.

[0118] In operation S110, when the motion prediction mode information indicates a bidirectional motion prediction mode, the video decoding apparatus 100 may obtain a first motion vector indicating a first reference block of a current block in a first reference picture and a second motion vector indicating a second reference block of the current block in a second reference picture from a bitstream.

[0119] In other words, the video decoding apparatus 100 may obtain a bitstream including information about the first motion vector and the second motion vector, and obtain the first motion vector and the second motion vector from the received bitstream. The video decoding apparatus 100 may obtain a reference picture index from the bitstream, and determine the first reference picture and the second reference picture from among previously decoded pictures based on the reference picture index.

[0120] In operation S115, the video decoding device 100 can generate a pixel value of a first pixel from among the pixels of a first reference block indicated by a first motion vector and a pixel value of a second pixel from among the pixels of a second reference block indicated by a second motion vector by applying an interpolation filter to a first neighboring area of ​​the first pixel and a second neighboring area of ​​the second pixel.

[0121] In operation S120, the video decoding apparatus 100 may generate a gradient value of the first pixel and a gradient value of the second pixel by applying a filter to the first neighboring area and the second neighboring area. Here, the filter may be a 1D filter and may include a gradient filter and an interpolation filter.

[0122] In operation S125, the video decoding apparatus 100 may generate a predicted pixel value of the current block by using a pixel value of the first pixel, a pixel value of the second pixel, a gradient value of the first pixel, and a gradient value of the second pixel.

[0123] The video decoding apparatus 100 may obtain a residual signal about the current block from the bitstream, and reconstruct the current block by using the residual signal and the predicted pixel value of the current block. In other words, the video decoding apparatus 100 may generate a pixel value of a reconstructed block of the current block by adding the predicted pixel value of the current block and the residual pixel value of the current block indicated by the residual signal.

[0124] Figure 1c is a block diagram of a video encoding apparatus according to various embodiments.

[0125] The video encoding apparatus 150 according to various embodiments includes an inter-frame predictor 155 and a bitstream generator 170 .

[0126] The inter predictor 155 performs inter prediction on the current block. In other words, when the prediction mode of the current block is the inter prediction mode, the inter predictor 155 may generate a predicted pixel value of the current block by using at least one of pictures encoded before the current picture included in the current block.

[0127] The inter predictor 155 may include a block-unit motion compensator 160 and a pixel-unit motion compensator 165 .

[0128] The block-unit motion compensator 160 may generate a motion compensation value by block unit by performing motion compensation on the current block by block unit.

[0129] The block-unit motion compensator 160 may determine at least one reference picture from among previously encoded pictures, and determine a reference block of the current block located in the at least one reference picture.

[0130] The block unit motion compensator 160 may perform motion compensation on the current block on a block unit basis by using pixel values ​​of the reference block, thereby generating motion compensation values ​​on a block unit basis. The block unit motion compensator 160 may perform motion compensation on the current block on a block unit basis by using reference pixel values ​​of the reference block corresponding to the current pixel of the current block, thereby generating motion compensation values ​​on a block unit basis.

[0131] The block unit motion compensator 160 may perform motion compensation on the current block in block units by using a plurality of reference blocks respectively included in a plurality of reference pictures, thereby generating motion compensation values ​​in block units. For example, when the motion prediction mode of the current block is a bidirectional prediction mode, the block unit motion compensator 160 may determine two reference pictures from previously encoded pictures, and determine two reference blocks included in the two reference pictures. Here, bidirectional prediction not only means performing inter-frame prediction by using a picture displayed before the current picture and a picture displayed after the current picture, but may also mean performing inter-frame prediction by using two pictures encoded before the current picture, regardless of the order in which they are displayed.

[0132] The block-unit motion compensator 160 may perform motion compensation on the current block on a block-by-block basis by using pixel values ​​of two reference pixels in two reference blocks, thereby generating motion compensation values ​​on a block-by-block basis. The block-unit motion compensator 160 may perform motion compensation on the current block on a block-by-block basis by using an average pixel value or a weighted sum of the two reference pixels, thereby generating motion compensation values ​​on a block-by-block basis.

[0133] The block-unit motion compensator 160 may output a reference picture index indicating a reference picture used for motion compensation of a current block from among previously encoded pictures.

[0134] The block unit motion compensator 160 may determine a motion vector having a current block as a starting point and a reference block of the current block as an end point, and output the motion vector. The motion vector may represent a vector indicating the displacement of the reference coordinates of the current block in the current picture and the reference coordinates of the reference block in the reference picture. For example, when the coordinates of the upper left corner of the current block are (1,1) and the upper left coordinates of the reference block in the reference picture are (3,3), the motion vector may be (2,2).

[0135] The reference position of the reference block may be the position of an integer pixel, but alternatively, may be the position of a fractional pixel. Here, the position of the fractional pixel may be determined in units of 1 / 4 pel or 1 / 16 pel. Alternatively, the position of the fractional pixel may be determined in units of various fractional pels.

[0136] For example, when the reference position of the reference block is (1.5, 1.5) and the coordinates of the upper left corner of the current block are (1, 1), the motion vector may be (0.5, 0.5). When the motion vector is determined in 1 / 4 or 1 / 16 pel units to indicate the reference position of the reference block as the position of a fractional pixel, the motion vector of the integer is determined by scaling the motion vector, and the reference position of the reference block may be determined by using the enlarged motion vector. When the reference position of the reference block is the position of a fractional pixel, the position of the reference pixel of the reference block may also be the position of a fractional pixel. Therefore, the pixel value at the fractional pixel position in the reference block may be determined by using the pixel values ​​of neighboring pixels whose components in a specific direction are integers.

[0137] For example, the block unit motion compensator 160 may determine the value obtained by filtering the pixel values ​​of neighboring pixels whose components in a specific direction are integers as the pixel value of the reference pixel at the fractional pixel position, and determine the motion compensation value in block units about the current block by using the pixel value of the reference pixel. Here, the interpolation filter may be an M-tap interpolation filter based on DCT. The coefficients of the M-tap interpolation filter based on DCT may be summarized from DCT and IDCT. Here, the coefficients of the interpolation filter may be filter coefficients scaled to integer coefficients to reduce real number operations during filtering.

[0138] Here, the interpolation filter may be a 1D interpolation filter in the horizontal or vertical direction.

[0139] The block unit motion compensator 160 may first perform filtering for adjacent integer pixels by using a 1D interpolation filter in the vertical direction, and then perform filtering for the filtered values ​​by using a 1D interpolation filter in the horizontal direction to determine the pixel value of the reference pixel at the fractional pixel position. When using scaled filter coefficients, the block unit motion compensator 160 may perform descaling on the filtered values ​​after filtering the pixels at the integer positions by using a 1D interpolation filter in the vertical direction. Here, descaling may include shifting the descaling bit number to the right. The descaling bit number may be determined based on the bit depth of the sampling point. For example, the descaling bit number may be a value obtained by subtracting 8 from the bit depth of the sampling point.

[0140] In addition, the block unit motion compensator 160 may perform filtering on pixels whose horizontal direction components are integers by using a 1D interpolation filter in the vertical direction, and then perform right shifting to descaling the number of bits. The descaling number of bits may be determined based on the scaling number of bits of the 1D interpolation filter in the vertical direction, the scaling number of bits of the 1D interpolation filter in the horizontal direction, and the bit depth of the sampling point.

[0141] When the block unit motion compensator 160 performs only the right shift descaling bit number, a rounding error may be generated, and therefore the block unit motion compensator 160 may perform filtering on pixels whose components in a specific direction are integers by using a 1D interpolation filter in the specific direction, adding an offset value to the value to which filtering is performed, and then performing descaling on the value to which the offset value is added. Here, the offset value may be 2^(descaling bit number-1).

[0142] In the above, it has been described that the number of descaling bits is determined based on the bit depth of the sampling point after filtering using the 1D interpolation filter in the vertical direction, but alternatively, the number of descaling bits may be determined based not only on the bit depth of the sampling point but also on the number of bits scaled for the interpolation filter coefficients. In other words, while taking into account the size of the register used during filtering and the size of the buffer storing the values ​​generated during filtering, the number of descaling bits may be determined based on the bit depth of the sampling point and the number of bits scaled for the interpolation coefficients within a range where overflow does not occur.

[0143] The pixel unit motion compensator 165 may generate motion compensation values ​​by pixel unit by performing motion compensation on the current block by pixel unit. For example, when the motion prediction mode is a bidirectional motion prediction mode, the pixel unit motion compensator 165 may generate motion compensation values ​​by pixel unit by performing motion compensation on the current block by pixel unit.

[0144] The pixel-unit motion compensator 165 may perform motion compensation on the current block by pixel units by using gradient values ​​of pixels included in a reference block of the current block, thereby generating motion compensation values ​​by pixel units.

[0145] The pixel-unit motion compensator 165 may generate a gradient value of a first pixel and a gradient value of a second pixel by applying a filter to a first neighboring area of ​​the first pixel among pixels of a first reference block in a first reference picture and a second neighboring area of ​​the second pixel among pixels of a second reference block in a second reference picture.

[0146] The pixel-unit motion compensator 165 may determine pixel values ​​and gradient values ​​of pixels in a first window having a specific size around a first reference pixel in a first reference picture and including the first reference pixel, and determine pixel values ​​and gradient values ​​of pixels in a second window having a specific size around a second reference pixel in a second reference picture and including the second reference pixel. The pixel-unit motion compensator 165 may determine a displacement vector per unit time about a current pixel by using the pixel values ​​and gradient values ​​of the pixels in the first window and the pixel values ​​and gradient values ​​of the pixels in the second window.

[0147] The pixel-unit motion compensator 165 may perform motion compensation on the current block by pixel units by using a displacement vector per unit time and a gradient value of a reference pixel, thereby generating a motion compensation value by pixel units.

[0148] The position of the reference pixel may be the position of an integer pixel, but may alternatively be the position of a fractional pixel.

[0149] When the reference position of the reference block is the position of a fractional pixel, the gradient value of the reference pixel in the reference block may be determined by using pixel values ​​of neighboring pixels whose components in a specific direction are integers.

[0150] For example, the pixel-unit motion compensator 165 may determine the result value obtained by filtering the pixel values ​​of neighboring pixels whose components in a specific direction are integers using a gradient filter as the gradient value of the reference pixel. Here, the filter coefficient of the gradient filter may be determined by using a coefficient predetermined for a DCT-based interpolation filter.

[0151] The filter coefficients of the gradient filter may be filter coefficients scaled to integer coefficients to reduce real number operations during filtering. Here, the gradient filter may be a 1D gradient filter in the horizontal or vertical direction.

[0152] The pixel-unit motion compensator 165 may perform filtering on neighboring pixels whose components in a corresponding direction are integers by using a 1D gradient filter in the horizontal or vertical direction so as to determine a gradient value of a reference pixel in the horizontal or vertical direction.

[0153] For example, the pixel unit motion compensator 165 may determine the pixel values ​​of pixels whose vertical components are fractional values ​​by performing filtering on pixels whose vertical components are integers using a 1D interpolation filter in the vertical direction, wherein the pixels whose vertical components are integers are from among the pixels in the vertical direction from the integer pixels adjacent to the reference pixel.

[0154] Regarding the pixel located in another column adjacent to the integer pixel adjacent to the reference pixel, the pixel-unit motion compensator 165 may determine the pixel value of the fractional pixel position located in another column by performing filtering on the adjacent integer pixels in the vertical direction using a 1D interpolation filter in the vertical direction. Here, the position of the pixel located in another column may be the position of the fractional pixel in the vertical direction and the position of the integer pixel in the horizontal direction.

[0155] In other words, when the position of the reference pixel is (x+α, y+β) (where x and y are each integers and α and β are each fractions), the pixel unit motion compensator 165 can determine the pixel value at the position of (x, y+β) by using an interpolation filter in the vertical direction to perform filtering on adjacent integer pixels in the vertical direction starting from the position of (x, y).

[0156] The pixel unit motion compensator 165 can determine the gradient value at the position (x+α, y+β) in the horizontal direction by filtering the pixel value at the position (x, y+β) and the pixel values ​​of the pixels whose horizontal components are integers using a gradient filter in the horizontal direction, wherein the pixels whose horizontal components are integers are from among the pixels positioned in the horizontal direction from the pixel value at the position (x, y+β).

[0157] The order of using the 1D gradient filter and the 1D interpolation filter is not limited. As described above, the pixels at the integer positions may be first filtered by using the 1D interpolation filter in the vertical direction to generate the interpolation filter values ​​in the vertical direction, and then the interpolation filter values ​​in the vertical direction may be filtered by using the 1D gradient filter in the horizontal direction, but alternatively, the pixels at the integer positions may be first filtered by using the 1D gradient filter in the horizontal direction to generate the gradient filter values ​​in the horizontal direction, and then the gradient filter values ​​in the horizontal direction may be filtered by using the 1D interpolation filter in the vertical direction.

[0158] Hereinabove, it has been described in detail that the pixel-unit motion compensator 165 determines the gradient value in the horizontal direction at the position (x+α, y+β).

[0159] The pixel-unit motion compensator 165 may determine the gradient value in the vertical direction at the position of (x+α, y+β) in a similar manner as determining the gradient value in the horizontal direction.

[0160] The pixel unit motion compensator 165 may perform filtering on the adjacent integer pixels in the vertical direction from the integer pixel adjacent to the reference pixel by using a 1D gradient filter in the vertical direction, thereby determining the gradient value of the reference pixel in the vertical direction. In addition, regarding the pixel adjacent to the reference pixel and positioned in another column, the pixel unit motion compensator 165 may perform filtering on the adjacent integer pixels in the vertical direction by using a 1D gradient filter in the vertical direction, thereby determining the gradient value in the vertical direction for the pixel adjacent to the reference pixel and positioned in another column. Here, the position of the pixel may be the position of the fractional pixel in the vertical direction and the position of the integer pixel in the horizontal direction.

[0161] In other words, when the position of the reference pixel is (x+α, y+β) (where x and y are each integers and α and β are each fractions), the pixel-unit motion compensator 165 may determine the gradient value in the vertical direction at the position (x, y+β) by performing filtering on adjacent integer pixels in the vertical direction starting from the position (x, y) using a gradient filter in the vertical direction.

[0162] The pixel-unit motion compensator 165 may determine a gradient value in the vertical direction at the position of (x+α, y+β) by filtering the gradient value at the position of (x, y+β) and the gradient values ​​of neighboring integer pixels located in the horizontal direction from the position of (x, y+β) using an interpolation filter in the horizontal direction.

[0163] The order of using the 1D gradient filter and the 1D interpolation filter is not limited. As described above, the pixels at the integer positions may be first filtered by using the 1D gradient filter in the vertical direction to generate the gradient filter values ​​in the vertical direction, and then the gradient filter values ​​in the vertical direction may be filtered by using the 1D interpolation filter in the horizontal direction, but alternatively, the pixels at the integer positions may be first filtered by using the 1D interpolation filter in the horizontal direction to generate the interpolation filter values ​​in the horizontal direction, and then the interpolation filter values ​​in the horizontal direction may be filtered by using the 1D gradient filter in the vertical direction.

[0164] In the above, it has been described in detail that the pixel-unit motion compensator 165 uses a gradient filter and an interpolation filter in order to determine a gradient value at a fractional pixel position. However, alternatively, a gradient filter and an interpolation filter may be used to determine a gradient value at an integer pixel position.

[0165] In the case of integer pixels, the pixel value may be determined without using an interpolation filter, but for a process consistent with the process in fractional pixels, filtering may be performed on the integer pixels and the neighboring integer pixels by using an interpolation filter. For example, the interpolation filter coefficient in the integer pixel may be {0, 0, 64, 0, 0}, and since the interpolation filter coefficient multiplied by the neighboring integer pixel is 0, filtering may be performed by using only the pixel value of the current integer pixel, and as a result, the pixel value of the current integer pixel may be similarly determined as a value generated by performing filtering on the current integer pixel and the neighboring integer pixels by using the interpolation filter.

[0166] Meanwhile, when using the scaled filter coefficient, the pixel unit motion compensator 165 may perform filtering on the pixels at the integer positions by using a 1D gradient filter in the horizontal direction, and then descaling the filtered values. Here, descaling may include shifting the descaling bit number to the right. The descaling bit number may be determined based on the bit depth of the sampling point. For example, the descaling bit number may be a value obtained by subtracting 8 from the bit depth of the sampling point.

[0167] The pixel unit motion compensator 165 may perform filtering on pixels whose components in the vertical direction are integers by using an interpolation filter in the vertical direction, and then perform descaling. Here, descaling may include shifting the descaling bit number to the right. The descaling bit number may be determined based on the scaling bit number of the 1D interpolation filter in the vertical direction, the scaling bit number of the 1D gradient filter in the horizontal direction, and the bit depth of the sampling point.

[0168] When the pixel-unit motion compensator 165 performs only the right shift descaling bit number, a rounding error may be generated. Therefore, the pixel-unit motion compensator 165 may perform filtering by using a 1D interpolation filter, and add an offset value to the value to which filtering is performed, and then perform descaling on the value to which the offset value is added. Here, the offset value may be 2^(descaling bit number-1).

[0169] The inter-frame predictor 155 may generate a predicted pixel value of the current block by using a motion compensation value in block units and a motion compensation value in pixel units with respect to the current block. For example, the inter-frame predictor 155 may generate a predicted pixel value of the current block by adding a motion compensation value in block units and a motion compensation value in pixel units with respect to the current block. In particular, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the inter-frame predictor 155 may generate a predicted pixel value of the current block by using a motion compensation value in block units and a motion compensation value in pixel units with respect to the current block.

[0170] When the motion prediction mode of the current block is a unidirectional motion prediction mode, the inter-frame predictor 155 can generate a predicted pixel value of the current block by using a motion compensation value in block units with respect to the current block. Here, unidirectional means using one reference picture from among previously encoded pictures. The one reference picture may be a picture displayed before the current picture, but alternatively, may be a picture displayed after the current picture.

[0171] The inter-frame predictor 155 may determine the motion prediction mode of the current block and output information indicating the motion prediction mode of the current block. For example, the inter-frame predictor 155 may determine the motion prediction mode of the current block as a bidirectional motion prediction mode and output information indicating the bidirectional motion prediction mode. Here, the bidirectional motion prediction mode refers to a mode of predicting motion by using reference blocks in two decoded reference pictures.

[0172] The bitstream generator 170 may generate a bitstream including a motion vector indicating a reference block. The bitstream generator 170 may encode the motion vector indicating the reference block and generate a bitstream including the encoded motion vector. The bitstream generator 170 may encode the differential value of the motion vector indicating the reference block and generate a bitstream including the encoded differential value of the motion vector. Here, the differential value of the motion vector may represent the difference between the motion vector and the predictor of the motion vector. Here, the differential value of the motion vector may represent the differential value of the motion vector of the reference picture related to the prediction directions including the L0 direction and the L1 direction, respectively. Here, the differential value of the motion vector of the L0 direction may represent the differential value of the motion vector of the reference picture in the reference picture included in the L0 reference picture list, and the differential value of the motion vector of the L1 direction may represent the differential value of the motion vector of the reference picture in the reference picture included in the L1 reference picture list.

[0173] In addition, the bitstream generator 170 may generate a bitstream further including information indicating a motion prediction mode of the current block. The bitstream generator 170 may encode a reference picture index indicating a reference picture of the current block from among previously encoded pictures, and generate a bitstream including the encoded reference picture index. Here, the reference picture index may represent a reference picture index for each of prediction directions including an L0 direction and an L1 direction. Here, the reference picture index for the L0 direction may represent an index indicating a reference picture among pictures included in the L0 reference picture list, and the reference picture index for the L1 direction may represent an index indicating a reference picture among pictures included in the L1 reference picture list.

[0174] The video encoding apparatus 150 may include an image encoder (not shown), and the image encoder may include an inter-frame predictor 155 and a bitstream generator 170. Figure 1f Describes a video encoder.

[0175] Figure 1d is a flowchart of a video encoding method according to various embodiments.

[0176] refer to Figure 1dIn operation S150, the video encoding device 150 may determine a first reference block of a first reference picture corresponding to a current block in a current picture and a second reference block of a second reference picture corresponding to the current block, and determine a first motion vector indicating the first reference block and a second motion vector indicating the second reference block.

[0177] In operation S155, the video encoding apparatus 150 may generate a pixel value of a first pixel and a pixel value of a second pixel by applying an interpolation filter to a first neighboring area of ​​the first pixel among pixels of the first reference block and a second neighboring area of ​​the second pixel among pixels of the second reference block.

[0178] In operation S160, the video encoding device 150 may generate a gradient value of a first pixel and a gradient value of a second pixel by applying a filter to a first neighboring area of ​​a first pixel among pixels of a first reference block and a second neighboring area of ​​a second pixel among pixels of a second reference block, and generate a predicted pixel value of a current block by using the pixel value of the first pixel, the pixel value of the second pixel, the gradient value of the first pixel, and the gradient value of the second pixel.

[0179] In operation S165, the video encoding apparatus 150 may generate a bitstream including information about the first motion vector and the second motion vector, and motion prediction mode information indicating that the motion prediction mode of the current block is the bi-directional motion prediction mode.

[0180] The video encoding device 150 may encode a residual signal of the current block, the residual signal indicating the difference between the predicted pixel value of the current block and the original block of the current block, and generate a bitstream further including the encoded residual signal. The video encoding device 150 may encode information about the prediction mode and the reference picture index of the current block, and generate a bitstream further including the encoded information about the prediction mode and the encoded reference picture index. For example, the video encoding device 150 may encode information indicating that the prediction mode of the current block is an inter-frame prediction mode and a reference picture index indicating at least one picture from among previously decoded pictures, and generate a bitstream further including the encoded information about the prediction mode and the encoded reference picture index.

[0181] Figure 1e is a block diagram of an image decoder 600 according to various embodiments.

[0182] The image decoder 600 according to various embodiments performs operations performed by the image decoder (not shown) of the video decoding apparatus 100 to decode image data.

[0183] refer to Figure 1e, the entropy decoder 615 parses the encoded image data to be decoded and the encoding information required for decoding from the bit stream 605. The encoded image data is a quantized transform coefficient, and the inverse quantizer 620 and the inverse transformer 625 reconstruct residual data from the quantized transform coefficient.

[0184] The intra predictor 640 performs intra prediction per block. The inter predictor 635 performs inter prediction by using a reference image obtained from the reconstructed picture buffer 630 per block. Figure 1e The inter-frame predictor 635 may correspond to Figure 1a The inter-frame predictor 110 is configured as follows.

[0185] The data in the spatial domain of the block of the current image can be reconstructed by adding the prediction data and the residual data of each block generated by the intra-frame predictor 640 or the inter-frame predictor 635, and the deblocking unit 645 and the SAO performer 650 can output a filtered reconstructed image by performing loop filtering on the reconstructed data in the spatial domain. In addition, the reconstructed image stored in the reconstructed picture buffer 630 can be output as a reference image.

[0186] In order for a decoder (not shown) of the video decoding apparatus 100 to decode image data, stepwise operations of the image decoder 600 according to various embodiments may be performed in blocks.

[0187] Figure 1f is a block diagram of an image encoder according to various embodiments.

[0188] The image encoder 700 according to various embodiments performs operations performed by the image encoder (not shown) of the video encoding apparatus 150 to encode image data.

[0189] In other words, the intra predictor 720 performs intra prediction on the current image 705 by block, and the inter predictor 715 performs inter prediction by using the current image 705 per block and a reference image obtained from the reconstructed picture buffer 710. Here, Figure 1e The inter-frame predictor 715 may correspond to Figure 1c The inter-frame predictor 155 is provided.

[0190] Residual data may be generated by subtracting prediction data about each block output from the intra predictor 720 or the inter predictor 715 from data about the coding block of the current image 705, and the transformer 725 and the quantizer 730 may output block-quantized transform coefficients by performing transform and quantization on the residual data. The dequantizer 745 and the inverse transformer 750 may reconstruct the residual data in the spatial domain by performing dequantization and inverse transform on the quantized transform coefficients. The reconstructed residual data in the spatial domain may be added to the prediction data about each block output from the intra predictor 720 or the inter predictor 715 to be reconstructed into data in the spatial domain of the block about the current image 705. The deblocking unit 755 and the SAO executor 760 generate a filtered reconstructed image by performing in-loop filtering on the reconstructed data in the spatial domain. The generated reconstructed image is stored in the reconstructed picture buffer 710. The reconstructed image stored in the reconstructed picture buffer 710 may be used as a reference image for inter prediction of another image. The entropy encoder 735 may entropy encode the quantized transform coefficients, and the entropy-encoded coefficients may be output as a bitstream 740 .

[0191] In order for the image encoder 700 according to various embodiments to be applied to the video encoding apparatus 150 , stepwise operations of the image encoder 700 according to various embodiments may be performed in blocks.

[0192] Figure 2 is a reference diagram for describing a block-based bidirectional motion prediction and compensation process according to an embodiment.

[0193] refer to Figure 2 , the video encoding apparatus 150 performs bidirectional motion prediction, in which a region most similar to the current block 201 of the current picture 200 to be encoded is searched in the first reference picture 210 and the second reference picture 220. Here, the first reference picture 210 may be a picture before the current picture 200, and the second reference picture 220 may be a picture after the current picture 200. As a result of the bidirectional motion prediction, the video encoding apparatus 150 determines a first corresponding region 212 most similar to the current block 201 from the first reference picture 210, and determines a second corresponding region 222 most similar to the current block 201 from the second reference picture 220. Here, the first corresponding region 212 and the second corresponding region 222 may be reference regions of the current block 201.

[0194] In addition, the video encoding device 150 can determine the first motion vector MV1 based on the position difference between the first corresponding area 212 and the block 211 of the first reference picture 210 at the same position as the current block 201, and determine the second motion vector MV2 based on the position difference between the second corresponding area 222 and the block 221 of the second reference picture 220 at the same position as the current block 201.

[0195] The video encoding apparatus 150 performs block-unit bidirectional motion compensation on the current block 201 by using the first motion vector MV1 and the second motion vector MV2.

[0196] For example, when the pixel value located at (i, j) of the first reference picture 210 is P0(i, j), the pixel value located at (i, j) of the second reference picture 220 is P1(i, j), MV1=(MVx1, MVy1) and MV2=(MVx2, MVy2) (where i and j are each integers), the block-unit bidirectional motion compensation value P_BiPredBlock(i, j) of the pixel at the (i, j) position of the current block 201 can be calculated according to the equation: P_BiPredBlock(i, j)={P0(i+MVx1, j+MVy1)+P1(i+MVx2, j+MVy2)} / 2. Therefore, the video encoding device 150 can perform motion compensation on the current block 201 by block units by using the average value or weighted sum of the pixels in the first corresponding area 212 and the second corresponding area 222 indicated by the first motion vector MV1 and the second motion vector MV2, thereby generating a motion compensation value by block units.

[0197] Figure 3a to Figure 3c is a reference diagram for describing a process of performing pixel-unit motion compensation according to an embodiment.

[0198] exist Figure 3a In the example, the first corresponding area 310 and the second corresponding area 320 correspond to Figure 2 The first corresponding region 212 and the second corresponding region 222 of the current block 300 may have been shifted to overlap with the current block 300 by using the bidirectional motion vectors MV1 and MV2.

[0199] In addition, P(i,j) represents a pixel of the current block 300 at the (i,j) position of the bidirectional prediction, P0(i,j) represents a first reference pixel value of the first reference picture corresponding to the pixel P(i,j) of the bidirectionally predicted current block 300, and P1(i,j) represents a second reference pixel value of the second reference picture corresponding to the pixel P(i,j) of the bidirectionally predicted current block 300, where i and j each represent an integer.

[0200] In other words, the first reference pixel value P0(i,j) is the pixel value of the pixel corresponding to the pixel P(i,j) of the current block 300 determined by the bidirectional motion vector MV1 indicating the first reference picture, and the second reference pixel value P1(i,j) is the pixel value of the pixel corresponding to the pixel P(i,j) of the current block 300 determined by the bidirectional motion vector MV2 indicating the second reference picture.

[0201] in addition, represents the gradient value of the first reference pixel in the horizontal direction, represents the gradient value of the first reference pixel in the vertical direction, represents the gradient value of the second reference pixel in the horizontal direction, and Represents the gradient value of the second reference pixel in the vertical direction.

[0202] In addition, τ 0 represents the temporal distance between the current picture to which the current block 300 belongs and the first reference picture to which the first corresponding region 310 belongs, and τ 1 represents a temporal distance between the current picture and the second reference picture to which the second corresponding region 320 belongs. Here, the temporal distance between pictures may represent a difference in picture order counts (POCs) of the pictures.

[0203] When there is a uniform small motion in the video sequence, the pixel most similar to the pixel P(i, j) for which bidirectional motion compensation is performed in pixel units in the first corresponding area 310 of the first reference picture is not the first reference pixel P0(i, j), but the first displacement reference pixel PA, where the first reference pixel P0(i, j) is shifted by a specific displacement vector. As described above, because there is a uniform motion in the video sequence, the pixel most similar to the pixel P(i, j) in the second corresponding area 320 of the second reference picture may be the second displacement reference pixel PB, where the second reference pixel P1(i, j) is shifted by a specific displacement vector.

[0204] The displacement vector may include a displacement vector Vx in the x-axis direction and a displacement vector Vy in the y-axis direction. Therefore, the pixel unit motion compensator 165 calculates the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction included in the displacement vector, and performs motion compensation in pixel units by using the displacement vector.

[0205] Optical flow represents the pattern of apparent motion on an object or surface caused by the relative motion between the scene and the observer (eye or video image acquisition device like a camera). In a video sequence, optical flow can be represented by calculating the motion between frames obtained at arbitrary times t and t+Δt. The pixel value located at (x, y) in the frame at time t can be I(x, y, t). In other words, I(x, y, t) can be a value that changes in time and space. I(x, y, t) can be differentiated according to Equation 1 for time t.

[0206] [Formula 1]

[0207] When the pixel value changes according to motion but does not change according to time relative to a small motion in a block, dI / dt is 0. In addition, when the motion of the pixel value according to time is uniform, dx / dt can represent a displacement vector Vx of the pixel value I (x, y, t) in the x-axis direction and dy / dt can represent a displacement vector Vy of the pixel value I (x, y, t) in the y-axis direction, and therefore, Formula 1 can be expressed as Formula 2.

[0208] [Formula 2]

[0209]

[0210] Here, the size of the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction may have a value smaller than the pixel accuracy used in the bidirectional motion prediction. For example, when the pixel accuracy is 1 / 4 or 1 / 16 during the bidirectional motion prediction, the size of the displacement vectors Vx and Vy may have a value smaller than 1 / 4 or 1 / 16.

[0211] The pixel unit motion compensator 165 calculates the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction according to Formula 2, and performs motion compensation in pixel units by using these displacement vectors Vx and Vy. In Formula 2, since the pixel value I (x, y, t) is the value of the original signal, high overhead may be induced during encoding when the value of the original signal is used. Therefore, the pixel unit motion compensator 165 may calculate the displacement vectors Vx and Vy according to Formula 2 by using the pixels of the first reference picture and the second reference picture determined as a result of performing bidirectional motion prediction in block units. In other words, the pixel unit motion compensator 165 determines the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction, where Δ is the minimum value in a window Ωij having a specific size and including neighboring pixels around the pixel P (i, j) on which bidirectional motion compensation is performed. Δ may be 0, but there may not be a displacement vector Vx in the x-axis direction and a displacement vector Vy in the y-axis direction that satisfies Δ=0 for all pixels in the window Ωij, and thus the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction where Δ is the minimum value are determined. Figure 8 The process of obtaining the displacement vectors Vx and Vy is described in detail.

[0212] In order to determine the predicted pixel value of the current pixel, the function P(t) for t may be determined according to Formula 3.

[0213] [Formula 3]

[0214] P(t)=a3*t 3 +a2*t 2 +a1*t+a0

[0215] Here, the picture when t=0 is the current picture including the current block. Therefore, the predicted pixel value of the current pixel included in the current block can be defined as the value of P(t) when t is 0.

[0216] When the temporal distance between the current picture and the first reference picture (the first reference picture is before the current picture in time) is τ 0 And the temporal distance between the current picture and the second reference picture (the second reference picture is temporally after the current picture) is τ 1 When the reference pixel value in the first reference picture is equal to P(-τ 0 ), and the reference pixel value in the second reference picture is equal to P(τ 1 ). In the following, for the convenience of calculation, it is assumed that τ 0 and τ 1 are all equal to τ.

[0217] The coefficient of each degree of P(t) may be determined according to Equation 4. Here, P0(i,j) may represent a pixel value at a position (i,j) of a first reference picture, and P1(i,j) may represent a pixel value at a position (i,j) of a second reference picture.

[0218] [Formula 4]

[0219]

[0220]

[0221]

[0222]

[0223] Therefore, the predicted pixel value P(0) of the current pixel in the current block can be determined according to Equation 5.

[0224] [Formula 5]

[0225]

[0226] Equation 5 can be expressed as Equation 6 by considering Equation 2.

[0227] [Formula 6]

[0228]

[0229] Therefore, the predicted pixel value of the current pixel can be determined by using the displacement vector Vx, the displacement vector Vy, the gradient values ​​of the first reference pixel in the horizontal and vertical directions, and the gradient values ​​of the second reference pixel in the horizontal and vertical directions. Here, a portion (P0(i,j)+P1(i,j)) / 2) that is not related to the displacement vectors Vx and Vy may be a motion compensation value in block units, and a portion that is related to the displacement vectors Vx and Vy may be a motion compensation value in pixel units. As a result, the predicted pixel value of the current pixel can be determined by adding the motion compensation value in block units and the motion compensation value in pixel units.

[0230] In the above, the process of determining the predicted pixel value of the current pixel when the temporal distance between the first reference picture and the current picture and the temporal distance between the second reference picture and the current picture are both τ is described, and thus the same is described for the convenience of description, but the temporal distance between the first reference picture and the current picture may be τ 0 And the temporal distance between the second reference picture and the current picture can be τ 1 Here, the predicted pixel value P(0) of the current pixel can be determined according to Formula 7.

[0231] [Formula 7]

[0232]

[0233] Considering Equation 2, Equation 7 can be expressed as Equation 8.

[0234] [Formula 8]

[0235]

[0236] In the above, the first reference picture is displayed temporally after the current picture and the second reference picture is displayed temporally before the current picture, but alternatively, both the first reference picture and the second reference picture may be displayed temporally before or after the current picture.

[0237] For example, Figure 3b As shown in , the first reference picture including the first corresponding area 310 and the second reference picture including the second corresponding area 320 may both be displayed temporally before the current picture including the current block 300 .

[0238] In this case, the predicted pixel value P(0) of the current pixel can be determined according to Formula 9, where Figure 3a In the formula 8 indicated by 1 -τ 1 replace.

[0239] [Formula 9]

[0240]

[0241] For example, Figure 3c As shown in , the first reference picture including the first corresponding area 310 and the second reference picture including the second corresponding area 320 may both be displayed temporally after the current picture including the current block 300 .

[0242] In this case, the predicted pixel value P(0) of the current pixel can be determined according to Formula 10, where Figure 3a In the formula 8 indicated by 0 -τ 0 replace.

[0243] [Formula 10]

[0244]

[0245] However, when Figure 3b and Figure 3c When both the first reference picture and the second reference picture shown in are displayed before or after the current picture in time, motion compensation in pixel units can be performed when the first reference picture and the second reference picture are not the same reference picture. In addition, in this case, motion compensation in pixel units can be performed only when both bidirectional motion vectors MV1 and MV2 have non-zero components. In addition, in this case, motion compensation in pixel units can be performed only when the ratio of motion vectors MV1 and MV2 is the same as such a ratio, wherein the ratio is the ratio of the temporal distance between the first reference picture and the current picture to the temporal distance between the second reference picture and the current picture. For example, when the ratio of the x component of motion vector MV1 to the x component of motion vector MV2 is the same as the ratio of the y component of motion vector MV1 to the y component of motion vector MV2 and the temporal distance τ between the first reference picture and the current picture 0 The time distance τ between the second reference picture and the current picture 1 When the ratio of is the same, motion compensation in pixel units can be performed.

[0246] Figure 4 is a reference diagram for describing a process of calculating gradient values ​​in horizontal and vertical directions according to an embodiment.

[0247] refer to Figure 4, the gradient value of the first reference pixel P0(i,j)410 in the horizontal direction can be calculated by obtaining the change in the horizontal direction of the pixel values ​​at the neighboring fractional pixel positions adjacent to the first reference pixel P0(i,j)410 of the first reference picture and the change in the vertical direction of the pixel values ​​at the neighboring fractional pixel positions adjacent to the first reference pixel P0(i,j)410. and the gradient value of the first reference pixel P0(i,j)410 in the vertical direction In other words, according to Formula 11, the gradient value in the horizontal direction can be calculated by calculating the change in the pixel value of the fractional pixel P0(ih,j)460 and the fractional pixel P0(i+h,j)470 which are h away from P0(i,j) in the horizontal direction. Where h is a fraction less than 1, and the gradient value in the vertical direction can be calculated by calculating the change in pixel value of the fractional pixel P0(i,jh)480 and the fractional pixel P0(i,j+h)490 which are h away from P0(i,j) in the vertical direction.

[0248] [Formula 11]

[0249]

[0250]

[0251] The pixel values ​​of the fractional pixels P0(ih,j) 460, P0(i+h,j) 470, P0(i,jh) 480, and P0(i,j+h) 490 may be calculated by using general interpolation. In addition, the gradient values ​​of the second reference pixel of the second reference picture in the horizontal and vertical directions may be calculated similarly to Equation 11.

[0252] According to an embodiment, instead of calculating the gradient value by calculating the change of the pixel value at the fractional pixel position as in Equation 11, the gradient value in the reference pixel may be calculated by using a specific filter. The filter coefficient of the specific filter may be determined based on the coefficient of the interpolation filter for obtaining the pixel value at the fractional pixel position considering the linearity of the filter.

[0253] Figure 5 is a reference diagram for describing a process of calculating gradient values ​​in horizontal and vertical directions according to another embodiment.

[0254] According to another embodiment, the gradient value may be determined by applying a specific filter to the pixels of the reference picture. Figure 5 The video decoding device 100 can obtain the current horizontal gradient value based on the reference pixel P0 500 of the M Max left pixel 520 and |M Min| A specific filter is applied to the right pixel 510 to calculate the gradient value of the reference pixel P0 500 in the horizontal direction. Max and M Min The value α of the interpolation position (fractional pel position) between integer pixels determines the filter coefficient used here, such as Figures 7a to 7d For example, refer to Figure 7a , when M is used to determine the window size Min and M Max are -2 and 3 respectively and are 1 / 4 away from the reference pixel P0 500, i.e. α=1 / 4, Figure 7a The filter coefficients {4, -17, -36, 60, -15, 4} in the second row of -2 , P -1 , P 0 , P 1 , P 2 and P 3 In this case, the gradient value of the reference pixel P0 500 in the horizontal direction may be calculated by using the filter coefficient and the weighted sum of the neighboring pixels. Such as equation: Similarly, the interpolation position and M used to determine the window size can also be used to determine the window size. Min and M Max Apply to neighboring pixels Figures 7a to 7e The filter coefficients shown in are used to calculate the gradient value in the vertical direction.

[0255] Figure 6a and Figure 6b is a diagram for describing a process of determining gradient values ​​in horizontal and vertical directions by using a 1D filter according to an embodiment.

[0256] refer to Figure 6a , filtering may be performed by using a plurality of 1D filters for integer pixels in order to determine the gradient value in the horizontal direction of the reference pixel in the reference picture. Motion compensation in pixel units is an additional motion compensation performed after motion compensation in block units. Therefore, the reference position of the reference block of the current block indicated by the motion vector during motion compensation in block units may be a fractional pixel position, and motion compensation in pixel units may be performed for the reference pixel in the reference block at the fractional pixel position. Therefore, filtering may be performed in consideration of the fact that the gradient value of the pixel at the fractional pixel position is determined.

[0257] refer to Figure 6aFirst, the video decoding apparatus 100 may perform filtering on pixels located in a horizontal or vertical direction from neighboring integer pixels of a reference pixel in a reference picture by using a first 1D filter. Similarly, the video decoding apparatus 100 may perform filtering on neighboring integer pixels in a row or column different from the reference pixel by using a first 1D filter. The video decoding apparatus 100 may perform filtering on a value generated by filtering by using a second 1D filter, thereby generating a gradient value of the reference pixel in a horizontal direction.

[0258] For example, when the position of the reference pixel is the position of a fractional pixel at (x+α, y+β), where x and y are each integers and α and β are each fractions, the reference pixel can be obtained by calculating the position of the fractional pixel at (x+α, y+β) for integer pixels (x, y), (x-1, y), (x+1, y) to (x+M Min ,y) and (x+M Max ,y) Use a 1D vertical interpolation filter to perform filtering according to Equation 12, where M Min and M Mmax Each is an integer.

[0259] [Formula 12]

[0260]

[0261] Here, fracFilter β may represent an interpolation filter for determining a pixel value at a fractional pixel position β in the vertical direction, and fracFilter β [j'] may represent the coefficients of the interpolation filter applied to the pixel at (i, j'). I[i, j'] may represent the pixel value at the position (i, j').

[0262] In other words, the first 1D filter may be an interpolation filter for determining fractional pixel values ​​in the vertical direction. 1 Can represent an offset to prevent rounding errors, and shift 1 Temp[i,j+β] may represent the number of descaling bits. Temp[i,j+β] may represent the pixel value at the fractional pixel position (i,j+β). Temp[i',j+β] may also be determined according to Equation 12 by replacing i with i', where i' is the value of i+M excluding i. min An integer up to i+M.

[0263] Then, the video decoding apparatus 100 may perform filtering on the pixel value at the fractional pixel position (i, j+β) and the pixel value at the fractional pixel position (i′, j+β) by using the second 1D filter.

[0264] [Formula 13]

[0265]

[0266] Here, gradFilter α It can be a gradient filter for determining the gradient value at the fractional pixel position α in the horizontal direction. α [i'] may represent a coefficient of an interpolation filter applied to a pixel at the position (i', j+β). In other words, the second 1D filter may be a gradient filter for determining a gradient value in the horizontal direction. 2 Can represent an offset to prevent rounding errors, and shift 2 The number of descaling bits can be indicated.

[0267] In other words, according to Formula 13, the video decoding device 100 can use the gradient filter gradFilter α To perform filtering on the pixel value (Temp[i,j+β]) at the pixel position (i,j+β) and the pixel value (Temp[i',j+β]) located in the vertical direction from the pixel position (i,j+β) to determine the gradient value in the horizontal direction at (i+α,j+β)

[0268] In the above, the gradient value in the horizontal direction is determined by first applying an interpolation filter and then applying a gradient filter, but alternatively, the gradient value in the horizontal direction may be determined by first applying a gradient filter and then applying an interpolation filter. Hereinafter, an embodiment of determining the gradient value in the horizontal direction by applying a gradient filter and then applying an interpolation filter will be described.

[0269] For example, when the position of the reference pixel is a fractional pixel position at (x+α, y+β), where x and y are each integers and α and β are each fractions, the reference pixel position can be obtained by calculating the position of the reference pixel for integer pixels (x, y), (x-1, y), (x+1, y) to (x+M Min ,y) and (x+M Max ,y) Use the first 1D filter to perform filtering according to equation 14, where M Min and M Mmax Each is an integer.

[0270] [Formula 14]

[0271]

[0272] Here, gradFilter α can represent a gradient filter for determining the gradient value at a fractional pixel position α in the horizontal direction, and gradFilterα I[i'] may represent the coefficient of the gradient filter applied to the pixel at the position (i', j). I[i', j] may represent the pixel value at the position (i', j).

[0273] In other words, the first 1D filter may be an interpolation filter for determining a gradient value of a pixel in a horizontal direction, wherein the horizontal component of the pixel position is a fractional position. 3 Can represent an offset to prevent rounding errors, and shift 3 It can represent the number of descaling bits. Temp[i+α,j] can represent the gradient value at the pixel position (i+α,j) in the horizontal direction. Temp[i+α,j'] can also be determined according to Formula 14 by replacing j with j', where j' is the value of j+M excluding j. min to j+M max An integer.

[0274] Then, the video decoding apparatus 100 may perform filtering on the gradient value at the pixel position (i+α, j) in the horizontal direction and the gradient value at the pixel position (i+α, j′) in the horizontal direction by using a second 1D filter according to Equation 15.

[0275] [Formula 15]

[0276]

[0277] Here, fracFilter β fracFilter may be an interpolation filter for determining the pixel value at the fractional pixel position β in the vertical direction. β [j'] may represent a coefficient of an interpolation filter applied to a pixel at the position (i+β, j'). In other words, the second 1D filter may be an interpolation filter for determining a pixel value at a fractional pixel position β in the vertical direction. 4 Can represent an offset to prevent rounding errors, and shift 4 The number of descaling bits can be indicated.

[0278] In other words, according to Formula 15, the video decoding apparatus 100 can decode the image by using the gradient filter fracFilter β The gradient value (Temp[i+α,j]) at the pixel position (i+α,j) in the horizontal direction and the gradient value (Temp[i+α,j']) in the horizontal direction of the pixel located in the vertical direction from the pixel position (i+α,j) are filtered to determine the gradient value in the horizontal direction at (i+α,j+β)

[0279] refer to Figure 6b , filtering may be performed by using a plurality of 1D filters for integer pixels in order to determine the gradient value of the reference pixel in the vertical direction in the reference picture. Motion compensation in pixel units is an additional motion compensation performed after motion compensation in block units. Therefore, the reference position of the reference block of the current block indicated by the motion vector during motion compensation in block units may be a fractional pixel position, and motion compensation in pixel units may be performed for the reference pixel in the reference block at the fractional pixel position. Therefore, filtering may be performed in consideration of the fact that the gradient value of the pixel at the fractional pixel position is determined.

[0280] refer to Figure 6b First, the video decoding apparatus 100 may perform filtering on pixels located in a horizontal or vertical direction from neighboring integer pixels of a reference pixel in a reference picture by using a first 1D filter. Similarly, the video decoding apparatus 100 may perform filtering on neighboring integer pixels in a row or column different from the reference pixel by using a first 1D filter. The video decoding apparatus 100 may perform filtering on a value generated by filtering by using a second 1D filter, thereby generating a gradient value of the reference pixel in a vertical direction.

[0281] For example, when the position of the reference pixel is a fractional pixel position at (x+α, y+β), where x and y are each integers and α and β are each fractions, the reference pixel position can be obtained by calculating the position of the reference pixel for integer pixels (x, y), (x-1, y-1), (x+1, y+1) to (x+M Min ,y+M min ) and (x+M Max ,y+M max ) uses a first 1D filter to perform filtering according to equation 16, where M Min and M Mmax Each is an integer.

[0282] [Formula 16]

[0283]

[0284] Here, fracFilter α may represent an interpolation filter for determining a pixel value at a fractional pixel position α in the horizontal direction, and fracFilter α I[i'] may represent the coefficient of the interpolation filter applied to the pixel at the position (i', j). I[i', j] may represent the pixel value at the position (i', j).

[0285] In other words, the first 1D filter may be an interpolation filter for determining a pixel value at a fractional pixel position α in the horizontal direction. 5Can represent an offset to prevent rounding errors, and shift 5 The number of descaling bits can be indicated.

[0286] Temp[i+α,j] can represent the pixel value at the fractional pixel position (i+α,j). Temp[i+α,j'] can also be determined according to Equation 16 by replacing j with j', where j' is the value of j+M excluding j. min to j+M max An integer.

[0287] Then, the video decoding apparatus 100 may perform filtering on the pixel value at the pixel position (i+α, j) and the pixel value at the pixel position (i+α, j′) according to Equation 17 by using the second 1D filter.

[0288] [Formula 17]

[0289]

[0290] Here, gradFilter β It can be a gradient filter for determining the gradient value at the fractional pixel position β in the vertical direction. β [j'] may represent a coefficient of an interpolation filter applied to a pixel at the position (i+α, j'). In other words, the second 1D filter may be a gradient filter for determining a gradient value in a vertical direction at the fractional pixel position β. 6 Can represent an offset to prevent rounding errors, and shift 6 The number of descaling bits can be indicated.

[0291] In other words, according to Formula 17, the video decoding device 100 can be decoded by using the gradient filter gradFilter β To perform filtering on the pixel value (Temp[i+α,j]) at the pixel position (i+α,j) and the pixel value (Temp[i+α,j']) located in the vertical direction from the pixel position (i+α,j) to determine the gradient value in the vertical direction at (i+α,j+β)

[0292] In the above, the gradient value in the vertical direction is determined by first applying an interpolation filter and then applying a gradient filter, but alternatively, the gradient value in the vertical direction may be determined by first applying a gradient filter and then applying an interpolation filter. Hereinafter, an embodiment of determining the gradient value in the vertical direction by applying a gradient filter and then applying an interpolation filter will be described.

[0293] For example, when the position of the reference pixel is a fractional pixel position at (x+α, y+β), where x and y are each integers and α and β are each fractions, the reference pixel position can be obtained by calculating the position of the reference pixel for integer pixels (x, y), (x, y-1), (x, y+1) to (x, y+M) in the vertical direction. Min ) and (x,y+M max ) uses a first 1D filter to perform filtering according to equation 18, where M Min and M Mmax Each is an integer.

[0294] [Formula 18]

[0295]

[0296] Here, gradFilter β can represent a gradient filter for determining the gradient value at a fractional pixel position β in the vertical direction, and gradFilter β [j'] may represent the coefficient of the gradient filter applied to the pixel at the position (i, j'). I[i, j'] may represent the pixel value at the position (i, j').

[0297] In other words, the first 1D filter may be an interpolation filter for determining a gradient value of a pixel in a vertical direction, wherein the vertical component of the pixel position is a fractional position. 7 Can represent an offset to prevent rounding errors, and shift 7 The number of descaling bits can be indicated.

[0298] Temp[i,j+β] can represent the gradient value at the pixel position (i,j+β) in the vertical direction. Temp[i',j+β] can also be determined according to Formula 18 by replacing i with i', where i' is the gradient value from i+M excluding i. min To i+M max An integer.

[0299] Then, the video decoding apparatus 100 may perform filtering on the gradient value at the pixel position (i, j+β) in the vertical direction and the gradient value at the pixel position (i′, j+β) in the vertical direction by using a second 1D filter according to Equation 19.

[0300] [Formula 19]

[0301]

[0302] Here, fracFilter α It can be an interpolation filter used to determine the pixel value at the fractional pixel position α in the horizontal direction.α [i'] may represent a coefficient of an interpolation filter applied to a pixel at the position (i', j+β). In other words, the second 1D filter may be an interpolation filter for determining a pixel value at a fractional pixel position α in the horizontal direction. 8 Can represent an offset to prevent rounding errors, and shift 8 The number of descaling bits can be indicated.

[0303] In other words, according to Formula 19, the video decoding device 100 can be decoded by using the gradient filter fracFilter α The gradient value in the vertical direction at the pixel position (i, j+β) is filtered (Temp[i, j+β]) and the gradient value in the vertical direction of the pixel located in the horizontal direction from the pixel position (i, j+β) (Temp[i', j+β]) to determine the gradient value in the vertical direction at (i+α, j+β).

[0304] According to an embodiment, in the video decoding device 100, the gradient values ​​in the horizontal and vertical directions at (i+α, j+β) may be determined according to a combination of the various filters described above. For example, in order to determine the gradient value in the horizontal direction, an interpolation filter for determining the pixel value in the vertical direction may be used as the first 1D filter and a gradient filter for determining the gradient value in the horizontal direction may be used as the second 1D filter. Alternatively, a gradient filter for determining the gradient value in the vertical direction may be used as the first 1D filter, and an interpolation filter for determining the pixel value in the horizontal direction may be used as the second 1D filter.

[0305] Figures 7a to 7e is a table showing filter coefficients of a filter for determining pixel values ​​at fractional pixel positions of fractional pixel units and gradient values ​​in horizontal and vertical directions according to an embodiment.

[0306] Figure 7a and Figure 7b is a table showing filter coefficients of a filter used to determine gradient values ​​at fractional pixel positions of a 1 / 4 pel unit in the horizontal or vertical direction.

[0307] As described above, a 1D gradient filter and a 1D interpolation filter may be used to determine the gradient value in the horizontal or vertical direction. Figure 7a , illustrates the filter coefficients of a 1D gradient filter. Here, a 6-tap filter can be used as a 1D gradient filter. The filter coefficients of the 1D gradient filter can be coefficients scaled by 2^4. minrepresents the difference between the position of the center integer pixel and the position of the farthest pixel from among the integer pixels in the negative direction applied to the filter based on the center integer pixel, and M max represents the difference between the position of the center integer pixel and the position of the farthest pixel from among the integer pixels in the positive direction of the filter applied based on the center integer pixel. For example, the gradient filter coefficients for obtaining the gradient value of the pixel in the horizontal direction (where the fractional pixel position α is 1 / 4 in the horizontal direction) may be {4, -17, -36, 60, -15, -4}. It may also be possible to refer to Figure 7a To determine the gradient filter coefficient for obtaining the gradient value in the horizontal direction of the pixel (where the fractional pixel position α is 0, 1 / 2 or 3 / 4 in the horizontal direction).

[0308] refer to Figure 7b , illustrates the filter coefficients of the 1D interpolation filter. Here, a 6-tap filter can be used as the 1D interpolation filter. The filter coefficients of the 1D interpolation filter can be coefficients scaled by 2^6. min represents the difference between the position of the center integer pixel and the position of the farthest pixel from among the integer pixels in the negative direction applied to the filter based on the center integer pixel, and M max Represents the difference between the position of the center integer pixel and the position of the farthest pixel from among the integer pixels in the positive direction applied to the filter based on the center integer pixel.

[0309] Figure 7c is a table showing filter coefficients of a 1D interpolation filter used to determine pixel values ​​at fractional pixel positions of a 1 / 4 pel unit.

[0310] As described above, two identical ID interpolation filters may be used in the horizontal and vertical directions to determine pixel values ​​at fractional pixel locations.

[0311] refer to Figure 7c , illustrates the filter coefficients of the 1D interpolation filter. Here, a 6-tap filter can be used as the 1D interpolation filter. The filter coefficients of the 1D interpolation filter can be coefficients scaled by 2^6. min represents the difference between the position of the center integer pixel and the position of the farthest pixel from among the integer pixels in the negative direction applied to the filter based on the center integer pixel, and M max Represents the difference between the position of the center integer pixel and the position of the farthest pixel from among the integer pixels in the positive direction applied to the filter based on the center integer pixel.

[0312] Figure 7d is a table showing filter coefficients of a filter used to determine a gradient value in the horizontal or vertical direction at a fractional pixel position of a 1 / 16 pel unit.

[0313] As described above, a 1D gradient filter and a 1D interpolation filter may be used to determine the gradient value in the horizontal or vertical direction. Figure 7d , illustrating the filter coefficients of a 1D gradient filter. Here, a 6-tap filter can be used as a 1D gradient filter. The filter coefficients of the 1D gradient filter can be coefficients scaled by 2^4. For example, the gradient filter coefficients for obtaining the gradient value in the horizontal direction of a pixel (where the fractional pixel position α is 1 / 16 in the horizontal direction) can be {8, -32, -13, 50, -18, 5}. Also refer to Figure 7d The gradient filter coefficient for obtaining the gradient value in the horizontal direction of the pixel (where the fractional pixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, or 1 / 2 in the horizontal direction) can be determined. At the same time, the gradient filter coefficient for obtaining the gradient value in the horizontal direction of the pixel (where the fractional pixel position α is 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, or 15 / 16 in the horizontal direction) can be determined by using the symmetry of the filter coefficient based on α=1 / 2. In other words, the gradient filter coefficient for obtaining the gradient value in the horizontal direction of the pixel (where the fractional pixel position α is 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, or 15 / 16 in the horizontal direction) can be determined by using Figure 7d The filter coefficient at the right fraction pixel position based on α=1 / 2 is determined by the filter coefficient at the left fraction pixel position based on α=1 / 2 shown in . For example, the filter coefficient at α=1 / 16 can be determined by using the filter coefficient {8, -32, -13, 50, -18, 5} at α=1 / 16 as the symmetrical position based on α=1 / 2. In other words, the filter coefficient at α=15 / 16 can be determined as {5, -18, 50, -13, -32, 8} by arranging {8, -32, -13, 50, -18, 5} in reverse order.

[0314] refer to Figure 7e , illustrating the filter coefficients of a 1D interpolation filter. Here, a 6-tap filter can be used as a 1D interpolation filter. The filter coefficients of the 1D interpolation filter can be coefficients scaled by 2^6. For example, the 1D interpolation filter coefficients for obtaining pixel values ​​in the horizontal direction (where the fractional pixel position α is 1 / 16 in the horizontal direction) can be {1, -3, 64, 4, -2, 0}. It can also be referred to Figure 7eTo determine the interpolation filter coefficients for obtaining pixel values ​​of pixels in the horizontal direction (where the fractional pixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, or 1 / 2 in the horizontal direction). At the same time, the interpolation filter coefficients for obtaining pixel values ​​of pixels in the horizontal direction (where the fractional pixel position α is 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, or 15 / 16 in the horizontal direction) can be determined by using the symmetry of the filter coefficient based on α=1 / 2. In other words, the interpolation filter coefficients can be determined by using Figure 7e The filter coefficient at the right fraction pixel position based on α=1 / 2 is determined by the filter coefficient at the left fraction pixel position based on α=1 / 2 shown in . For example, the filter coefficient at α=1 / 16 can be determined by using the filter coefficient {1,-3,64,4,-2,0} at α=1 / 16 as a symmetrical position based on α=1 / 2. In other words, the filter coefficient at α=15 / 16 can be determined as {0,-2,4,64,-3,1} by arranging {1,-3,64,4,-2,0} in reverse order.

[0315] Figure 8 is a reference diagram for describing a process of determining a horizontal direction displacement vector and a vertical direction displacement vector according to an embodiment.

[0316] refer to Figure 8 , the window Ωij 800 with a specific size has a size of (2M+1)*(2N+1) based on the pixel P(i,j) bidirectionally predicted from the current block, where M and N are each integers.

[0317] When P(i',j') represents a pixel of a bidirectionally predicted current block in window Ωij800, wherein when iM≤i'≤i+M and jN≤j'≤j+N, (i',j')∈Ωij, P0(i',j') represents a pixel value of a first reference pixel of a first reference picture 810 corresponding to the pixel P(i',j') of the bidirectionally predicted current block, P1(i',j') represents a pixel value of a second reference pixel of a second reference picture 820 corresponding to the pixel P(i',j') of the bidirectionally predicted current block, represents the gradient value of the first reference pixel in the horizontal direction, represents the gradient value of the first reference pixel in the vertical direction, represents the gradient value of the second reference pixel in the horizontal direction, and represents the gradient value of the second reference pixel in the vertical direction, and the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can be determined according to Formula 20. Here, PA' and PB' can be determined by using the first linear term of the local Taylor expansion.

[0318] [Formula 20]

[0319]

[0320]

[0321] In Formula 20, the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction can be changed according to the position of the pixel P(i,j), that is, depending on (i,j), the displacement vectors Vx and Vy can be expressed as Vx(i,j) and Vy(i,j).

[0322] The difference Δi′j′ between the first shift corresponding pixel PA′ and the second shift corresponding pixel PB′ can be determined according to Formula 21.

[0323] [Formula 21]

[0324]

[0325] The displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction that minimizes the difference Δi'j' between the first shift corresponding pixel PA' and the second shift corresponding pixel PB' can be determined by using the square sum Φ(Vx,Vy) of the difference Δi'j' as in Equation 22.

[0326] [Formula 22]

[0327]

[0328] In other words, the displacement vectors Vx and Vy can be determined by using the local maximum or local minimum of Φ(Vx, Vy). Φ(Vx, Vy) represents a function using the displacement vectors Vx and Vy as parameters, and the local maximum or local minimum can be determined by calculating a value that becomes 0 by partially differentiating Φ(Vx, Vy) arranged for τVx and τVy with respect to τVx and τVy according to Formula 23. In the following, for convenience of calculation, τ 0 and τ 1 are all the same, that is, they are all τ.

[0329] [Formula 23]

[0330]

[0331] This can be done by using the equation: And the equation: To obtain two linear equations like Equation 24 using Vx(i,j) and Vy(i,j) as variables.

[0332] [Equation 24]

[0333] τVx*s1+τVy(i,j)*s2=s3

[0334] τVx*s4+rVy(i,j)*s5=s6

[0335] In Formula 24, s1 to s6 can be calculated according to Formula 25.

[0336] [Formula 25]

[0337]

[0338]

[0339]

[0340]

[0341]

[0342] By solving the simultaneous equations of Formula 24, the values ​​of Vx(i, j) and Vy(i, j) can be obtained according to τ*Vx(i, j)=-det1 / det and τ*Vy(i, j)=-det2 / det based on the Cramer formula. Here, det1=s3*s5-s2*s6, det2=s1*s6-s3*s4, and det=s1*s5-s2*s2.

[0343] By performing minimization first in the horizontal direction and then in the vertical direction, a simplified solution to the above equation can be determined. In other words, when only the displacement vector in the horizontal direction changes, Vy=0 in the first equation of equation 24, and therefore the equation can be determined: τVx=s3 / s1.

[0344] Then, when the second equation of Formula 24 is arranged by using the equation: τVx=s3 / s1, the equation: τVy=(s6-τVx*S2) / s5 can be determined.

[0345] Here, the gradient value can be adjusted without changing the result values ​​Vx(i, j) and Vy(i, j). and Scaling is performed. However, this assumes that overflow does not occur and that no rounding errors are generated.

[0346] Normalization parameters r and m may be introduced to prevent division by zero or very small values ​​from being performed while calculating Vx(i, j) and Vy(i, j).

[0347] For convenience, we consider Vx(i, j) and Vy(i, j) to be Figure 3a For example, based on Figure 3a The directions of Vx(i, j) and Vy(i, j) derived by equation 24 may have the same meaning as that determined by Figure 3a Vx(i,j) and Vy(i,j) with opposite directions have the same size.

[0348] The first shift corresponding pixel PA' and the second shift corresponding pixel PB' may be determined according to Equation 26. Here, the first shift corresponding pixel PA' and the second shift corresponding pixel PB' may be determined by using a first linear term of a local Taylor expansion.

[0349] [Equation 26]

[0350]

[0351]

[0352] The difference Δi′j′ between the first shift corresponding pixel PA′ and the second shift corresponding pixel PB′ can be determined according to Formula 27.

[0353] [Formula 27]

[0354]

[0355]

[0356] The displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction that minimize the difference Δi'j' between the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can be determined by using the square sum Φ(Vx, Vy) of the difference Δ as in Formula 28. In other words, the displacement vectors Vx and Vy when Φ(Vx, Vy) is the minimum value as in Formula 29 can be determined, and can be determined by using the local maximum value or the local minimum value of Φ(Vx, Vy).

[0357] [Equation 28]

[0358]

[0359] [Formula 29]

[0360] (Vx, Vy) = argmin Vx，Vy Φ(Vx,Vy)

[0361] Φ(Vx, Vy) is a function using displacement vectors Vx and Vy as parameters, and the local maximum or local minimum can be determined by calculating the value that becomes 0 by partially differentiating Φ(Vx, Vy) with respect to displacement vectors Vx and Vy as in Formula 30.

[0362] [Formula 30]

[0363]

[0364] In other words, the displacement vectors Vx and Vy that minimize Φ(Vx, Vy) can be determined. To solve the optimization problem, minimization can be performed first in the vertical direction and then in the horizontal direction. Based on the minimization, the displacement vector Vx can be determined according to Equation 31.

[0365] [Equation 31]

[0366]

[0367] Here, function clip3(x, y, z) is a function that outputs x when z<x, outputs y when z>y, and outputs z when x<z<y. According to Formula 31, when s1+r>m, displacement vector Vx may be clip3(-thBIO, thBIO, -s3 / (s1+r)), and when s1+r>m is not, displacement vector Vx may be 0.

[0368] Based on minimization, the displacement vector Vy can be determined according to equation 32.

[0369] [Formula 32]

[0370]

[0371] Here, function clip3(x, y, z) is a function that outputs x when Z<x, outputs y when z>y, and outputs z when x<z<y. According to Formula 32, when s5+r>m, displacement vector Vy may be clip3(-thBIO, thBIO, -(s6-Vx*s2) / 2 / (s5+r), and when s5+r>m is not, displacement vector Vy may be 0.

[0372] Here, s1, s2, s3 and s5 can be determined according to Formula 33.

[0373] [Formula 33]

[0374]

[0375]

[0376]

[0377]

[0378]

[0379] As described above, r and m may be normalization parameters introduced to prevent the division result value from being 0 or less and determined according to Equation 34 based on the internal bit depth d of the input video.

[0380] [Equation 34]

[0381] r=500*4 d-8

[0382] m=700*4 d-8

[0383] The displacement vectors Vx and Vy may have upper and lower limits of ±thBIO. Because there may be situations where motion compensation in pixel units may be unreliable due to noise or irregular motion, the displacement vectors Vx and Vy may be trimmed by a specific threshold thBIO. thBIO may be determined based on whether the directions of all reference pictures are the same. For example, when the directions of all reference pictures are the same, thBIO may be determined as 12*2^(14-d). When the directions of all reference pictures are different, thBIO may be determined as 12*2^(13-d).

[0384] Fig. 9 is a diagram for describing a process of adding an offset value after performing filtering and determining a gradient value in a horizontal or vertical direction by performing inverse scaling according to an embodiment.

[0385] refer to Fig. 9 , the video decoding device 100 can determine the gradient value in the horizontal or vertical direction by filtering the pixels whose components in a specific direction are at integer positions using the first 1D filter and the second 1D filter. However, the value obtained by filtering the pixels whose components in a specific direction are at integer positions using the first 1D filter or the second 1D filter may be outside a specific range. This phenomenon is called an overflow phenomenon. The coefficients of the 1D filter can be determined as integers for integer operations rather than inaccurate and complex fractional operations. The coefficients of the 1D filter can be scaled to be determined as integers. When filtering is performed by using the scaled coefficients of the 1D filter, integer operations can be performed, but the size of the value of the filtering may be high and an overflow phenomenon may occur compared to when filtering is performed by using the unscaled coefficients of the 1D filter. Therefore, in order to prevent the overflow phenomenon, descaling can be performed after filtering is performed by using the 1D filter. Here, descaling may include shifting the descaling bit number to the right. While maximizing the accuracy of the calculation, the descaling bit number can be determined by considering the maximum number of bits of the register used for the filtering operation and the maximum number of bits of the temporary buffer storing the filtering result. In particular, the descaling bit number may be determined based on the internal bit depth, the scaling bit number of the interpolation filter, and the scaling bit number for the gradient filter.

[0386] Hereinafter, the performance of descaling during a process of generating an interpolation filter value in a vertical direction by first performing filtering on pixels at integer positions by using an interpolation filter in a vertical direction so as to determine a gradient value in a horizontal direction and then performing filtering on the interpolation filter value in the vertical direction by using a gradient filter in a horizontal direction will be described.

[0387] According to the above Equation 12, the video decoding apparatus 100 may first perform filtering on pixels at integer positions by using an interpolation filter in the vertical direction in order to determine a gradient value in the horizontal direction. 1 It can be b-8. Here, b can represent the internal bit depth of the input image. In the following, reference will be made to Table 1 to describe the case where the shift 1 The bit depth of the registers (Reg BitDepth) and the bit depth of the temporary buffer (TempBitDepth) when actually performing the descaling.

[0388] [Table 1]

[0389]

[0390] Here, the values ​​of the variables in Table 1 can be determined according to Formula 35.

[0391] [Formula 35]

[0392] RegMin=Min(I)*FilterSumPos+Max(I)*FilterSumNeg

[0393] RegMax=Max(I)*FilterSumPos+Min(I)*FilterSumNeg

[0394] Reg BitDepth = ceiling (log 2 (RegMax-RegMin)+1)

[0395] TempMin=(RegMin+offset1)>>shift1

[0396] TempMax=(RegMax+offset1)>>shift1

[0397] Temp BitDepth = ceiling (log 2 (TempMax-TempMin)+1)

[0398] Here, Min(I) may represent the minimum value of the pixel value I determined by the internal bit depth, and Max(I) may represent the maximum value of the pixel value I determined by the internal bit depth. FilterSumPos represents the maximum value of the sum of positive filter coefficients, and FilterSumNeg represents the minimum value of the sum of negative filter coefficients.

[0399] For example, when using Figure 7c When using the gradient filter FracFilter with a 1 / 4 pel unit, FilterSumPos can be 88 and FilterSumNeg can be -24.

[0400] Function Celing(x) may be a function that outputs the smallest integer from among integers equal to or higher than x for a real number x. 1 is added to the value to perform the filtering to prevent using shift 1 To perform the rounding error that may be generated when descaling, and you can use the offset 1 Determined to be 2^(shift 1 -1).

[0401] Referring to Table 1, when the internal bit depth b is 8, the bit depth of the register (Reg BitDepth) may be 16, when the internal bit depth b is 9, the bit depth of the register may be 17, and when the internal bit depth b is 10, 11, 12, and 16, the bit depth of the register may be 18, 19, and 24. When the register used to perform filtering is a 32-bit register, since the bit depth of all registers in FIG. 1 does not exceed 32, an overflow phenomenon does not occur.

[0402] Similarly, when the internal bit depth b is 8, 9, 10, 11, 12, and 16, the bit depths (TempBitDepth) of the temporary buffers are all 16. When the temporary buffer for storing the values ​​for performing filtering and then descaling is a 16-bit buffer, since the bit depths of all temporary buffers in Table 1 are 16 and therefore do not exceed 16, an overflow phenomenon does not occur.

[0403] According to Formula 12, the video decoding apparatus 100 may generate an interpolation filter value in the vertical direction by first performing filtering on pixels at integer positions by using interpolation filtering in the vertical direction so as to determine a gradient value in the horizontal direction, and then perform filtering on the interpolation filter value in the vertical direction by using a gradient filter in the horizontal direction according to Formula 13. Here, the shift 2 Determined as p+q-shift 1 Here, p can represent the Figure 7c The number of bits of the interpolation filter scaling of the filter coefficients shown in , and q can represent the number of bits for the filter coefficients including Figure 7a The number of bits of the gradient filter scaling of the filter coefficients shown in . For example, p can be 6 and q can be 4, and thus, shift 2 =18-b.

[0404] shift 2 This is determined because shift 1 +shift 2 (i.e., the sum of the number of descaled bits) should be the same as the sum of the number of bits upscaled for the filter (p+q), so that the final filtered result value is the same when the filter coefficients are upscaled and when the filter coefficients are not upscaled.

[0405] In the following, reference will be made to Table 2 to describe the case where the shift 2 The bit depth of the register (Reg BitDepth) and the bit depth of the temporary buffer (Temp BitDepth) when actually performing descaling.

[0406] [Table 2]

[0407]

[0408] Here, the values ​​of the variables in Table 2 can be determined according to Formula 36.

[0409] [Equation 36]

[0410] RegMin=TempMin*FilterSumPos+TempMax*FilterSumNeg

[0411] RegMax=TempMax*FilterSumPos+TempMin*FilterSumNeg

[0412] Reg BitDepth = ceiling (log 2 (RegMax-RegMin)+1)

[0413] TempMin=(RegMin+offset2)>>shift2

[0414] TempMax=(RegMax+offset2)>>shift2

[0415] Temp BitDepth = ceiling (log 2 (TempMax-TempMin)+1)

[0416] Here, TempMax may represent TempMax of Table 1 and TempMin may represent TempMin of Table 1. FilterSumPos represents the maximum value of the sum of positive filter coefficients and FilterSumNeg represents the minimum value of the sum of negative filter coefficients. For example, when using Figure 7c For the gradient filter gradFilter with a 1 / 4 pel unit as shown in , FilterSumPos can be 68 and FilterSumNeg can be -68.

[0417] offset 2 is added to the value after filtering is performed to prevent the use of shift 2 To perform the rounding error that may be generated when descaling, and you can use the offset 1 Determined to be 2^(shift 2 -1).

[0418] shift 1 and shift 2 may be determined as such, but alternatively, the shift may be determined differently 1 and shift 2 , just shift 1 and shift 2 Here, the shift can be determined based on the premise that no overflow occurs. 1 and shift 2 The shift can be determined based on the internal bit depth of the input image and the number of scaling bits for the filter. 1 and shift 2 .

[0419] However, shift 1 and shift 2 may not necessarily be determined to make shift 1 and shift 2 The sum of the number of scaling bits for the filter is equal to the sum of the number of scaling bits for the filter. 1 Determined to be d-8, but you can shift 2 Determined as a fixed number.

[0420] When shift 1 Same as before and shift 2When it is a fixed number 7, OutMax, OutMin and Temp Bitdepth described with reference to Table 2 can be changed. Hereinafter, the bit depth (TempBitDepth) of the temporary buffer will now be described with reference to Table 3.

[0421] [Table 3]

[0422] b OutMax OutMin Temp Bitdepth 8 15173 -15174 16 9 15203 -15204 16 10 15218 -15219 16 11 15225 -15226 16 12 15229 -15230 16 16 15233 -15234 16

[0423] Unlike Table 2, in Table 3, the bit depth (Temp BitDepth) of the temporary buffer is the same in all b, that is, 16, and when the result data is stored by using a 16-bit temporary buffer, the bit depth (Temp BitDepth) of the temporary buffer is less than 16, and thus an overflow phenomenon does not occur for the internal bit depths of all input images. Meanwhile, referring to Table 2, when the internal bit depth of the input image is 12 and 16 and the result data is stored by using a 16-bit temporary buffer, the bit depth (Temp BitDepth) of the temporary buffer is higher than 16, and thus an overflow phenomenon may occur.

[0424] When shift 2 When is a fixed number, the scaled filter coefficient is not used, and the result value of performing filtering and the result value of performing filtering and then descaling may be different. In this case, it will be obvious to those skilled in the art that descaling needs to be performed additionally.

[0425] In the above, the performance of descaling during a process of generating interpolation filtered values ​​in the vertical direction by first performing filtering on pixels at integer positions by using an interpolation filter in the vertical direction in order to determine gradient values ​​in the horizontal direction and then performing filtering on the interpolation filtered values ​​in the vertical direction by using a gradient filter in the horizontal direction has been described, but it will be apparent to those of ordinary skill in the art that descaling may be performed in a similar manner when filtering is performed on pixels whose components in a specific direction are integers through a combination of various 1D filters in order to determine gradient values ​​in the horizontal and vertical directions.

[0426] In the following, reference will be made to Figures 10 to 23 A method of determining a data unit that can be used when the video decoding apparatus 100 according to an embodiment decodes an image is described. The operation of the video encoding apparatus 150 may be similar to or opposite to various embodiments of the operation of the video decoding apparatus 100 described below.

[0427] Fig.10 A process of determining at least one coding unit as the video decoding apparatus 100 splits a current coding unit according to an embodiment is illustrated.

[0428] According to an embodiment, the video decoding apparatus 100 may determine the shape of the coding unit by using block shape information, and determine the shape into which the coding unit is divided by using segmentation shape information. In other words, the segmentation method of the coding unit indicated by the segmentation shape information may be determined based on the block shape indicated by the block shape information used by the video decoding apparatus 100.

[0429] According to an embodiment, the video decoding apparatus 100 may use block shape information indicating that the current coding unit has a square shape. For example, the video decoding apparatus 100 may determine whether to not split the square coding unit, split the square coding unit vertically, split the square coding unit horizontally, or split the square coding unit into four coding units according to the split shape information. Fig.10 , when the block shape information of the current coding unit 1000 indicates a square shape, the video decoding device 100 may not split the coding unit 1010a having the same size as the current coding unit 1000 according to the split shape information indicating non-split, or determine the coding unit 1010b, 1010c or 1010d based on the split shape information indicating a specific split method.

[0430] refer to Fig.10 According to an embodiment, the video decoding device 100 may determine two coding units 1010b by splitting the current coding unit 1000 in the vertical direction based on the split shape information indicating the split in the vertical direction. The video decoding device 100 may determine two coding units 1010c by splitting the current coding unit 1000 in the horizontal direction based on the split shape information indicating the split in the horizontal direction. The video decoding device 100 may determine four coding units 1010d by splitting the current coding unit 1000 in the vertical and horizontal directions based on the split shape information indicating the split in the vertical and horizontal directions. However, the split shapes into which the square coding unit may be split are not limited to the above shapes, and may include any shape that may be indicated by the split shape information. Some of the split shapes into which the square coding unit may be split will now be described in detail through various embodiments.

[0431] Fig.11 A process of determining at least one coding unit when the video decoding apparatus 100 splits coding units having a non-square shape according to an embodiment is illustrated.

[0432] According to an embodiment, the video decoding apparatus 100 may use block shape information indicating that the current coding unit has a non-square shape. The video decoding apparatus 100 may determine whether to not split the non-square current coding unit or to split the non-square current coding unit via a specific method according to the split shape information. Fig.11, when the block shape information of the current coding unit 1100 or 1150 indicates a non-square shape, the video decoding apparatus 100 may not split the coding unit 1110 or 1160 having the same size as the current coding unit 1100 or 1150 according to the split shape information indicating non-split, or determine the coding units 1120a, 1120b, 1130a, 1130b, 1130c, 1170a, 1170b, 1180a, 1180b, and 1180c based on the split shape information indicating a specific split method. A specific split method for splitting a non-square coding unit will now be described in detail through various embodiments.

[0433] According to an embodiment, the video decoding apparatus 100 may determine the shape into which the coding unit is split by using the split shape information, and in this case, the split shape information may indicate the number of at least one coding unit generated as the coding unit is split. Fig.11 , when the segmentation shape information indicates that the current coding unit 1100 or 1150 is segmented into two coding units, the video decoding device 100 can determine the two coding units 1120a and 1120b or 1170a and 1170b included in the current coding unit 1100 or 1150 by segmenting the current coding unit 1100 or 1150 based on the segmentation shape information.

[0434] According to an embodiment, when the video decoding apparatus 100 splits the current coding unit 1100 or 1150 having a non-square shape based on the split shape information, the video decoding apparatus 100 may split the current coding unit 1100 or 1150 considering the position of the long side of the current coding unit 1100 or 1150 having the non-square shape. For example, the video decoding apparatus 100 may determine a plurality of coding units by splitting the current coding unit 1100 or 1150 in a direction in which the long side of the current coding unit 1100 or 1150 is split considering the shape of the current coding unit 1100 or 1150.

[0435] According to an embodiment, when the segmentation shape information indicates that the coding unit is segmented into an odd number of blocks, the video decoding apparatus 100 may determine an odd number of coding units included in the current coding unit 1100 or 1150. For example, when the segmentation shape information indicates that the current coding unit 1100 or 1150 is segmented into three coding units, the video decoding apparatus 100 may segment the current coding unit 1100 or 1150 into three coding units 1130a to 1130c or 1180a to 1180c. According to an embodiment, the video decoding apparatus 100 may determine an odd number of coding units included in the current coding unit 1100 or 1150, and the sizes of the determined coding units may not all be the same. For example, the size of the coding unit 1130b or 1180b from among the determined odd number of coding units 1130a to 1130c or 1180a to 1180c may be different from the sizes of the coding units 1130a and 1130c or 1180a and 1180c. In other words, the coding units that may be determined when the current coding unit 1100 or 1150 is split may have various types of sizes, and in some cases, the coding units 1130a to 1130c or 1180a to 1180c may have different sizes.

[0436] According to an embodiment, when the partition shape information indicates that the coding unit is partitioned into an odd number of blocks, the video decoding apparatus 100 may determine an odd number of coding units included in the current coding unit 1100 or 1150, and in addition, may set a specific restriction on at least one coding unit from among the odd number of coding units generated through partitioning. Fig.11 , the video decoding apparatus 100 may distinguish a decoding process performed on a coding unit 1130b or 1180b located at the center, wherein the coding unit 1130b or 1180b is from among three coding units 1130a to 1130c or 1180a to 1180c generated as the current coding unit 1100 or 1150 is split from other coding units 1130a and 1130c or 1180a and 1180c. For example, the video decoding apparatus 100 may limit the coding unit 1130b or 1180b located at the center to not be split any more, or to be split only a specific number of times, unlike other coding units 1130a and 1130c or 1180a and 1180c.

[0437] Fig.12 A process in which the video decoding apparatus 100 according to an embodiment splits a coding unit based on at least one of block shape information and partition shape information is illustrated.

[0438] According to an embodiment, the video decoding device 100 may determine whether the first coding unit 1200 having a square shape is segmented or not segmented into coding units based on at least one of the block shape information and the segmentation shape information. According to an embodiment, when the segmentation shape information indicates that the first coding unit 1200 is segmented in the horizontal direction, the video decoding device 100 may determine the second coding unit 1210 by segmenting the first coding unit 1200 in the horizontal direction. The first coding unit, the second coding unit, and the third coding unit used according to the embodiment are terms for indicating the relationship before and after the segmentation of the coding unit. For example, the second coding unit may be determined by segmenting the first coding unit, and the third coding unit may be determined by segmenting the second coding unit. In the following, it should be understood that the relationship between the first to third coding units is consistent with the above-mentioned features.

[0439] According to an embodiment, the video decoding apparatus 100 may determine whether the determined second coding unit 1210 is split or not split into coding units based on at least one of block shape information and partition shape information. Fig.12 , the video decoding apparatus 100 may split the second coding unit 1210 having a non-square shape and determined by splitting the first coding unit 1200 into at least one third coding unit 1210a, 1220b, 1220c, or 1220d, or may not split the second coding unit 1210 based on at least one of the block shape information and the split shape information. The video decoding apparatus 100 may obtain at least one of the block shape information and the split shape information, and obtain a plurality of second coding units (e.g., the second coding unit 1210) having various shapes by splitting the first coding unit 1200 based on at least one of the obtained block shape information and the split shape information, wherein the second coding unit 1210 may be split according to a method of splitting the first coding unit 1200 based on at least one of the block shape information and the split shape information. According to an embodiment, when the first coding unit 1200 is split into the second coding unit 1210 based on at least one of the block shape information and the segmentation shape information about the first coding unit 1200, the second coding unit 1210 may also be split into third coding units (e.g., third coding units 1220a to 1220d) based on at least one of the block shape information and the segmentation shape information about the second coding unit 1210. In other words, the coding units may be recursively split based on at least one of the segmentation shape information and the block shape information related to each coding unit. Therefore, a square coding unit may be determined from a non-square coding unit, and such a square coding unit may be recursively split so that the non-square coding unit is determined. Reference Fig.12, a specific coding unit (for example, a coding unit located at the center or a square coding unit) may be recursively split, wherein the specific coding unit is from among the odd number of third coding units 1220b to 1220d determined when the second coding unit 1210 having a non-square shape is split. According to an embodiment, a third coding unit 1220c having a square shape from among the third coding units 1220b to 1220d may be split into a plurality of fourth coding units in the horizontal direction. A fourth coding unit 1240 having a non-square shape from among the plurality of fourth coding units may be split into a plurality of coding units again. For example, the fourth coding unit 1240 having a non-square shape may be split into an odd number of coding units 1250a to 1250c.

[0440] A method that may be used to recursively split a coding unit will be described below through various embodiments.

[0441] According to an embodiment, the video decoding device 100 may determine that each of the third coding units 1220a to 1220d is split into coding units or the second coding unit 1210 is not split based on at least one of the block shape information and the split shape information. According to an embodiment, the video decoding device 100 may split the second coding unit 1210 having a non-square shape into an odd number of third coding units 1220b to 1220d. The video decoding device 100 may set a specific restriction on a specific third coding unit from among the third coding units 1220b to 1220d. For example, the video decoding device 100 may restrict the third coding unit 1220c located at the center of the third coding units 1220b to 1220d from being no longer split, or from being split a settable number of times. Reference Fig.12 , the video decoding apparatus 100 may limit the third coding unit 1220c located at the center of the third coding units 1220b to 1220d included in the second coding unit 1210 having a non-square shape to no longer be divided, to be divided into a specific division shape (for example, to be divided into four coding units or to be divided into shapes corresponding to those into which the second coding unit 1210 is divided), or to be divided only a certain number of times (for example, to be divided only n times, where n>0). However, such limitation on the third coding unit 1220c located at the center is merely an example and should not be interpreted as being limited by those examples, but should be interpreted as including various limitations as long as the third coding unit 1220c located at the center is decoded differently from the other third coding units 1220b and 1220d.

[0442] According to an embodiment, the video decoding apparatus 100 may obtain at least one of block shape information and partition shape information for partitioning the current coding unit from a specific position in the current coding unit.

[0443] Fig.13 A method for determining a specific coding unit from among odd-numbered coding units by the video decoding apparatus 100 according to an embodiment is illustrated. Fig.13 , at least one of the block shape information and the partition shape information of the current coding unit 1300 can be obtained from a sampling point at a specific position (for example, a sampling point 1340 located at the center) from among a plurality of sampling points included in the current coding unit 1300. However, the specific position in the current coding unit 1300 where at least one of the block shape information and the partition shape information is obtained is not limited to Fig.13 , but may be any position (e.g., an uppermost position, a lowermost position, a left position, a right position, an upper left position, a lower left position, an upper right position, or a lower right position) included in the current coding unit 1300. The video decoding apparatus 100 may determine whether the current coding unit is split into coding units having various shapes and sizes or not split by obtaining at least one of block shape information and partition shape information from a specific position.

[0444] According to an embodiment, the video decoding apparatus 100 may select one coding unit when the current coding unit is split into a certain number of coding units. A method of selecting one of a plurality of coding units may vary, and details thereof will be described below through various embodiments.

[0445] According to an embodiment, the video decoding apparatus 100 may split a current coding unit into a plurality of coding units and determine a coding unit at a specific position.

[0446] Fig.13 A method of determining, by the video decoding apparatus 100 , a coding unit at a specific position from among odd-numbered coding units according to an embodiment is illustrated.

[0447] According to an embodiment, the video decoding apparatus 100 may use information indicating the position of each of the odd-numbered coding units in order to determine a coding unit located at the center from among the odd-numbered coding units. Fig.13, the video decoding apparatus 100 may determine the odd-numbered coding units 1320a to 1320c by dividing the current coding unit 1300. The video decoding apparatus 100 may determine the center coding unit 1320b by using information about the positions of the odd-numbered coding units 1320a to 1320c. For example, the video decoding apparatus 100 may determine the coding unit 1320b located at the center by determining the positions of the coding units 1320a to 1320b based on information indicating the positions of some sampling points included in the coding units 1320a to 1320c. In detail, the video decoding apparatus 100 may determine the coding unit 1320b located at the center by determining the positions of the coding units 1320a to 1320c based on information indicating the positions of the upper left sampling points 1330a to 1330c of the coding units 1320a to 1320c.

[0448] According to an embodiment, the information indicating the positions of the upper left sampling points 1330a to 1330c respectively included in the coding units 1320a to 1320c may include information about the positions or coordinates of the coding units 1320a to 1320c in the picture. According to an embodiment, the information indicating the positions of the upper left sampling points 1330a to 1330c respectively included in the coding units 1320a to 1320c may include information indicating the width or height of the coding units 1320a to 1320c included in the current coding unit 1300, and such width or height may correspond to information indicating the difference between the coordinates of the coding units 1320a to 1320c in the picture. In other words, the video decoding apparatus 100 may determine the coding unit 1320b located at the center by directly using the information about the positions or coordinates of the coding units 1320a to 1320c in the picture or by using the information about the width or height of the coding units 1320a to 1320c corresponding to the difference between the coordinates.

[0449] According to an embodiment, the information indicating the position of the upper left sampling point 1330a of the upper coding unit 1320a may indicate (xa, ya) coordinates, the information indicating the position of the upper left sampling point 1330b of the center coding unit 1320b may indicate (xb, yb) coordinates, and the information indicating the position of the upper left sampling point 1330c of the lower coding unit 1320c may indicate (xc, yc) coordinates. The video decoding apparatus 100 may determine the center coding unit 1320b by using the coordinates of the upper left sampling points 1330a to 1330c respectively included in the coding units 1320a to 1320c. For example, when the coordinates of the upper left sampling points 1330a to 1330c are arranged in ascending or descending order, the coding unit 1320b including the coordinates (xb, yb) of the sampling point 1330b located at the center may be determined as the coding unit located at the center from among the coding units 1320a to 1320c determined when the current coding unit 1300 is split. However, the coordinates indicating the positions of the upper left sampling points 1330a to 1330c may be coordinates indicating absolute positions in the picture, and in addition, the (dxb, dyb) coordinates (i.e., information indicating the relative position of the upper left sampling point 1330b of the center coding unit 1320b) and the (dxc, dyc) coordinates (i.e., information indicating the relative position of the upper left sampling point 1330c of the lower coding unit 1320c) may be used based on the position of the upper left sampling point 1330a of the upper coding unit 1320a. In addition, a method of determining a coding unit at a specific position by using the coordinates of the sampling points as information indicating the positions of the sampling points included in the coding unit is not limited to the above, and various arithmetic methods capable of using the coordinates of the sampling points may be used.

[0450] According to an embodiment, the video decoding apparatus 100 may split the current coding unit 1300 into a plurality of coding units 1320a to 1320c and select a coding unit from the coding units 1320a to 1320c according to a specific criterion. For example, the video decoding apparatus 100 may select a coding unit 1320b having a different size from among the coding units 1320a to 1320c.

[0451] According to an embodiment, the video decoding apparatus 100 may determine the width or height of the coding units 1320a to 1320c by using the (xa, ya) coordinates (i.e., information indicating the position of the upper left sampling point 1330a of the upper coding unit 1320a), the (xb, yb) coordinates (i.e., information indicating the position of the upper left sampling point 1330b of the center coding unit 1320b), and the (xc, yc) coordinates (i.e., information indicating the position of the upper left sampling point 1330c of the lower coding unit 1320c). The video decoding apparatus 100 may determine the sizes of the coding units 1320a to 1320c by using the coordinates (xa, ya), (xb, yb), and (xc, yc) indicating the positions of the coding units 1320a to 1320c, respectively.

[0452] According to an embodiment, the video decoding apparatus 100 may determine the width of the upper coding unit 1320a as xb-xa, and the height as yb-ya. According to an embodiment, the video decoding apparatus 100 may determine the width of the center coding unit 1320b as xc-xb, and the height as yc-yb. According to an embodiment, the video decoding apparatus 100 may determine the width or height of the lower coding unit 1320c by using the width and height of the current coding unit 1300 and the widths and heights of the upper coding unit 1320a and the center coding unit 1320b. The video decoding apparatus 100 may determine coding units having different sizes from other coding units based on the determined widths and heights of the coding units 1320a to 1320c. Reference Fig.13 , the video decoding apparatus 100 may determine the center coding unit 1320b having a size different from those of the upper coding unit 1320a and the lower coding unit 1320c as the coding unit at the specific position. However, the process in which the video decoding apparatus 100 determines the coding unit having a different size from other coding units is merely an example of determining the coding unit at the specific position by using the size of the coding unit determined based on the sampling point coordinates, and thus various processes of determining the coding unit at the specific position by comparing the sizes of the coding units determined according to certain sampling point coordinates may be used.

[0453] However, the position of the sampling point considered to determine the position of the coding unit is not limited to the upper left as described above, and information on the position of any sampling point included in the coding unit may be used.

[0454] According to an embodiment, the video decoding device 100 may select a coding unit at a specific position from among the odd number of coding units determined when the current coding unit is split while considering the shape of the current coding unit. For example, when the current coding unit has a non-square shape in which the width is longer than the height, the video decoding device 100 may determine a coding unit at a specific position in the horizontal direction. In other words, the video decoding device 100 may determine one of the coding units having different positions in the horizontal direction and set a restriction on the one coding unit. When the current coding unit has a non-square shape in which the height is longer than the width, the video decoding device 100 may determine a coding unit at a specific position in the vertical direction. In other words, the video decoding device 100 may determine one of the coding units having different positions in the vertical direction and set a restriction on the one coding unit.

[0455] According to an embodiment, the video decoding apparatus 100 may use information indicating the position of each of the even-numbered coding units in order to determine a coding unit at a specific position from among the even-numbered coding units. The video decoding apparatus 100 may determine the even-numbered coding units by splitting the current coding unit, and determine the coding unit at the specific position by using the information about the positions of the even-numbered coding units. The detailed process may correspond to Fig.13 The processes of determining a coding unit at a specific position (e.g., a center position) from among an odd number of coding units are described in , and therefore the details thereof are not provided again.

[0456] According to an embodiment, when a current coding unit having a non-square shape is split into a plurality of coding units, specific information about a coding unit at a specific position may be used during a splitting process to determine a coding unit at a specific position from among the plurality of coding units. For example, the video decoding apparatus 100 may use at least one of block shape information and split shape information stored in a sampling point included in a center coding unit during a splitting process in order to determine a coding unit located at the center from among a plurality of coding units obtained by splitting the current coding unit.

[0457] refer to Fig.13, the video decoding apparatus 100 may divide the current coding unit 1300 into a plurality of coding units 1320a to 1320c based on at least one of the block shape information and the segmentation shape information, and determine the coding unit 1320b located at the center from among the plurality of coding units 1320a to 1320c. In addition, the video decoding apparatus 100 may determine the coding unit 1320b located at the center in consideration of the position where at least one of the block shape information and the segmentation shape information is obtained. In other words, at least one of the block shape information and the segmentation shape information of the current coding unit 1300 may be obtained from a sampling point 1340 located at the center of the current coding unit 1300, and when the current coding unit 1300 is divided into a plurality of coding units 1320a to 1320c based on at least one of the block shape information and the segmentation shape information, the coding unit 1320b including the sampling point 1340 may be determined as the coding unit located at the center. However, information for determining the coding unit located at the center is not limited to at least one of the block shape information and the segmentation shape information, and various types of information may be used when determining the coding unit located at the center.

[0458] According to an embodiment, specific information for identifying a coding unit at a specific position may be obtained from a specific sampling point included in a coding unit to be determined. Fig.13 , the video decoding apparatus 100 may use at least one of the block shape information and the partition shape information obtained from the sampling point at the specific position in the current coding unit 1300 (for example, the sampling point located at the center of the current coding unit 1300) to determine the coding unit at the specific position (for example, the coding unit located at the center from among the multiple coding units) from among the multiple coding units 1320a to 1320c determined when the current coding unit 1300 is partitioned. In other words, the video decoding apparatus 100 may determine the sampling point at the specific position considering the block shape of the current coding unit 1300, and determine the coding unit 1320b including the sampling point for which specific information (for example, at least one of the block shape information and the partition shape information) can be obtained from among the multiple coding units 1320a to 1320c determined when the current coding unit 1300 is partitioned and set specific restrictions thereon. Reference Fig.13 According to an embodiment, the video decoding apparatus 100 may determine a sampling point 1340 located at the center of the current coding unit 1300 as a sampling point at which specific information can be obtained, and set a specific restriction on the coding unit 1320b including such a sampling point 1340 during the decoding process. However, the position of the sampling point at which the specific information can be obtained is not limited to the above, and may be a sampling point included at an arbitrary position in the coding unit 1320b determined to set the restriction.

[0459] According to an embodiment, the position of the sampling point from which specific information can be obtained may be determined according to the shape of the current coding unit 1300. According to an embodiment, the block shape information may determine whether the shape unit of the current coding is a square or a non-square, and the position of the sampling point from which specific information can be obtained is determined according to the shape. For example, the video decoding apparatus 100 may determine a sampling point located on such a boundary as a sampling point from which specific information can be obtained, wherein the boundary is a boundary that divides at least one of the width and the height of the current coding unit into two halves by using at least one of the information about the width of the current coding unit and the information about the height of the current coding unit. As another example, when the block shape information related to the current coding unit indicates a non-square shape, the video decoding apparatus 100 may determine one of the sampling points adjacent to the boundary that divides the long side of the current coding unit into two halves as a sampling point from which specific information can be obtained.

[0460] According to an embodiment, when a current coding unit is split into a plurality of coding units, the video decoding device 100 may use at least one of block shape information and split shape information in order to determine a coding unit at a specific position from among the plurality of coding units. According to an embodiment, the video decoding device 100 may obtain at least one of block shape information and split shape information from a sampling point at a specific position included in the coding unit, and may split a plurality of coding units generated as the current coding unit is split by using at least one of the split shape information and block shape information obtained from a sampling point at a specific position included in each of the plurality of coding units. In other words, the coding unit may be recursively split by using at least one of the block shape information and split shape information obtained from a sampling point at a specific position included in each of the plurality of coding units. As has been mentioned above with reference to Fig.12 The process of recursively splitting the coding unit is described, so its details are not provided again.

[0461] According to an embodiment, the video decoding apparatus 100 may determine at least one coding unit by splitting a current coding unit, and determine an order in which to decode the at least one coding unit according to a specific block (for example, the current coding unit).

[0462] Fig.14 An order in which a plurality of coding units are processed when a plurality of coding units are determined when the video decoding apparatus 100 splits a current coding unit according to an embodiment is illustrated.

[0463] According to an embodiment, the video decoding device 100 can determine the second coding units 1410a and 1410b by splitting the first coding unit 1400 in the vertical direction according to the block shape information and the segmentation shape information, determine the second coding units 1430a and 1430b by splitting the first coding unit 1400 in the horizontal direction, or determine the second coding units 1450a to 1450d by splitting the first coding unit 140 in the horizontal and vertical directions.

[0464] refer to Fig.14 , the video decoding apparatus 100 may determine the second coding units 1410a and 1410b determined by splitting the first coding unit 1400 in the vertical direction to be processed in the horizontal direction 1410c. The video decoding apparatus 100 may determine the second coding units 1430a and 1430b determined by splitting the first coding unit 1400 in the horizontal direction to be processed in the vertical direction 1430c. The video decoding apparatus 100 may process the second coding units 1450a to 1450d according to a specific order (e.g., a raster scan order or a z scan order 1450e) in which the coding units located in one row are processed and then the coding units located in the next row are processed, wherein the second coding units 1450a to 1450d are determined by splitting the first coding unit 1400 in the vertical and horizontal directions.

[0465] According to an embodiment, the video decoding apparatus 100 may recursively split the coding unit. Fig.14 , the video decoding apparatus 100 may determine a plurality of second coding units 1410a and 1410b, 1430a and 1430b, or 1450a to 1450d by splitting the first coding unit 1400, and recursively split each of the plurality of second coding units 1410a and 1410b, 1430a and 1430b, or 1450a to 1450d. A method of splitting the plurality of second coding units 1410a and 1410b, 1430a and 1430b, or 1450a to 1450d may correspond to a method of splitting the first coding unit 1400. Therefore, each of the plurality of second coding units 1410a and 1410b, 1430a and 1430b, or 1450a to 1450d may be independently split into a plurality of coding units. Reference Fig.14 , the video decoding apparatus 100 may determine the second coding units 1410a and 1410b by splitting the first coding unit 1400 in a vertical direction, and additionally, determine whether each of the second coding units 1410a and 1410b is independently split or not split.

[0466] According to an embodiment, the video decoding apparatus 100 may split the second coding unit 1410a at the left portion in a horizontal direction into third coding units 1420a and 1420b, and may not split the second coding unit 1410b at the right portion.

[0467] According to an embodiment, the order in which the coding units are processed may be determined based on the segmentation process of the coding units. In other words, the order in which the segmented coding units are processed may be determined based on the order in which the coding units before being segmented are processed. The video decoding device 100 may determine the order in which the third coding units 1420a and 1420b determined when the second coding unit 1410a on the left is segmented independently of the second coding unit 1410b on the right are processed. Since the third coding units 1420a and 1420b are determined when the second coding unit 1410a on the left is segmented in the horizontal direction, the third coding units 1420a and 1420b may be processed in the vertical direction 1420c. In addition, since the order of processing the second coding unit 1410a in the left portion and the second coding unit 1410b in the right portion corresponds to the horizontal direction 1410c, after the third coding units 1420a and 1420b included in the second coding unit 1410a in the left portion are processed in the vertical direction 1420c, the second coding unit 1410b in the right portion may be processed. The above description is a related process of processing the order of coding units according to the coding units before being divided, but such a process is not limited to the above embodiment, and any method of independently processing coding units divided into various shapes in a specific order may be used.

[0468] Fig.15 A process of determining that a current coding unit is split into an odd number of coding units when coding units may not be processed by the video decoding apparatus 100 in a specific order according to an embodiment is illustrated.

[0469] According to an embodiment, the video decoding apparatus 100 may determine whether the current coding unit is split into an odd number of coding units based on the obtained block shape information and partition shape information. Fig.15 , the first coding unit 1500 having a square shape may be split into second coding units 1510a and 1510b having a non-square shape, and the second coding units 1510a and 1510b may be independently split into third coding units 1520a and 1520b and 1520c to 1520e, respectively. According to an embodiment, the video decoding apparatus 100 may split the second coding unit 1510a at the left portion from among the second coding units 1510a and 1510b into a horizontal direction to determine a plurality of third coding units 1520a and 1520b, and split the second coding unit 1510b at the right portion into an odd number of third coding units 1520c to 1520e.

[0470] According to an embodiment, the video decoding apparatus 100 may determine whether there are coding units split into odd numbers by determining whether the third coding units 1520a to 1520e can be processed in a specific order. Fig.15 , the video decoding apparatus 100 may determine the third coding units 1520a to 1520e by recursively splitting the first coding unit 1500. The video decoding apparatus 100 may determine whether the coding unit is split into an odd number from among the shapes into which the first coding unit 1500, the second coding units 1510a and 1510b, or the third coding units 1520a to 1520e are split, based on at least one of the block shape information and the partition shape information. For example, the second coding unit 1510b at the right portion from among the second coding units 1510a and 1510b may be split into an odd number of third coding units 1520c to 1520e. The order of processing the plurality of coding units included in the first coding unit 1500 may be a specific order (e.g., a z scan order 1530), and the video decoding apparatus 100 may determine whether the third coding units 1520c to 1520e determined when the second coding unit 1510b at the right portion is split into an odd number satisfy a condition that can be processed according to the specific order.

[0471] According to an embodiment, the video decoding apparatus 100 may determine whether the third coding units 1520a to 1520e included in the first coding unit 1500 satisfy a condition that can be processed according to a specific order, wherein the condition is related to whether at least one of the width and the height of each of the second coding units 1510a and 1510b is divided into two halves according to the boundary of the third coding units 1520a to 1520e. For example, the third coding units 1520a and 1520b determined when the height of the second coding unit 1510a in the left portion and having a non-square shape is divided into two halves satisfy the condition, but it may be determined that the third coding units 1520c to 1520e do not satisfy the condition because the boundary of the third coding units 1520c to 1520e determined when the second coding unit 1510b in the right portion is divided into three coding units does not divide the width or height of the second coding unit 1510b in the right portion into two halves. The video decoding device 100 may determine the disconnection of the scanning order when the condition is not satisfied, and determine that the second coding unit 1510b on the right is split into an odd number of coding units based on the result of the determination. According to an embodiment, the video decoding device 100 may set a specific restriction on the coding unit at a specific position from among the odd number of coding units obtained by splitting the coding unit, and since such restriction or specific position has been described above through various embodiments, the details thereof are not provided.

[0472] Fig.16The process of determining at least one coding unit when the video decoding apparatus 100 splits the first coding unit 1600 according to an embodiment is illustrated. According to an embodiment, the video decoding apparatus 100 may split the first coding unit 1600 based on at least one of the block shape information and the split shape information obtained by the obtainer 105. The first coding unit 1600 having a square shape may be split into four coding units having a square shape or a plurality of coding units having a non-square shape. For example, referring to Fig.16 , when the block shape information indicates that the first coding unit 1600 is a square and the segmentation shape information indicates segmentation into non-square coding units, the video decoding apparatus 100 may segment the first coding unit 1600 into a plurality of non-square coding units. In detail, when the segmentation shape information indicates that an odd number of coding units is determined by segmenting the first coding unit 1600 in a horizontal direction or a vertical direction, the video decoding apparatus 100 may determine second coding units 1610a to 1610c determined by segmenting the first coding unit 1600 having a square shape in a vertical direction or second coding units 1620a to 1620c determined by segmenting the first coding unit 1600 in a horizontal direction as odd number of coding units.

[0473] According to an embodiment, the video decoding apparatus 100 may determine whether the second coding units 1610a to 1610c and 1620a to 1620c included in the first coding unit 1600 satisfy a condition that can be processed in a specific order, wherein the condition is related to whether at least one of the width and the height of the first coding unit 1600 is split into two halves according to the boundary of the second coding units 1610a to 1610c and 1620a to 1620c. Fig.16 , because the boundary of the second coding units 1610a to 1610c determined when the first coding unit 1600 having a square shape is divided in the vertical direction does not divide the width of the first coding unit 1600 into two halves, it can be determined that the first coding unit 1600 does not satisfy the condition that can be processed in a specific order. In addition, because the boundary of the second coding units 1620a to 1620c determined when the first coding unit 1600 having a square shape is divided in the horizontal direction does not divide the height of the first coding unit 1600 into two halves, it can be determined that the first coding unit 1600 does not satisfy the condition that can be processed in a specific order. The video decoding device 100 may determine the disconnection of the scanning order when the condition is not satisfied, and determine that the first coding unit 1600 is divided into an odd number of coding units based on the result of the determination. According to an embodiment, the video decoding device 100 may set a specific restriction on the coding unit at a specific position from among the odd number of coding units obtained by dividing the coding unit, and since such a restriction or a specific position has been described above through various embodiments, the details thereof are not provided.

[0474] According to an embodiment, the video decoding apparatus 100 may determine coding units having various shapes by splitting a first coding unit.

[0475] refer to Fig.16 , the video decoding apparatus 100 may split the first coding unit 1600 having a square shape and the first coding units 1630 or 1650 having a non-square shape into coding units having various shapes.

[0476] Fig.17 It is illustrated that shapes into which the second coding unit may be split by the video decoding apparatus 100 are limited when a second coding unit having a non-square shape determined when the first coding unit 1700 is split satisfies a specific condition according to an embodiment.

[0477] According to an embodiment, the video decoding apparatus 100 may determine that the first coding unit 1700 having a square shape is split into second coding units 1710a and 1710b or 1720a and 1720b having a non-square shape based on at least one of the block shape information and the segmentation shape information obtained by the obtainer 105. The second coding units 1710a and 1710b or 1720a and 1720b may be split independently. Therefore, the video decoding apparatus 100 may determine that the second coding units 1710a and 1710b or 1720a and 1720b are split into a plurality of coding units or are not split based on at least one of the block shape information and the segmentation shape information related to each of the second coding units 1710a and 1710b or 1720a and 1720b. According to an embodiment, the video decoding apparatus 100 may determine the third coding units 1712a and 1712b by splitting the second coding unit 1710a on the left having a non-square shape determined when the first coding unit 1700 is split in the vertical direction in the horizontal direction. However, when the second coding unit 1710a on the left is split in the horizontal direction, the video decoding apparatus 100 may set a restriction that the second coding unit 1710b on the right is not split in the horizontal direction like the second coding unit 1710a on the left. When the third coding units 1714a and 1714b are determined when the second coding unit 1710b on the right is split in the same direction, that is, the horizontal direction, the third coding units 1712a, 1712b, 1714a, and 1714b are determined when the second coding unit 1710a on the left and the second coding unit 1710b on the right are each independently split in the horizontal direction. However, this is the same result as splitting the first coding unit 1700 into four second coding units 1730a to 1730d having a square shape based on at least one of block shape information and partition shape information, and thus may be inefficient in terms of image decoding.

[0478] According to an embodiment, the video decoding apparatus 100 may determine the third coding units 1722a and 1722b or 1724a and 1724b by vertically splitting the second coding unit 1720a or 1720b having a non-square shape determined when the first coding unit 1700 is split in the horizontal direction. However, when one of the second coding units (e.g., the second coding unit 1720a at the top) is split in the vertical direction, the video decoding apparatus 100 may set a restriction that another second coding unit (e.g., the second coding unit 1720b at the bottom) is not split in the vertical direction like the second coding unit 1720a at the top due to the above-mentioned reason.

[0479] Fig.18 A process in which the video decoding apparatus 100 splits a coding unit having a square shape when the split shape information may not indicate that the coding unit is split into four square shapes according to an embodiment is illustrated.

[0480] According to an embodiment, the video decoding device 100 may determine the second coding units 1810a and 1810b or 1820a and 1820b by segmenting the first coding unit 1800 based on at least one of the block shape information and the segmentation shape information. The segmentation shape information may include information about various shapes into which the coding unit may be segmented, but such information about the various shapes may not include information for segmenting the coding unit into four square coding units. According to such segmentation shape information, the video decoding device 100 cannot segment the first coding unit 1800 having a square shape into four second coding units 1830 to 1830d having a square shape. The video decoding device 100 may determine the second coding units 1810a and 1810b or 1820a and 1820b having a non-square shape based on the segmentation shape information.

[0481] According to an embodiment, the video decoding apparatus 100 may independently split each of the second coding units 1810a and 1810b or 1820a and 1820b having a non-square shape. Each of the second coding units 1810a and 1810b or 1820a and 1820b may be split in a specific order via a recursive method, which may be a splitting method corresponding to a method of splitting the first coding unit 1800 based on at least one of block shape information and split shape information.

[0482] For example, the video decoding device 100 may determine the third coding units 1812a and 1812b having a square shape by dividing the second coding unit 1810a on the left in the horizontal direction, or determine the third coding units 1814a and 1814b having a square shape by dividing the second coding unit 1810b on the right in the horizontal direction. In addition, the video decoding device 100 may determine the third coding units 1816a to 1816d having a square shape by dividing both the second coding unit 1810a on the left and the second coding unit 1810b on the right in the horizontal direction. In this case, the coding unit may be determined in the same manner as when the first coding unit 1800 is divided into four second coding units 1830a to 1830d having a square shape.

[0483] As another example, the video decoding device 100 may determine the third coding units 1822a and 1822b having a square shape by splitting the second coding unit 1820a at the top in the vertical direction, and determine the third coding units 1824a and 1824b having a square shape by splitting the second coding unit 1820b at the bottom in the vertical direction. In addition, the video decoding device 100 may determine the third coding units 1826a to 1826d having a square shape by splitting the second coding unit 1820a at the top and the second coding unit 1820b at the bottom in the vertical direction. In this case, the coding unit may be determined in the same manner as when the first coding unit 1800 is split into four second coding units 1830a to 1830d having a square shape.

[0484] Fig.19 It is illustrated that an order of processing a plurality of coding units may be changed according to a process of splitting the coding units according to an embodiment.

[0485] According to an embodiment, the video decoding apparatus 100 may split the first coding unit 1900 based on the block shape information and the segmentation shape information. When the block shape information indicates a square shape and the segmentation shape information indicates that the first coding unit 1900 is split in at least one of the horizontal direction and the vertical direction, the video decoding apparatus 100 may split the first coding unit 1900 to determine the second coding units 1910a and 1910b or 1920a and 1920b. Fig.19, the second coding units 1910a and 1910b or 1920a and 1920b having a non-square shape and determined when the first coding unit 1900 is divided in the horizontal direction or the vertical direction may be independently divided based on the block shape information and the division shape information. For example, the video decoding apparatus 100 may determine the third coding units 1916a to 1916d by dividing in the horizontal direction each of the second coding units 1910a and 1910b generated as the first coding unit 1900 is divided in the vertical direction, or may determine the third coding units 1926a to 1926d by dividing in the horizontal direction the second coding units 1920a and 1920b generated as the first coding unit 1900 is divided in the horizontal direction. As has been described above with reference to Fig.17 A process of splitting the second coding units 1910a and 1910b or 1920a and 1920b is described, and thus details thereof are not provided.

[0486] According to an embodiment, the video decoding apparatus 100 may process the coding units according to a specific order. Fig.14 A feature regarding processing coding units according to a specific order is described, and therefore the details thereof are not provided. Fig.19 , the video decoding apparatus 100 may determine four third coding units 1916a to 1916d or 1926a to 1926d having a square shape by dividing the first coding unit 1900 having a square shape. According to an embodiment, the video decoding apparatus 100 may determine an order of processing the third coding units 1916a to 1916d or 1926a to 1926d based on how the first coding unit 1900 is divided.

[0487] According to an embodiment, the video decoding device 100 can determine the third coding units 1916a to 1916d by splitting the second coding units 1910a and 1910b generated as the first coding unit 1900 is split in the vertical direction in the horizontal direction, and process the third coding units 1916a to 1916d according to the order 1917 of first processing the third coding units 1916a and 1916b included in the second coding unit 1910a on the left in the vertical direction and then processing the third coding units 1916c and 1916d included in the second coding unit 1910b on the right in the vertical direction.

[0488] According to an embodiment, the video decoding device 100 can determine the third coding units 1926a to 1926d by splitting the second coding units 1920a and 1920b generated as the first coding unit 1900 is split in the horizontal direction, and process the third coding units 1926a to 1926d according to the order 1927 of first processing the third coding units 1926a and 1926b included in the second coding unit 1920a at the top in the horizontal direction and then processing the third coding units 1926c and 1926d included in the second coding unit 1920b at the bottom in the horizontal direction.

[0489] refer to Fig.19 , when the second coding units 1910a and 1910b or 1920a and 1920b are both split, third coding units 1916a to 1916d or 1926a to 1926d having a square shape may be determined. When the second coding units 1910a and 1910b determined when the first coding unit 1900 is split in the vertical direction and the second coding units 1920a and 1920b determined when the first coding unit 1900 is split in the horizontal direction are split in different shapes but according to the third coding units 1916a to 1916d and 1926a to 1926d determined thereafter, the first coding unit 1900 is split according to coding units having the same shape. Therefore, even when coding units having the same shape are determined when the coding units are recursively split by different processes based on at least one of the block shape information and the split shape information, the video decoding apparatus 100 may process a plurality of coding units determined in the same shape in different orders.

[0490] Fig. 20 A process of determining a depth of a coding unit as a shape and a size of a coding unit are changed when a plurality of coding units are determined when the coding unit is recursively split according to an embodiment is illustrated.

[0491] According to an embodiment, the video decoding device 100 may determine the depth of the coding unit according to a specific criterion. For example, the specific criterion may be the length of the long side of the coding unit. When the length of the long side of the current coding unit is split to be 2n times shorter than the length of the long side of the coding unit before being split, it can be determined that the depth of the current coding unit is increased by n times the depth of the coding unit before being split, where n>0. Hereinafter, a coding unit with an increased depth is referred to as a coding unit of a lower depth.

[0492] refer to Fig. 20, the video decoding apparatus 100 may determine the second coding unit 2002 and the third coding unit 2004 of the lower depth by splitting the first coding unit 2000 having a square shape based on block shape information indicating a square shape according to an embodiment (for example, the block shape information may indicate "0: square"). When the size of the first coding unit 2000 having a square shape is 2N×2N, the second coding unit 2002 determined by splitting the width and height of the first coding unit 2000 by 1 / 2^1 may have a size of N×N. In addition, the third coding unit 2004 determined by splitting the width and height of the second coding unit 2002 by 1 / 2 may have a size of N / 2×N / 2. In this case, the width and height of the third coding unit 2004 correspond to 1 / 2^2 of the first coding unit 2000. When the depth of the first coding unit 2000 is D, the depth of the second coding unit 2002 having a width and height of 1 / 2^2 of the first coding unit 2000 may be D+1, and the depth of the third coding unit 2004 having a width and height of 1 / 2^2 of the first coding unit 2000 may be D+2.

[0493] According to an embodiment, the video decoding device 100 can determine the second coding unit 2012 or 2022 and the third coding unit 2014 or 2024 by splitting the first coding unit 2010 or 2020 having a non-square shape based on block shape information indicating a non-square shape according to an embodiment (for example, the block shape information may indicate "1: NS_VER" indicating a non-square shape whose height is longer than its width or "2: NS_HOR" indicating a non-square shape whose width is longer than its height).

[0494] The video decoding apparatus 100 may determine the second coding unit (e.g., the second coding unit 2002, 2012, or 2022) by splitting at least one of the width and the height of the first coding unit 2010 having a size of N×2N. In other words, the video decoding apparatus 100 may determine the second coding unit 2002 having a size of N×N or the second coding unit 2022 having a size of N×N / 2 by splitting the first coding unit 2010 in a horizontal direction, or determine the second coding unit 2012 having a size of N / 2×N by splitting the first coding unit 2010 in horizontal and vertical directions.

[0495] The video decoding apparatus 100 may determine the second coding unit (e.g., the second coding unit 2002, 2012, or 2022) by splitting at least one of the width and the height of the first coding unit 2020 having a size of 2N×N. In other words, the video decoding apparatus 100 may determine the second coding unit 2002 having a size of N×N or the second coding unit 2012 having a size of N / 2×N by splitting the first coding unit 2020 in the horizontal direction, or determine the second coding unit 2022 having a size of N×N / 2 by splitting the first coding unit 2010 in the horizontal and vertical directions.

[0496] According to an embodiment, the video decoding apparatus 100 may determine a third coding unit (e.g., a third coding unit 2004, 2014, or 2024) by splitting at least one of a width and a height of the second coding unit 2002 having a size of N×N. In other words, the video decoding apparatus 100 may determine a third coding unit 2004 having a size of N / 2×N / 2, a third coding unit 2014 having a size of N / 22×N / 2, or a third coding unit 2024 having a size of N / 2×N / 22 by splitting the second coding unit 2002 in vertical and horizontal directions.

[0497] According to an embodiment, the video decoding apparatus 100 may determine a third coding unit (e.g., the third coding unit 2004, 2014, or 2024) by splitting at least one of the width and the height of the second coding unit 2022 having a size of N / 2×N. In other words, the video decoding apparatus 100 may determine the third coding unit 2004 having a size of N / 2×N / 2 by splitting the second coding unit 2012 in a horizontal direction, or may determine the third coding unit 2014 having a size of N / 22×N / 2 by splitting the second coding unit 2012 in vertical and horizontal directions.

[0498] According to an embodiment, the video decoding apparatus 100 may determine a third coding unit (e.g., the third coding unit 2004, 2014, or 2024) by splitting at least one of the width and the height of the second coding unit 2022 having a size of N×N / 2. In other words, the video decoding apparatus 100 may determine the third coding unit 2004 having a size of N / 2×N / 2 or the third coding unit 2014 having a size of N / 22×N / 2 by splitting the second coding unit 2022 in a vertical direction, or determine the third coding unit 2024 having a size of N / 2×N / 22 by splitting the second coding unit 2022 in vertical and horizontal directions.

[0499] According to an embodiment, the video decoding apparatus 100 may split a coding unit (e.g., the first coding unit 2000, the second coding unit 2002, or the third coding unit 2004) having a square shape in a horizontal or vertical direction. For example, a first coding unit 2010 having a size of N×2N may be determined by splitting the first coding unit 2000 having a size of 2N×2N in a vertical direction, or a first coding unit 2020 having a size of 2N×N may be determined by splitting the first coding unit 2000 in a horizontal direction. According to an embodiment, when the depth is determined based on the length of the longest side of the coding unit, the depth of the coding unit determined when the first coding unit 2000 having a size of 2N×2N is split in a horizontal or vertical direction may be the same as the depth of the first coding unit 2000.

[0500] According to an embodiment, the width and height of the third coding unit 2014 or 2024 may be 1 / 2^2 of those of the first coding unit 2010 or 2020. When the depth of the first coding unit 2010 or 2020 is D, the depth of the second coding unit 2012 or 2022, which is 1 / 2 of the width and height of the first coding unit 2010 or 2020, may be D+1, and the depth of the third coding unit 2014 or 2024, which is 1 / 2^2 of the width and height of the first coding unit 2010 or 2020, may be D+2.

[0501] Fig.21 An example of distinguishing a depth and a partial index (PID) of a coding unit, which may be determined according to a shape and a size of a coding unit, according to an embodiment is illustrated.

[0502] According to an embodiment, the video decoding apparatus 100 may determine second coding units having various shapes by splitting the first coding unit 2100 having a square shape. Fig.21 , the video decoding apparatus 100 may determine the second coding units 2102a and 2102b, 2104a and 2104b, or 2106a to 2106d by splitting the first coding unit 2100 in at least one of the vertical direction and the horizontal direction according to the split shape information. In other words, the video decoding apparatus 100 may determine the second coding units 2102a and 2102b, 2104a and 2104b, or 2106a to 2106d based on the split shape information of the first coding unit 2100.

[0503] According to an embodiment, the depth of the second coding units 2102a and 2102b, 2104a and 2104b, or 2106a to 2106d determined according to the segmentation shape information of the first coding unit 2100 having a square shape may be determined based on the length of the long side. For example, since the length of one side of the first coding unit 2100 having a square shape is the same as the length of the long side of the second coding units 2102a and 2102b or 2104a and 2104b having a non-square shape, the depths of the first coding unit 2100 and the second coding units 2102a and 2102b or 2104a and 2104b having a non-square shape may be the same, i.e., D. On the other hand, when the video decoding device 100 splits the first coding unit 2100 into four second coding units 2106a to 2106d having a square shape based on the segmentation shape information, the length of one side of the second coding units 2106a to 2106d having a square shape is 1 / 2 of the length of one side of the first coding unit 2100, and the depth of the second coding units 2106a to 2106d can be D+1, that is, a depth lower than the depth D of the first coding unit 2100.

[0504] According to an embodiment, the video decoding apparatus 100 may split a first coding unit 2110 whose height is longer than its width into a plurality of second coding units 2112a and 2112b or 2114a to 2114c in a horizontal direction according to the segmentation shape information. According to an embodiment, the video decoding apparatus 100 may split a first coding unit 2120 whose width is longer than its height into a plurality of second coding units 2122a and 2122b or 2124a to 2124c in a vertical direction according to the segmentation shape information.

[0505] According to an embodiment, the depth of the second coding units 2112a and 2112b, 2114a to 2114c, 2122a and 2122b, or 2124a to 2124c determined according to the segmentation shape information of the first coding unit 2110 or 2120 having a non-square shape may be determined based on the length of the long side. For example, since the length of one side of the second coding units 2112a and 2112b having a square shape is 1 / 2 of the length of the long side of the first coding unit 2110 having a non-square shape in which the height is longer than the width, the depth of the second coding units 2112a and 2112b is D+1, that is, a depth lower than the depth D of the first coding unit 2110 having a non-square shape.

[0506] In addition, the video decoding apparatus 100 may split the first coding unit 2110 having a non-square shape into an odd number of second coding units 2114a to 2114c based on the segmentation shape information. The odd number of second coding units 2114a to 2114c may include second coding units 2114a and 2114c having a non-square shape and a second coding unit 2114b having a square shape. In this case, since the length of the long sides of the second coding units 2114a and 2114c having a non-square shape and the length of one side of the second coding unit 2114b having a square shape are 1 / 2 of the length of one side of the first coding unit 2110, the depth of the second coding units 2114a to 2114b may be D+1, that is, a depth lower than the depth D of the first coding unit 2110. The video decoding apparatus 100 may determine the depth of the coding unit related to the first coding unit 2120 having a non-square shape having a width longer than a height in the same manner as determining the depth of the coding unit related to the first coding unit 2110.

[0507] According to an embodiment, regarding determining a PID for distinguishing coding units, when odd-numbered coding units do not have the same size, the video decoding apparatus 100 may determine the PID based on a size ratio of the coding units. Fig.21 , the second coding unit 2114b located at the center from among the odd-numbered second coding units 2114a to 2114c may have the same width as the second coding units 2114a and 2114c, but has a height twice the height of the second coding units 2114a and 2114c. In this case, the second coding unit 2114b located at the center may include two of the second coding units 2114a and 2114c. Therefore, when the PID of the second coding unit 2114b located at the center is 1 according to the scanning order, the PID of the second coding unit 2114c in the next order may be 3, which has increased by 2. In other words, the value of the PID may be discontinuous. According to an embodiment, the video decoding device 100 may determine whether the odd-numbered coding units have the same size based on the discontinuity of the PID used to distinguish the coding units.

[0508] According to an embodiment, the video decoding apparatus 100 may determine whether a plurality of coding units determined when a current coding unit is split has a specific split shape based on a value of a PID. Fig.21, the video decoding apparatus 100 may determine an even number of second coding units 2112a and 2112b or an odd number of second coding units 2114a to 2114c by segmenting a first coding unit 2110 having a rectangular shape having a height longer than a width. The video decoding apparatus 100 may use a PID indicating each coding unit in order to distinguish a plurality of coding units. According to an embodiment, the PID may be obtained from a sampling point at a specific position (e.g., an upper left sampling point) of each coding unit.

[0509] According to an embodiment, the video decoding device 100 may determine a coding unit at a specific position from among the coding units determined by using a PID for distinguishing the coding units. According to an embodiment, when the segmentation shape information of the first coding unit 2110 having a rectangular shape having a height longer than a width indicates that the first coding unit 2110 is segmented into three coding units, the video decoding device 100 may segment the first coding unit 2110 into three second coding units 2114a to 2114c. The video decoding device 100 may assign a PID to each of the three second coding units 2114a to 2114c. The video decoding device 100 may compare the PIDs of the odd-numbered coding units to determine a center coding unit from among the coding units. The video decoding device 100 may determine, based on the PIDs of the coding units, a second coding unit 2114b having a PID corresponding to a center value from among the PIDs as a coding unit at a center position from among the coding units determined when the first coding unit 2110 is segmented. According to an embodiment, when determining the PID for distinguishing the coding units, when the coding units do not have the same size, the video decoding apparatus 100 may determine the PID based on a size ratio of the coding units. Fig.21, the second coding unit 2114b generated when the first coding unit 2110 is split may have the same width as the second coding units 2114a and 2114c, but may have a height twice the height of the second coding units 2114a and 2114c. In this case, when the PID of the second coding unit 2114b located at the center is 1, the PID of the second coding unit 2114c in the next order may be 3, which has increased by 2. Therefore, when the increase range of the PID is different when uniformly increasing, the video decoding device 100 may determine that the current coding unit is split into a plurality of coding units including coding units having sizes different from other coding units. According to an embodiment, when the segmentation shape information indicates segmentation into an odd number of coding units, the video decoding device 100 may split the current coding unit into a plurality of coding units, wherein the coding unit at a specific position (e.g., a center coding unit) has a size different from other coding units. In this case, the video decoding device 100 may determine the center coding unit having different sizes by using the PID of the coding unit. However, the size or position of the encoding unit at the above-mentioned specific position is specified to describe the embodiment and thus should not be restrictively interpreted, and various PIDs and various positions and sizes of the encoding unit may be used.

[0510] According to an embodiment, the video decoding apparatus 100 may use a specific data unit that starts recursive splitting of a coding unit.

[0511] Fig. 22 It is illustrated that a plurality of coding units are determined according to a plurality of specific data units included in a picture according to an embodiment.

[0512] According to an embodiment, a specific data unit may be defined as a data unit from which at least one of block shape information and segmentation shape information is recursively split. In other words, the specific data unit may correspond to a coding unit of the highest depth used when determining a plurality of coding units by splitting the current picture. Hereinafter, the specific data unit is referred to as a reference data unit for convenience of description.

[0513] According to an embodiment, the reference data unit may indicate a specific size and shape. According to an embodiment, the reference data unit may include M×N sampling points. Here, M and N may be the same and may be an integer expressed as a multiple of 2. In other words, the reference data unit may indicate a square shape or a non-square shape and may be split into an integer number of coding units later.

[0514] According to an embodiment, the video decoding apparatus 100 may divide the current picture into a plurality of reference data units. According to an embodiment, the video decoding apparatus 100 may divide the current picture by using the division shape information about each of the plurality of reference data units, thereby obtaining the plurality of reference data units. The division process of such reference data units may correspond to the division process using a quadtree structure.

[0515] According to an embodiment, the video decoding apparatus 100 may predetermine a minimum size of a reference data unit that can be included in the current picture. Therefore, the video decoding apparatus 100 may determine reference data units of various sizes equal to or larger than the minimum size, and determine at least one coding unit based on the determined reference data unit by using block information and partition shape information.

[0516] refer to Fig. 22 , the video decoding apparatus 100 may use a reference coding unit 2200 having a square shape, or may use a reference coding unit 2202 having a non-square shape. According to an embodiment, the shape and size of the reference coding unit may be determined according to various data units (e.g., a sequence, a picture, a slice, a slice segment, and a maximum coding unit) that may include at least one reference coding unit.

[0517] According to an embodiment, the obtainer 105 of the video decoding apparatus 100 obtains at least one of information about the shape of the reference coding unit and information about the size of the reference coding unit from the bitstream according to various data units. Fig.10 The process of splitting the current coding unit 1000 describes a process of determining at least one coding unit included in the reference coding unit 2200 having a square shape, and has been described above by Fig.11 The process of splitting the current coding unit 1100 or 1150 describes a process of determining at least one coding unit included in the reference coding unit 2200 having a non-square shape, and thus details thereof are not provided.

[0518] According to an embodiment, in order to determine the size and shape of a reference coding unit according to some data units predetermined based on a predetermined condition, the video decoding apparatus 100 may use a PID for distinguishing the size and shape of the reference coding unit. In other words, the obtainer 105 may obtain a PID for distinguishing only the size and shape of the reference coding unit from a bitstream according to a slice, a slice segment, and a maximum coding unit, wherein the slice, the slice segment, and the maximum coding unit are data units (e.g., data units with a size less than or equal to a slice) that meet a predetermined condition from among various data units (e.g., a sequence, a picture, a slice, a slice segment, and a maximum coding unit). The video decoding apparatus 100 may determine the size and shape of the reference data unit according to the data unit that meets the predetermined condition by using the PID. When the information about the shape of the reference coding unit and the information about the size of the reference coding unit are obtained from a bitstream and used according to a data unit with a relatively small size, the use efficiency of the bitstream may be insufficient, and therefore, instead of directly obtaining the information about the shape of the reference coding unit and the information about the size of the reference coding unit, only the PID may be obtained and used. In this case, at least one of the size and shape of the reference coding unit corresponding to the PID indicating the size and shape of the reference coding unit may be predetermined. In other words, the video decoding apparatus 100 may select at least one of a size and a shape of a predetermined reference coding unit according to the PID in order to determine at least one of a size and a shape of a reference coding unit included in a data unit as a criterion for obtaining the PID.

[0519] According to an embodiment, the video decoding device 100 may use at least one reference coding unit included in one maximum coding unit. In other words, the maximum coding unit of the segmented image may include at least one reference coding unit, and the coding unit may be determined when each reference coding unit is recursively segmented. According to an embodiment, at least one of the width and height of the maximum coding unit may be an integer multiple of at least one of the width and height of the reference coding unit. According to an embodiment, the size of the reference coding unit may be equal to the size of the maximum coding unit segmented n times according to the quadtree structure. In other words, the video decoding device 100 may determine the reference coding unit by segmenting the maximum coding unit n times according to the quadtree structure, and segment the reference coding unit based on at least one of the block shape information and the partition shape information according to various embodiments.

[0520] Fig.23 A processing block serving as a criterion for determining an order of reference coding units included in the picture 2300 according to an embodiment is illustrated.

[0521] According to an embodiment, the video decoding device 100 may determine at least one processing block of a split picture. A processing block is a data unit including at least one reference coding unit of a split picture, and at least one reference coding unit included in the processing block may be determined in a specific order. In other words, the determination order of at least one reference coding unit determined in each processing block may correspond to one of various orders for determining the reference coding unit, and may vary according to the processing block. The determination order of the reference coding units determined by the processing block may be one of various orders, such as a raster scanning order, a Z scanning order, an N scanning order, an upper right diagonal scanning order, a horizontal scanning order, and a vertical scanning order, but should not be interpreted limitedly with respect to the scanning order.

[0522] According to an embodiment, the video decoding apparatus 100 may determine the size of at least one processing block included in the image by obtaining information about the size of the processing block. The video decoding apparatus 100 may obtain information about the size of the processing block from the bitstream to determine the size of at least one processing block included in the image. The size of the processing block may be a specific size of a data unit indicated by the information about the size of the processing block.

[0523] According to an embodiment, the obtainer 105 of the video decoding device 100 may obtain information about the size of the processing block from the bitstream according to a specific data unit. For example, the information about the size of the processing block may be obtained from the bitstream in data units of an image, a sequence, a picture, a slice, and a slice segment. In other words, the obtainer 105 may obtain information about the size of the processing block from the bitstream according to such several data units, and the video decoding device 100 may determine the size of at least one processing block of the partitioned picture by using the obtained information about the size of the processing block, wherein the size of the processing block may be an integer multiple of the size of the reference coding unit.

[0524] According to an embodiment, the video decoding apparatus 100 may determine the sizes of the processing blocks 2302 and 2312 included in the picture 2300. For example, the video decoding apparatus 100 may determine the sizes of the processing blocks based on information about the sizes of the processing blocks, the information being obtained from the bitstream. Fig.23 According to an embodiment, the video decoding apparatus 100 may determine the horizontal size of the processing blocks 2302 and 2312 to be four times the horizontal size of the reference coding unit, and determine the vertical size thereof to be four times the vertical size of the reference coding unit. The video decoding apparatus 100 may determine the determination order of at least one reference coding unit in at least one processing block.

[0525] According to an embodiment, the video decoding apparatus 100 may determine each of the processing blocks 2302 and 2312 included in the picture 2300 based on the size of the processing block, and determine a determination order of at least one reference coding unit included in each of the processing blocks 2302 and 2312. According to an embodiment, the determination of the reference coding unit may include determining the size of the reference coding unit.

[0526] According to an embodiment, the video decoding apparatus 100 may obtain information about a determination order of at least one reference coding unit included in at least one processing block from a bitstream, and determine the determination order of at least one coding unit based on the obtained information. The information about the determination order may be defined as an order or direction of determining the reference coding units in the processing block. In other words, the order of determining the reference coding units may be determined independently by processing block.

[0527] According to an embodiment, the video decoding apparatus 100 may obtain information about the determined order of the reference coding units from the bitstream according to a specific data unit. For example, the obtainer 105 may obtain information about the determined order of the reference coding units from the bitstream according to data units such as an image, a sequence, a picture, a slice, a slice segment, and a processing block. Since the information about the determined order of the reference coding units indicates the determined order of the reference coding units in the processing block, the information about the determined order may be obtained according to a specific data unit including an integer number of processing blocks.

[0528] According to an embodiment, the video decoding apparatus 100 may determine at least one reference coding unit based on the determined order.

[0529] According to an embodiment, the obtainer 105 may obtain information about the determined order of the reference coding units from the bitstream as information related to the processing blocks 2302 and 2312, and the video decoding device 100 may determine the order of determining at least one reference coding unit included in the processing blocks 2302 and 2312, and determine at least one reference coding unit included in the picture 2300 according to the determined order of the coding units. Fig.23, the video decoding apparatus 100 may determine determination orders 2304 and 2314 of at least one reference coding unit respectively related to the processing blocks 2302 and 2312. For example, when information about the determination order of the reference coding unit is obtained by processing block, the determination orders of the reference coding units related to the processing blocks 2302 and 2312 may be different from each other. When the determination order 2304 related to the processing block 2302 is a raster scan order, the reference coding units included in the processing block 2302 may be determined according to the raster scan order. On the other hand, when the determination order 2314 related to the processing block 2312 is a reverse order of the raster scan order, the reference coding units included in the processing block 2312 may be determined in a reverse order of the raster scan order.

[0530] According to an embodiment, the video decoding apparatus 100 may decode the determined at least one reference coding unit. The video decoding apparatus 100 may decode an image based on the reference coding unit determined by the above embodiment. Examples of methods for decoding a reference coding unit may include various methods for decoding an image.

[0531] According to an embodiment, the video decoding device 100 may obtain and use block shape information indicating the shape of the current coding unit or segmentation shape information indicating a method of segmenting the current coding unit from a bitstream. The block shape information or segmentation shape information may be included in a bitstream related to various data units. For example, the video decoding device 100 may use block shape information or segmentation shape information included in a sequence parameter set, a picture parameter set, a video parameter set, a slice header, and a slice segment header. In addition, the video decoding device 100 may obtain and use syntax corresponding to the block shape information or segmentation shape information from a bitstream according to a maximum coding unit, a reference coding unit, and a processing block.

[0532] Although the present disclosure has been particularly shown and described with reference to the embodiments of the present disclosure, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims. These embodiments should be considered in a descriptive sense only and not for the purpose of limitation. Therefore, the scope of the present disclosure is defined by the appended claims rather than by the detailed description of the present disclosure, and all differences within the scope will be interpreted as being included in the present disclosure.

[0533] The embodiments of the present disclosure can be written as a computer program and can be implemented by a general-purpose digital computer that uses a computer-readable recording medium to execute the program. Examples of computer-readable recording media include magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.), optical recording media (e.g., CD-ROM or DVD), etc.

Claims

1. A video decoding device, the video decoding device include: An obtainer, the obtainer being configured to: Obtaining motion prediction mode information about a current block from a bitstream; as well as When the obtained motion prediction mode information indicates a bi-prediction mode, obtaining information about a first motion vector of the current block and information about a second motion vector of the current block from the bitstream, and An inter-frame predictor, the inter-frame predictor being configured to: Obtaining a pixel value of a first pixel using a first neighboring pixel determined based on the first motion vector in a first reference picture, and obtaining a pixel value of a second pixel using a second neighboring pixel determined based on the second motion vector in a second reference picture; Using a third neighboring pixel of the first pixel to obtain a first gradient value of the first pixel, and using a fourth neighboring pixel of the second pixel to obtain a second gradient value of the second pixel; determining a first displacement vector in a horizontal direction and a second displacement vector in a vertical direction; as well as obtaining a predicted pixel value of the current block by using a pixel value of the first pixel, a pixel value of the second pixel, the first gradient value, the second gradient value, the first displacement vector, and the second displacement vector, wherein, when a first intermediate value used to determine at least one of the first displacement vector and the second displacement vector is greater than a predetermined value, the at least one of the first displacement vector and the second displacement vector is determined by performing a clipping operation based on a first threshold value and a second threshold value on a second intermediate value, the second intermediate value being obtained using the first intermediate value, and When the first intermediate value is less than or equal to the predetermined value, at least one of the first displacement vector and the second displacement vector is determined to be 0 without performing the clipping operation.

2. A video encoding device, the video encoding device include: An inter-frame predictor, the inter-frame predictor being configured to: When the motion prediction mode for the current block is a bi-prediction mode, determining a first motion vector of the current block and a second motion vector of the current block; Obtaining a pixel value of a first pixel by using a first neighboring pixel determined based on the first motion vector of the current block in a first reference picture, and obtaining a pixel value of a second pixel by using a second neighboring pixel determined based on the second motion vector of the current block in a second reference picture; Using a third neighboring pixel of the first pixel to obtain a first gradient value of the first pixel, and using a fourth neighboring pixel of the second pixel to obtain a second gradient value of the second pixel; determining a first displacement vector in a horizontal direction and a second displacement vector in a vertical direction; as well as obtaining a predicted pixel value of the current block using a pixel value of the first pixel, a pixel value of the second pixel, the first gradient value, the second gradient value, the first displacement vector, and the second displacement vector, and a bitstream generator configured to generate a bitstream including motion prediction mode information about the current block, information about the first motion vector, and information about the second motion vector, wherein, when a first intermediate value used to determine at least one of the first displacement vector and the second displacement vector is greater than a predetermined value, the at least one of the first displacement vector and the second displacement vector is determined by performing a clipping operation based on a first threshold value and a second threshold value on a second intermediate value, the second intermediate value being obtained using the first intermediate value, and When the first intermediate value is less than or equal to the predetermined value, at least one of the first displacement vector and the second displacement vector is determined to be 0 without performing the clipping operation.

Images