Video decoding method and apparatus therefor, and video encoding method and apparatus therefor

Abstract

The disclosure provides a video decoding method and apparatus and a video encoding method and apparatus. A video decoding method is disclosed, by which a first motion vector indicating a first reference block in a first reference picture and a second motion vector indicating a second reference block in a second reference picture are acquired; parameters associated with motion compensation according to a pixel group unit for a current block are acquired based on information of parameters associated with compensation according to a pixel group unit and at least one of parameters associated with a picture including a current picture; for the current block, motion compensation according to a block unit based on the first motion vector and the second motion vector is performed, and motion compensation according to a pixel group unit based on the parameters associated with motion compensation according to a pixel group unit is performed, thereby generating a predicted block and thus restoring the current block. Here, a pixel group can include at least one pixel.

Description

[0001] This application is a divisional application of the invention patent application filed on July 14, 2017, with application number "201780056738.1" and titled "Video Decoding Method and Apparatus Thereof and Video Encoding Method and Apparatus Thereof". Technical Field

[0002] This disclosure relates to a video decoding method and a video coding method. More specifically, this disclosure relates to video decoding and video coding that perform inter-frame prediction in a bidirectional motion prediction mode. Background Technology

[0003] As hardware for reproducing and storing high-resolution or high-quality video content is developed and made available, the demand for video codecs for efficiently encoding or decoding such content has increased. In traditional video codecs, tree-structured coding units encode video according to a limited coding method.

[0004] Spatial domain image data is transformed into frequency domain coefficients via a frequency transform. According to the video codec, the image is divided into blocks of predetermined size, a discrete cosine transform (DCT) is performed on each block, and the frequency coefficients are encoded on a block-by-block basis for rapid calculation of the frequency transform. Compared to spatial domain image data, frequency domain coefficients are easier to compress. Specifically, since spatial domain image pixel values ​​are represented by prediction errors from inter-frame or intra-frame predictions via the video codec, a large amount of data can be transformed to zero when performing a frequency transform on the prediction errors. According to the video codec, the amount of data can be reduced by using a small amount of data instead of continuously and repeatedly generated data. Summary of the Invention

[0005] Technical issues

[0006] According to various embodiments, in bidirectional motion prediction mode, the predicted pixel value of the current block can be generated not only by using the pixel values ​​of the first reference block of the first reference frame and the pixel values ​​of the second reference block of the second reference frame, but also by using the first gradient value of the first reference block and the second gradient value of the second reference block. Therefore, since a predicted block similar to the original block can be generated, encoding and decoding efficiency can be improved.

[0007] When pixel-level motion compensation is performed, the first gradient value of the first reference block and the second gradient value of the second reference block are used. The parameters used when pixel-level motion compensation is performed are transmitted via a bit stream or obtained by using image-related parameters. Therefore, pixel-level motion compensation can be performed adaptively on the image.

[0008] A computer-readable recording medium is provided, which records a program for performing methods according to various embodiments.

[0009] Hereinafter, various aspects of the various embodiments are not limited thereto, and other aspects will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments.

[0010] Technical solution

[0011] The various aspects of this disclosure are not limited thereto, and other aspects will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments.

[0012] According to one aspect of this disclosure, a video decoding method includes: obtaining motion prediction mode information about a current block in a current frame from a bitstream; when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtaining a first motion vector and a second motion vector from the bitstream, wherein the first motion vector indicates a first reference block of the current block in a first reference frame, and the second motion vector indicates a second reference block of the current block in a second reference frame; obtaining parameters related to pixel group unit motion compensation of the current block based on information about parameters related to pixel group unit motion compensation of the current block obtained from the bitstream and at least one of parameters related to an image including the current frame; generating a prediction block of the current block by performing block unit motion compensation for the current block based on the first motion vector and the second motion vector and performing pixel group unit motion compensation based on the parameters related to pixel group unit motion compensation; obtaining a residual block of the current block from the bitstream; and reconstructing the current block based on the prediction block and the residual block, wherein the pixel group includes at least one pixel.

[0013] The video decoding method may further include: determining whether to perform pixel group unit motion compensation based on at least one of flag information, the size of the current block, the prediction direction, the magnitude of the motion vector, the difference in frame sequence count (POC) between the reference frame and the current frame, and the availability of a predetermined encoding / decoding tool, wherein the flag information is obtained from the bitstream and is related to whether to perform pixel group unit motion compensation, wherein the step of generating a prediction block may include: generating a prediction block of the current block by determining to perform pixel group unit motion compensation.

[0014] The steps of obtaining parameters related to pixel group unit motion compensation may include: obtaining a descaling offset value for after interpolation or gradient operation based on at least one of the bit depth of the sample, the input range of the filter used for interpolation or gradient operation, and the coefficients of the filter; and the steps of generating a prediction block for the current block may include: performing descaling on pixels included in the first reference block and the second reference block after interpolation or gradient operation by using the descaling offset value.

[0015] The step of obtaining parameters related to unit motion compensation of pixel groups may include: obtaining regularization parameters related to displacement vectors per unit time in the horizontal or vertical direction based on information obtained from the bitstream regarding parameters related to displacement vectors per unit time in the horizontal or vertical direction, bit depth of the sample, size of the group of frames (GOP), motion vector, parameters related to the time distance between the reference frame and the current frame, frame rate, setting parameters related to the coding prediction structure, and prediction direction, and the step of generating a prediction block for the current block may include: determining the displacement vectors per unit time in the horizontal or vertical direction based on the regularization parameters related to displacement vectors per unit time in the horizontal or vertical direction by using the gradient values ​​of pixels in a first window having a specific size and including a first pixel group included in a first reference block, the gradient values ​​of pixels in a second window having a specific size and including a second pixel group included in a second reference block, the pixel values ​​of pixels in the first window, and the pixel values ​​of pixels in the second window.

[0016] The steps of obtaining parameters related to pixel group unit motion compensation may include: obtaining parameters related to the size of the window used to calculate the displacement vector per unit time based on at least one of the following: information about window size obtained from the bitstream, layer depth of the frame, size of the GOP, image resolution, parameters related to the time distance between the reference frame and the current frame, frame rate, motion vector, setting parameters related to the coded prediction structure, and prediction direction; and the steps of generating the prediction block of the current block may include: determining the displacement vector per unit time in the horizontal or vertical direction based on the parameters related to the window size by using the gradient values ​​of pixels in a first window having a specific size and including a first pixel group included in a first reference block, the gradient values ​​of pixels in a second window having a specific size and including a second pixel group included in a second reference block, the pixel values ​​of pixels in the first window, and the pixel values ​​of pixels in the second window.

[0017] A pixel group may include multiple pixels. The step of obtaining parameters related to pixel group unit motion compensation may include: obtaining parameters related to the size of the pixel group based on at least one of information about the size of the pixel group obtained from the bitstream, image resolution, and frame rate. The step of generating a prediction block for the current block may include: generating a prediction block for the current block by performing block unit motion compensation based on a first motion vector and a second motion vector and performing pixel group unit motion compensation based on parameters related to the size of the pixel group.

[0018] According to another aspect of this disclosure, a video decoding apparatus includes: an acquirer configured to: acquire motion prediction mode information about a current block in a current frame from a bitstream; when the acquired motion prediction mode information indicates a bidirectional motion prediction mode, acquire from the bitstream a first motion vector indicating a first reference block in a first reference frame and a second motion vector indicating a second reference block in a second reference frame; acquire parameters related to pixel group unit motion compensation for the current block based on information about parameters related to pixel group unit motion compensation for the current block obtained from the bitstream and at least one of parameters related to an image including the current frame; and acquire a residual block of the current block from the bitstream; an inter-frame predictor configured to generate a prediction block of the current block by performing block unit motion compensation for the current block based on the first motion vector and the second motion vector and performing pixel group unit motion compensation based on the parameters related to pixel group unit motion compensation; and a decoder configured to reconstruct the current block based on the prediction block and the residual block, wherein the pixel group includes at least one pixel.

[0019] The inter-frame predictor can also be configured to determine whether to perform pixel group unit motion compensation based on at least one of flag information, the size of the current block, the prediction direction, the magnitude of the motion vector, the difference in frame sequence count (POC) between the reference frame and the current frame, and the availability of a predetermined encoding / decoding tool, wherein the flag information is obtained from the bitstream and is related to whether to perform pixel group unit motion compensation; and to generate a predicted block of the current block by performing pixel group unit motion compensation based on the determination.

[0020] The inter-frame predictor can also be configured to: obtain a descaling offset value for the interpolation or gradient operation based on at least one of the bit depth of the sample, the input range of the filter for the interpolation or gradient operation, and the coefficients of the filter, and perform descaling on the pixels included in the first and second reference blocks after the interpolation or gradient operation by using the descaling offset value.

[0021] The inter-frame predictor can also be configured to: obtain regularization parameters related to the displacement vector per unit time in the horizontal or vertical direction based on information obtained from the bitstream regarding parameters related to the displacement vector per unit time in the horizontal or vertical direction, bit depth of the sample, size of the group of frames (GOP), motion vector, parameters related to the time distance between the reference frame and the current frame, frame rate, setting parameters related to the coding prediction structure, and prediction direction, and based on the regularization parameters related to the displacement vector per unit time in the horizontal or vertical direction, determine the displacement vector per unit time in the horizontal or vertical direction by using the gradient values ​​of pixels in a first window having a specific size and including a first pixel group included in a first reference block, the gradient values ​​of pixels in a second window having a specific size and including a second pixel group included in a second reference block, the pixel values ​​of pixels in the first window, and the pixel values ​​of pixels in the second window.

[0022] The acquirer can also be configured to: obtain parameters related to the size of the window used to calculate the displacement vector per unit time based on at least one of the following: information about the window size obtained from the bitstream, the layering depth of the frame, the size of the GOP, the image resolution, parameters related to the time distance between the reference frame and the current frame, the frame rate, the motion vector, setting parameters related to the coding prediction structure, and the prediction direction; and the inter-frame predictor can also be configured to: determine the displacement vector per unit time in the horizontal or vertical direction based on the parameters related to the window size by using the gradient values ​​of pixels in a first window having a specific size and including a first pixel group included in a first reference block, the gradient values ​​of pixels in a second window having a specific size and including a second pixel group included in a second reference block, the pixel values ​​of pixels in the first window, and the pixel values ​​of pixels in the second window.

[0023] A pixel group may include multiple pixels, and the inter-frame predictor may also be configured to: obtain parameters related to the size of the pixel group based on at least one of information about the size of the pixel group obtained from the bitstream, image resolution, and frame rate, and generate a predicted block of the current block by performing block-unit motion compensation based on a first motion vector and a second motion vector and performing pixel-group-unit motion compensation based on the parameters related to the size of the pixel group.

[0024] According to another aspect of this disclosure, a video coding method includes: obtaining a prediction block, a first motion vector, a second motion vector, and parameters related to pixel group unit motion compensation for a current block by performing block-unit motion compensation and pixel group unit motion compensation on the current block; generating a bitstream including information related to the first and second motion vectors, and motion prediction mode information indicating that the motion prediction mode for the current block is a bidirectional motion prediction mode, wherein the pixel group includes at least one pixel, the first motion vector is a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, the second motion vector is a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block, the parameters related to pixel group unit motion compensation for the current block are obtained from parameters related to an image including the current frame when performing pixel group unit motion compensation on the current block, or the parameters related to pixel group unit motion compensation for the current block are determined when performing pixel group unit motion compensation on the current block, and information about the determined parameters related to pixel group unit motion compensation is included in the bitstream.

[0025] According to another aspect of this disclosure, a video encoding apparatus includes: an inter-frame predictor configured to obtain a predicted block of the current block, a first motion vector, a second motion vector, and parameters related to the pixel group unit motion compensation by performing block-unit motion compensation and pixel group unit motion compensation on the current block; and a bitstream generator configured to generate a bitstream including information related to the first and second motion vectors, and motion prediction mode information indicating that the motion prediction mode for the current block is a bidirectional motion prediction mode, wherein the pixel group includes at least one pixel, the first motion vector is a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, the second motion vector is a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block, and the parameters related to the pixel group unit motion compensation for the current block are obtained from parameters related to an image including the current frame when performing pixel group unit motion compensation on the current block, or the parameters related to the pixel group unit motion compensation for the current block are determined when performing pixel group unit motion compensation on the current block, and information about the determined parameters related to the pixel group unit motion compensation is included in the bitstream.

[0026] According to another aspect of this disclosure, a computer-readable medium records a program for performing the video decoding method.

[0027] Beneficial effects

[0028] According to various embodiments, encoding and decoding efficiency can be improved by predicting values ​​similar to those of the original block of the current block through inter-frame prediction of the current block using the gradient values ​​of a reference block of a reference frame in bidirectional motion prediction mode. Attached Figure Description

[0029] Figure 1a This is a block diagram of a video decoding device according to various embodiments.

[0030] Figure 1b This is a flowchart of a video decoding method according to various embodiments.

[0031] Figure 1c It is a block diagram of a video encoding device according to various embodiments.

[0032] Figure 1d This is a flowchart of a video encoding method according to various embodiments.

[0033] Figure 1e This is a block diagram of an image decoder according to various embodiments.

[0034] Figure 1f This is a block diagram of an image encoder according to various embodiments.

[0035] Figure 2 This is a reference diagram illustrating block-based bidirectional motion prediction and compensation processing according to an embodiment.

[0036] Figures 3a to 3c This is a reference diagram illustrating the process of performing pixel-unit motion compensation according to an embodiment.

[0037] Figure 4 This is a reference diagram illustrating the process of calculating gradient values ​​in the horizontal direction and gradient values ​​in the vertical direction, according to an embodiment.

[0038] Figure 5 This is a reference diagram according to another embodiment for describing the process of calculating gradient values ​​in the horizontal direction and gradient values ​​in the vertical direction.

[0039] Figure 6a and Figure 6b This is a diagram illustrating a process for determining gradient values ​​in the horizontal and vertical directions using a one-dimensional (1D) filter, according to an embodiment.

[0040] Figures 7a to 7e This is a table showing filter coefficients for a filter used to determine pixel values ​​at fractional pixel positions in fractional pixel units, as well as gradient values ​​in the horizontal and vertical directions, according to an embodiment.

[0041] Figure 8aThis is a reference diagram according to an embodiment for describing the process of determining the horizontal and vertical displacement vectors of a pixel.

[0042] Figure 8b This is a reference diagram according to an embodiment for describing the process of determining the horizontal and vertical displacement vectors of a pixel group.

[0043] Figure 9a This is a diagram illustrating, according to an embodiment, the process of adding an offset value after filtering is performed and the process of determining gradient values ​​in the horizontal or vertical direction by performing descaling.

[0044] Figure 9b This is a diagram used to describe the range required to determine the horizontal and vertical displacement vectors during the processing of pixel-unit motion compensation for the current block.

[0045] Figure 9c and Figure 9d This is a diagram illustrating the range of regions used during motion compensation processing performed on a pixel-by-pixel basis, according to various embodiments.

[0046] Figure 9e This is a diagram illustrating the process of determining the horizontal and vertical displacement vectors without expanding the reference block.

[0047] Figure 9f This is a diagram illustrating the process for obtaining candidates for time motion vector predictors, wherein pixel group unit motion compensation is considered in the process.

[0048] Figure 10 This illustrates a process for determining at least one coding unit when the current coding unit is divided, according to an embodiment.

[0049] Figure 11 The illustration shows the process of determining at least one coding unit when coding units with non-square shapes are divided, according to an embodiment.

[0050] Figure 12 The process of dividing the coding unit based on at least one of block shape information and partition shape information according to an embodiment is illustrated.

[0051] Figure 13 A method for determining a specific coding unit from an odd number of coding units is shown according to an embodiment.

[0052] Figure 14 This illustrates the order in which multiple coding units are processed when multiple coding units are determined during the partitioning of the current coding unit, according to an embodiment.

[0053] Figure 15 The illustration shows the process of determining that the current coding unit is divided into an odd number of coding units when coding units cannot be processed in a specific order, according to an embodiment.

[0054] Figure 16 The illustration shows the process of determining at least one coding unit when the first coding unit is divided, according to an embodiment.

[0055] Figure 17 The embodiment shows that when a second coding unit having a non-square shape, determined when the first coding unit is divided, satisfies predetermined conditions, the shape into which the second coding unit can be divided is defined.

[0056] Figure 18 The illustration shows the process of dividing a coding unit with a square shape when the division shape information fails to indicate that the coding unit should be divided into four square shapes, according to an embodiment.

[0057] Figure 19 The order in which multiple coding units are processed according to an embodiment can be changed depending on the process of dividing the coding units.

[0058] Figure 20 The illustration shows the process of determining the depth of a coding unit when the shape and size of the coding unit are changed, as multiple coding units are determined during recursive division according to an embodiment.

[0059] Figure 21 The diagram illustrates a partial index (PID) for distinguishing depth and coding units, which can be determined based on the shape and size of the coding unit, according to an embodiment.

[0060] Figure 22 The multiple encoding units shown in the embodiment are determined based on multiple specific data units included in the image.

[0061] Figure 23 The diagram illustrates a standard processing block according to an embodiment, which serves as a step for determining the order of reference coding units included in a frame. Detailed Implementation

[0062] Optimal mode

[0063] According to one aspect of this disclosure, a video decoding method includes: obtaining motion prediction mode information about a current block in a current frame from a bitstream; when the obtained motion prediction mode information indicates a bidirectional motion prediction mode, obtaining a first motion vector and a second motion vector from the bitstream, wherein the first motion vector indicates a first reference block of the current block in a first reference frame, and the second motion vector indicates a second reference block of the current block in a second reference frame; obtaining parameters related to pixel group unit motion compensation of the current block based on information about parameters related to pixel group unit motion compensation of the current block obtained from the bitstream and at least one of parameters related to an image including the current frame; generating a prediction block of the current block by performing block unit motion compensation for the current block based on the first motion vector and the second motion vector and performing pixel group unit motion compensation based on the parameters related to pixel group unit motion compensation; obtaining a residual block of the current block from the bitstream; and reconstructing the current block based on the prediction block and the residual block, wherein the pixel group includes at least one pixel.

[0064] According to another aspect of this disclosure, a video decoding apparatus includes: an acquirer configured to: acquire motion prediction mode information about a current block in a current frame from a bitstream; when the acquired motion prediction mode information indicates a bidirectional motion prediction mode, acquire from the bitstream a first motion vector indicating a first reference block in a first reference frame and a second motion vector indicating a second reference block in a second reference frame; acquire parameters related to pixel group unit motion compensation for the current block based on information about parameters related to pixel group unit motion compensation for the current block obtained from the bitstream and at least one of parameters related to an image including the current frame; and acquire a residual block of the current block from the bitstream; an inter-frame predictor configured to generate a prediction block of the current block by performing block unit motion compensation for the current block based on the first motion vector and the second motion vector and performing pixel group unit motion compensation based on the parameters related to pixel group unit motion compensation; and a decoder configured to reconstruct the current block based on the prediction block and the residual block, wherein the pixel group includes at least one pixel.

[0065] According to another aspect of this disclosure, a video coding method includes: obtaining a prediction block, a first motion vector, a second motion vector, and parameters related to pixel group unit motion compensation for a current block by performing block-unit motion compensation and pixel group unit motion compensation on the current block; generating a bitstream including information related to the first and second motion vectors, and motion prediction mode information indicating that the motion prediction mode for the current block is a bidirectional motion prediction mode, wherein the pixel group includes at least one pixel, the first motion vector is a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, the second motion vector is a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block, the parameters related to pixel group unit motion compensation for the current block are obtained from parameters related to an image including the current frame when performing pixel group unit motion compensation on the current block, or the parameters related to pixel group unit motion compensation for the current block are determined when performing pixel group unit motion compensation on the current block, and information about the determined parameters related to pixel group unit motion compensation is included in the bitstream.

[0066] According to another aspect of this disclosure, a video encoding apparatus includes: an inter-frame predictor configured to obtain a predicted block of the current block, a first motion vector, a second motion vector, and parameters related to the pixel group unit motion compensation by performing block-unit motion compensation and pixel group unit motion compensation on the current block; and a bitstream generator configured to generate a bitstream including information related to the first and second motion vectors, and motion prediction mode information indicating that the motion prediction mode for the current block is a bidirectional motion prediction mode, wherein the pixel group includes at least one pixel, the first motion vector is a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, the second motion vector is a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block, and the parameters related to the pixel group unit motion compensation for the current block are obtained from parameters related to an image including the current frame when performing pixel group unit motion compensation on the current block, or the parameters related to the pixel group unit motion compensation for the current block are determined when performing pixel group unit motion compensation on the current block, and information about the determined parameters related to the pixel group unit motion compensation is included in the bitstream.

[0067] According to another aspect of this disclosure, a computer-readable medium records a program for performing the video decoding method.

[0068] Open mode

[0069] In the following text, “image” can refer to a still image or a moving image of a video (i.e., the video itself).

[0070] In the following text, "sample point" refers to a sampling location assigned to an image and the data that will be processed. For example, pixels in a spatial domain image can be samples.

[0071] In the following text, "current block" may refer to a block of an image that will be encoded or decoded.

[0072] Figure 1a This is a block diagram of a video decoding device according to various embodiments.

[0073] Video decoding device 100 according to various embodiments includes an acquirer 105, an inter-frame predictor 110, and a reconstructor 125.

[0074] The acquirer 105 receives a bit stream that includes information about the prediction mode of the current block, information indicating the motion prediction mode of the current block, and information about the motion vector.

[0075] The acquirer 105 can obtain information about the prediction mode of the current block, information indicating the motion prediction mode of the current block, and information about the motion vector from the received bitstream. Furthermore, the acquirer 105 can obtain a reference frame index from the bitstream indicating a reference frame among previously decoded frames.

[0076] When the prediction mode of the current block is inter-frame prediction mode, the inter-frame predictor 110 performs inter-frame prediction on the current block. In other words, the inter-frame predictor 110 can generate the predicted pixel value of the current block by using at least one of the frames decoded before the current frame that includes the current block. For example, when the motion prediction mode of the current block is bidirectional motion prediction mode, the inter-frame predictor 110 can generate the predicted pixel value of the current block by using two frames decoded before the current frame. In other words, when the information about the motion prediction mode obtained from the bitstream indicates bidirectional motion prediction mode, the inter-frame predictor 110 can generate the predicted pixel value of the current block by using the two frames decoded before the current frame.

[0077] The inter-frame predictor 110 may include a block unit motion compensator 115 and a pixel group unit motion compensator 120.

[0078] The block-unit motion compensator 115 can perform motion compensation on a block-by-block basis for the current block.

[0079] The block unit motion compensator 115 can determine at least one reference frame from previously decoded frames using a reference frame index obtained from the bitstream. Here, the reference frame index can represent a reference frame index for each of the prediction directions, including the L0 direction and the L1 direction. Specifically, the reference frame index for the L0 direction can represent an index indicating a reference frame among the frames included in the L0 reference frame list, and the reference frame index for the L1 direction can represent an index indicating a reference frame among the frames included in the L1 reference frame list.

[0080] The block-unit motion compensator 115 can determine the reference block of the current block by using information about motion vectors received from the bitstream, wherein the reference block is located within the at least one reference frame. Here, the corresponding block in the reference frame that corresponds to the current block in the current frame can be the reference block. In other words, the block-unit motion compensator 115 can determine the reference block of the current block by using a motion vector from the current block that indicates the reference block. Here, the motion vector represents a vector indicating the displacement between the reference coordinates of the current block in the current frame and the reference coordinates of the reference block in the reference frame. For example, when the top-left coordinate of the current block is (1,1) and the top-left coordinate of the reference block is (3,3), the motion vector could be (2,2).

[0081] Here, the information about the motion vector may include the difference value of the motion vector. The block unit motion compensator 115 can reconstruct the motion vector by using the prediction factor of the motion vector and the difference value of the motion vector obtained from the bitstream, and can determine the reference block of the current block located in the at least one reference frame by using the reconstructed motion vector. Here, the difference value of the motion vector may represent the difference value of the motion vector of the reference frame associated with each of the prediction directions, including the L0 direction and the L1 direction. Here, the difference value of the motion vector for the L0 direction may represent the difference value of the motion vector indicating the reference block in the reference frame included in the L0 reference frame list, and the difference value of the motion vector for the L1 direction may represent the difference value of the motion vector indicating the reference block in the reference frame included in the L1 reference frame list.

[0082] The block-unit motion compensator 115 can perform motion compensation on a block-by-block basis by using the pixel values ​​of a reference block. Alternatively, the block-unit motion compensator 115 can perform motion compensation on a block-by-block basis by using the pixel values ​​of a reference pixel in the reference block that corresponds to the current pixel in the current block. Here, the reference pixel can be a pixel included in the reference block, and the corresponding pixel in the current block can be a reference pixel.

[0083] The block-unit motion compensator 115 can perform motion compensation on a block-by-block basis by using multiple reference blocks included in multiple reference frames. For example, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the block-unit motion compensator 115 can determine two reference frames from previously decoded frames and determine two reference blocks included in said two reference frames.

[0084] The block-unit motion compensator 115 can perform motion compensation on a block-by-block basis by using the pixel values ​​of two reference pixels from the two reference blocks. The block-unit motion compensator 115 can generate a block-by-block motion compensation value by performing motion compensation on a block-by-block basis by using the average or weighted sum of the pixel values ​​of the two reference pixels.

[0085] The reference position of the reference block can be an integer pixel position, but is not limited to this, and can also be a fractional pixel position. Here, an integer pixel can represent a pixel whose position component is an integer, and can be a pixel at an integer pixel position. A fractional pixel can represent a pixel whose position component is a fraction, and can be a pixel at a fractional pixel position.

[0086] For example, when the top-left coordinate of the current block is (1, 1) and the motion vector is (2.5, 2.5), the top-left coordinate of the reference block in the reference frame could be (3.5, 3.5). Here, the position of the fractional pixel can be determined in units of 1 / 4 pel or 1 / 16 pel, where pel represents a pixel element. Alternatively, the position of the fractional pixel can be determined in various fractional pel units.

[0087] When the reference position of the reference block is the position of a fractional pixel, the block unit motion compensator 115 can generate the pixel value of the first pixel in the first reference block indicated by the first motion vector and the pixel value of the second pixel in the second reference block indicated by the second motion vector by applying an interpolation filter to a first neighboring region of the first pixel included in the pixels of the first reference block indicated by the first motion vector and a second neighboring region of the second pixel included in the pixels of the second reference block indicated by the second motion vector.

[0088] In other words, the pixel value of a reference pixel in a reference block can be determined by using the pixel values ​​of neighboring pixels whose components are integers in a specific direction. Here, the specific direction can be either horizontal or vertical.

[0089] For example, the block-unit motion compensator 115 can determine the pixel value of a reference pixel by filtering the pixel values ​​of pixels whose components in a particular direction are integers using an interpolation filter, and determine a block-based motion compensation value for the current block using the pixel value of the reference pixel. The block-based motion compensation value is determined by using the average or weighted sum of the reference pixels. Here, the interpolation filter can be an M-tap interpolation filter based on Discrete Cosine Transform (DCT). The coefficients of the DCT-based M-tap interpolation filter can be derived from DCT and Inverse DCT (IDCT). Here, the coefficients of the interpolation filter can be filter coefficients scaled to integer coefficients to reduce real-number operations during filtering. Here, the interpolation filter can be a one-dimensional (1D) interpolation filter in the horizontal or vertical direction. For example, when the position of a pixel is represented by orthogonal x and y coordinate components, the horizontal direction can be a direction parallel to the x-axis. The vertical direction can be a direction parallel to the y-axis.

[0090] The block unit motion compensator 115 can first perform filtering on the pixel value of the pixel at the integer position using a 1D interpolation filter in the vertical direction, and then perform filtering on the value generated by the filtering using a 1D interpolation filter in the horizontal direction to determine the pixel value of the reference pixel at the fractional pixel position.

[0091] Meanwhile, the value obtained by filtering when the scaled filter coefficients are used can be higher than the value obtained by filtering when the unscaled filter is used. Therefore, the block unit motion compensator 115 can perform descaling for the value obtained by filtering.

[0092] The block unit motion compensator 115 can perform descaling after filtering the pixel values ​​of pixels at integer positions using a 1D interpolation filter in the vertical direction. Here, descaling may include a right bit shift according to the descaling bit number. The descaling bit number can be determined based on the bit depth of the sample points of the input image. For example, the descaling bit number can be a value obtained by subtracting 8 from the bit depth of the sample points.

[0093] Furthermore, the block unit motion compensator 115 can perform filtering on the pixel values ​​of pixels at integer positions using a 1D interpolation filter in the vertical direction, and filtering on the values ​​generated by the filtering using a 1D interpolation filter in the horizontal direction, and then perform descaling. Here, descaling may include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the number of scaling bits of the 1D interpolation filter in the vertical direction, the number of scaling bits of the 1D interpolation filter in the horizontal direction, and the bit depth of the sample. For example, when the number of scaling bits p of the 1D interpolation filter in the vertical direction is 6, the number of scaling bits q of the 1D interpolation filter in the horizontal direction is 6, and the bit depth of the sample is b, the number of descaling bits can be p + q + 8 - b, that is, 20 - b.

[0094] When the block unit motion compensator 115 performs a right bit shift based on the descaling bit count after filtering for pixels with integer components in a specific direction using a 1D interpolation filter, rounding errors may occur. Therefore, the block unit motion compensator 115 can perform descaling after filtering for pixels with integer components in a specific direction using a 1D interpolation filter and then adding an offset value. Here, the offset value can be 2^(descaling bit count - 1).

[0095] The pixel group unit motion compensator 120 can generate pixel group unit motion compensation values ​​by performing motion compensation on the current block in units of pixels. When the motion prediction mode of the current block is bidirectional motion prediction mode, the pixel group unit motion compensator 120 can generate pixel group unit motion compensation values ​​by performing pixel group unit motion compensation on the current block.

[0096] The pixel group unit motion compensator 120 can generate a motion compensation value in pixels by performing pixel group unit motion compensation on the current block based on the optical flow of the pixel groups of the first reference frame and the pixel groups of the second reference frame. (See below for further details.) Figure 3a Describe optical flow.

[0097] The pixel group unit motion compensator 120 can generate pixel group-based motion compensation values ​​by performing motion compensation on a pixel group basis for a reference block included in the current block. A pixel group may include at least one pixel. For example, a pixel group may be a single pixel. Alternatively, a pixel group may be a plurality of pixels including at least two pixels. A pixel group may be a plurality of pixels included in a block of size K×K (K is an integer).

[0098] The pixel group unit motion compensator 120 can obtain parameters related to the size of the pixel group based on at least one of image resolution, frame rate, and information about the size of the pixel group obtained from the bitstream. The pixel group unit motion compensator 120 can determine the pixel group based on the parameters related to the size of the pixel group, and can perform pixel group unit motion compensation for the current block based on the determined pixel group.

[0099] The pixel group unit motion compensator 120 can determine the size of the pixel group based on the resolution of the image. For example, when the image resolution is higher than a certain resolution, the size of the pixel group can be determined to be larger than the size of the pixel group corresponding to the specific resolution.

[0100] The pixel group unit motion compensator 120 can determine the size of the pixel group based on the frame rate. For example, when the frame rate is higher than a certain frame rate, the pixel group unit motion compensator 120 can determine the size of the pixel group to be larger than the size of the pixel group corresponding to the specific frame rate.

[0101] The pixel group unit motion compensator 120 can determine the size of the pixel group based on the image resolution and the image frame rate. For example, when the image resolution is higher than the specific resolution and the frame rate is higher than the specific frame rate, the pixel group unit motion compensator 120 can determine the size of the pixel group to be larger than the size of the pixel group corresponding to the specific resolution and the specific frame rate.

[0102] The pixel group unit motion compensator 120 can perform motion compensation on a pixel group basis, comprising multiple pixels, thereby reducing the encoding / decoding complexity compared to performing motion compensation on a pixel-by-pixel basis at high image resolutions. Furthermore, the pixel group unit motion compensator 120 can perform motion compensation on a pixel group basis, comprising multiple pixels, thereby reducing the encoding / decoding complexity compared to performing motion compensation on a pixel-by-pixel basis at high frame rates.

[0103] The acquirer 105 can obtain information about the size of the pixel group included in the bitstream. When the size of the pixel group is K×K, the information about the size of the pixel group can be information indicating the height or width K. The information about the size of the pixel group can be included in the high-level syntax carrier.

[0104] The pixel group unit motion compensator 120 can determine at least one pixel group region and perform motion compensation on the pixel group region, wherein the pixel group region includes pixels with similar pixel values ​​among a plurality of pixels included in the pixel group. Here, the pixel group region including pixels with similar pixel values ​​is likely to be the same object and is likely to have similar motion, and the pixel group motion compensator 120 is able to perform more accurate motion compensation at the pixel group unit level.

[0105] Meanwhile, when the motion prediction mode information indicates a bidirectional motion prediction mode, pixel group unit motion compensation is performed, but pixel group unit motion compensation is not always performed, but can be performed selectively.

[0106] The pixel group unit motion compensator 120 can determine whether to perform motion compensation on a pixel group basis based on at least one of the following: pixel group unit motion flag information obtained from the bitstream, the size of the current block, the prediction direction, the magnitude of the motion vector, the frame sequence count (POC) difference between the reference frame and the current frame, and the availability of a particular encoding / decoding tool. The pixel group unit motion compensator 120 can perform pixel group unit motion compensation on the current block based on the above determination.

[0107] The acquirer 105 can obtain information from the bitstream indicating whether to perform pixel group unit motion compensation. Here, the information indicating whether to perform pixel group unit motion compensation can be on / off information in the form of a flag. The information indicating whether to perform pixel group unit motion compensation can be included in a block-level syntax element. The pixel group unit motion compensator 120 can determine whether to perform pixel group unit motion compensation on the current block based on the information indicating whether to perform pixel group unit motion compensation, wherein this information is obtained from the bitstream.

[0108] Alternatively, the pixel group unit motion compensator 120 can determine whether to perform pixel group unit motion compensation on the current block in the current frame by using parameters related to the image including the current frame.

[0109] The pixel group unit motion compensator 120 can determine whether to perform pixel group unit motion compensation on the current block of the current frame based on the availability of a specific encoding / decoding tool. The pixel group unit motion compensator 120 can determine the availability of a specific encoding / decoding tool, different from the encoding / decoding tool associated with pixel group unit motion compensation for the current block, and determine whether to perform pixel group unit motion compensation on the current block in the current frame based on the availability of said specific encoding / decoding tool.

[0110] For example, when encoding / decoding tools related to Overlap Block Motion Compensation (OBMC) are available, the pixel group unit motion compensator 120 can determine whether to perform pixel group unit motion compensation for the current block in the current frame. When encoding / decoding tools related to OBMC are available, the pixel group unit motion compensator 120 can determine that pixel group unit motion compensation is not used for the current block.

[0111] OMBC is block-based motion compensation that allows reference blocks in the reference frame to overlap with their neighboring blocks in the current frame, preventing block degradation. Unlike general block-based motion compensation, OBMC compensates for motion by allowing overlap of reference blocks and taking into account the precise motion of pixels within the block. Therefore, when OBMC-related encoding / decoding tools are available, the pixel group-based motion compensator 120 can determine that pixel group-based motion compensation is not used for the current block. In other words, since two or more prediction directions are combined for the overlapping area, the pixel group-based motion compensator 120 can determine that pixel group-based motion compensation that takes into account two prediction directions is generally not used. However, the embodiment is not limited to this; when the area overlapping via OBMC is small, the pixel group-based motion compensator 120 can determine that pixel group-based motion compensation is used for the current block when OBMC-related encoding / decoding tools are available.

[0112] Optionally, since two or more prediction directions are combined for overlapping regions, the pixel group unit motion compensator 120 can determine that pixel group-based motion compensation considering two prediction directions is not limited to overlapping regions. Since only two prediction directions are used for non-overlapping regions, the pixel group unit motion compensator 120 can determine that pixel group-based motion compensation considering two prediction directions is limited to non-overlapping regions.

[0113] When an encoding / decoding tool related to illumination compensation is available, the pixel group unit motion compensator 120 can determine whether to perform pixel group unit motion compensation on the current block. For example, when an encoding / decoding tool related to illumination compensation is available, the pixel group unit motion compensator 120 can determine to perform pixel group unit motion compensation on the current block. The encoding / decoding tools related to pixel group unit motion compensation and those related to illumination compensation are not contradictory, therefore the pixel group unit motion compensator 120 can perform illumination compensation on the current block and motion compensation on a pixel group basis together. Here, illumination compensation refers to the operation of compensating the luminance pixel values ​​to be close to the luminance pixel values ​​of the original image by using linear coefficients and offsets on a block basis.

[0114] However, since illuminance compensation is performed when there is an illuminance difference ΔI over time, the motion of the actual object may not be properly compensated when pixel-group-based motion compensation (see Equation 1) is performed, because one side of the optical flow has a non-zero value. Therefore, when the degree of illuminance compensation is large, i.e., when ΔI is sufficiently large, the pixel-group-based motion compensator 120 can determine not to perform pixel-group-based motion compensation for the current block when encoding / decoding tools related to illuminance compensation are available.

[0115] When the encoding / decoding tools associated with weighted prediction are available, the pixel group unit motion compensator 120 can determine whether to perform pixel group unit motion compensation on the current block. For example, when the encoding / decoding tools associated with weighted compensation are available, the pixel group unit motion compensator 120 can determine not to perform pixel group unit motion compensation on the current block. The encoding / decoding tools associated with weighted compensation refer to the encoding / decoding tools that assign weights to each reference block of each reference frame and assign offsets to each reference block of each reference frame to produce a prediction block associated with the current block when bidirectional motion prediction is performed.

[0116] When an encoding / decoding tool associated with affine motion is available, the pixel group unit motion compensator 120 can determine whether to perform pixel group unit motion compensation on the current block. For example, when an encoding / decoding tool associated with affine motion is available, the pixel group unit motion compensator 120 can determine not to perform pixel group unit motion compensation on the current block. Since the encoding / decoding tools associated with affine motion are similar to those associated with pixel group unit motion compensation for compensating for precise motion, these encoding / decoding tools are contradictory and therefore cannot be used together on the same block.

[0117] The pixel group unit motion compensator 120 can determine whether to perform pixel group unit motion compensation on the current block based on the motion vector of the current block. For example, the pixel group unit motion compensator 120 can determine whether the ratio between the first motion vector MV1 associated with the first reference frame PICreference1 and the POC difference POCreference1 between the current frame and the first reference frame (Ratioreference1=MV1 / POCreference1), and the ratio between the second motion vector MV2 associated with the second reference frame PICreference2 and the POC difference POCreference2 between the current frame and the second reference frame (Ratioreference2=MV2 / POCreference2) are within a specific range, and when these ratios are within the specific range, it is determined that motion compensation should be performed on the current block on a pixel group basis.

[0118] When the magnitude of the motion vector is a specific size, the pixel group unit motion compensator 120 can determine to perform pixel group unit motion compensation on the current block. For example, when the magnitude of the motion vector is greater than the specific size, the pixel group unit motion compensator 120 can determine to perform motion compensation on the current block in units of pixels. Here, the specific size can be 0.

[0119] The pixel group unit motion compensator 120 can determine whether to perform motion compensation on the current block in units of pixel groups based on the time direction of the first prediction direction and the second prediction direction.

[0120] For example, when both the first prediction direction associated with the first reference frame and the second prediction direction associated with the second reference frame are oriented towards a reference frame that is temporally preceding the current frame or both are oriented towards a reference frame that is temporally following the current frame, the pixel group unit motion compensator 120 can determine not to perform motion compensation on a pixel group basis for the current block. Here, the temporal order of the frames is related to the display order; even if a frame will be displayed after the current frame in time, it can still be pre-decoded and stored in a buffer, and then displayed after the current frame.

[0121] When the time directions of the first prediction direction and the second prediction direction are different from each other, that is, when one prediction unit faces a reference frame that is in time before the current frame and the other prediction unit faces a reference frame that is in time after the current frame, the pixel group unit motion compensator 120 can determine to perform motion compensation on the current block on a pixel group basis.

[0122] When the size of the current block is a specific size, the pixel group unit motion compensator 120 can determine to perform pixel group unit motion compensation on the current block. For example, when the size of the current block is equal to or greater than the specific size, the pixel group unit motion compensator 120 can determine to perform pixel group unit motion compensation on the current block.

[0123] The pixel group unit motion compensator 120 can determine the availability of a particular encoding / decoding tool based on information about its availability obtained from high-level syntax carriers (such as strip headers, image parameter sets, and sequence parameter sets). Furthermore, the pixel group unit motion compensator 120 can also determine the availability of a particular encoding / decoding tool based on information about its availability obtained from block-level syntax elements.

[0124] However, the embodiments are not limited to this. The pixel group unit motion compensator 120 may obtain information about the availability of a specific encoding / decoding tool for the current block from the block-level syntax elements obtained from the bitstream, determine whether to use the specific encoding / decoding tool for the current block based on the information, and determine whether to perform pixel group unit motion compensation for the current block based on the determination of whether to use the specific encoding / decoding tool.

[0125] The pixel group unit motion compensator 120 can determine the reference pixel group in the reference block that corresponds to the current pixel group of the current block, and determine the gradient value of the reference pixel group.

[0126] The pixel group unit motion compensator 120 can generate pixel group motion compensation values ​​by performing motion compensation on a pixel group basis for the current block using the gradient values ​​of a reference pixel group.

[0127] The pixel group unit motion compensator 120 can generate gradient values ​​for the first pixel and the second pixel by applying a filter to a first surrounding region (including the first pixel group) of the first pixel group in the pixel group of the first reference block indicated by a first motion vector, and a second surrounding region (including the second pixel group) of the second pixel group in the pixel group of the second reference block indicated by a second motion vector.

[0128] The pixel group unit motion compensator 120 can determine the pixel values ​​and gradient values ​​of pixels in a first window of a specific size that includes the first pixel group in a first reference frame, and determine the pixel values ​​and gradient values ​​of pixels in a second window of a specific size that includes the second pixel group in a second reference frame. The pixel group unit motion compensator 120 can obtain parameters related to the size of the window used to calculate the displacement vector per unit time based on at least one of the following: information about the window size obtained from the bitstream, the layering depth of the frame, the size of the group of frames (GOP), the image resolution, parameters related to the temporal distance between the reference frame and the current frame, the frame rate, the motion vector, setting parameters related to the coding prediction structure, and the prediction direction. Based on these parameters, motion compensation is performed on the current block in pixels group units. For example, a window size of M×N ensures motion consistency and reduces the probability of errors when calculating the displacement vector per unit time for the current pixel group. When factors that may increase the likelihood of errors exist, the pixel group unit motion compensator 120 can increase the window size to ensure motion consistency and reduce the probability of errors during calculation.

[0129] When the size of the group of pixels (GOP) is large, the distance between the current frame and the reference frame can be increased, thus increasing the likelihood of errors. Therefore, the pixel group unit motion compensator 120 can perform motion compensation on the current block in units of pixel groups by expanding the window size.

[0130] Furthermore, for example, when the pixel group is K×K in size, motion consistency is better guaranteed compared to when the pixel group consists of only one pixel. Therefore, the pixel group unit motion compensator 120 can determine the window size for a K×K pixel group to be smaller than the window size for a pixel group consisting of only one pixel.

[0131] Information about the size of a window (such as a first window and a second window) can be explicitly sent by signaling through a high-level syntax carrier included in the bitstream and in strip headers, screen parameter sets, sequence parameter sets, or other various forms.

[0132] Alternatively, the window size can be derived from parameters associated with the image including the current frame. For example, the window size can be determined based on the layering depth of the current frame. In other words, errors accumulate as the layering depth of the current frame increases, thus reducing prediction accuracy. Therefore, the window size can be set to large.

[0133] Here, the layering depth of the current frame can be greater than the layering depth of the image referenced by the current image. For example, the layering depth of the intraframe can be 0, the layering depth of the first frame referencing the intraframe can be 1, and the layering depth of the second frame referencing the first frame can be 2.

[0134] In addition, the pixel group unit motion compensator 120 can determine the window size based on the size of the GOP.

[0135] Optionally, the pixel group unit motion compensator 120 can determine the window size based on the resolution of the image.

[0136] The pixel group unit motion compensator 120 can determine the window size based on the frame rate. Furthermore, the pixel group unit motion compensator 120 can determine the window size based on the motion vector of the current block. Specifically, the pixel group unit motion compensator 120 can determine the window size based on at least one of the magnitude and angle of the motion vector of the current block.

[0137] The pixel group unit motion compensator 120 can determine the window size based on a reference frame index that indicates one of a plurality of frames stored in a reference frame buffer.

[0138] The pixel group unit motion compensator 120 can determine the window size based on the availability of bidirectional prediction from different time directions. Furthermore, the pixel group unit motion compensator 120 can determine the window size based on setting parameters associated with the coded prediction structure. Here, the setting parameters associated with the coded prediction structure can indicate low latency or random access.

[0139] The pixel group unit motion compensator 120 can determine the window size differently based on whether the coded prediction structure is low-latency or random access.

[0140] The pixel group unit motion compensator 120 performs motion compensation on a pixel group basis by using the gradient value and pixel value of a pixel, wherein the difference between the pixel value and the value of a pixel included in the current pixel group among the pixels included in the window is no greater than a specific threshold. This is to ensure consistent motion over regions of the same object.

[0141] The pixel group unit motion compensator 120 determines the displacement vector per unit time for the current pixel group by using the pixel values ​​and gradient values ​​of pixels in a first window and pixels in a second window. Here, the value of the displacement vector per unit time for the current pixel group can be adjusted by a regularization parameter. The regularization parameter is introduced to prevent errors when an inappropriate displacement vector per unit time for the current pixel group is determined, thus performing motion compensation on a pixel group basis. The pixel group unit motion compensator 120 can obtain the regularization parameter related to the displacement vector per unit time in the horizontal or vertical direction based on at least one of the following: information about the regularization parameter related to the displacement vector per unit time in the horizontal or vertical direction, information obtained from the bitstream, the bit depth of the sample, the size of the GOP, the motion vector, parameters related to the time distance between the reference frame and the current frame, the frame rate, setting parameters related to the coding prediction structure, and the prediction direction. The pixel group unit motion compensator 120 can perform pixel group unit motion compensation on the current block based on the regularization parameter related to the displacement vector per unit time in the horizontal or vertical direction. Then refer to Figure 8a Used to describe regular expression parameters.

[0142] The pixel group unit motion compensator 120 can determine the regularization parameters based on information about the regularization parameters obtained from the bitstream. This information about the regularization parameters can be included in a high-level syntax carrier in various forms, such as a strip header, a frame parameter set, a sequence parameter set, or others.

[0143] However, the embodiments are not limited to this; the pixel group unit motion compensator 120 may determine the regularization parameters based on image-related parameters. For example, the pixel group unit motion compensator 120 may determine the regularization parameters based on the size of the GOP. The pixel group unit motion compensator 120 may determine the regularization parameters based on the distance from the current frame to the reference frame. Here, the distance to the reference frame may be the difference in point of view (POC) between the current frame and the reference frame.

[0144] The pixel group unit motion compensator 120 can determine regularization parameters based on the motion vector of the current block. The pixel group unit motion compensator 120 can determine regularization parameters based on at least one of the magnitude and angle of the motion vector of the current block.

[0145] The pixel group unit motion compensator 120 can determine the regularization parameters based on the reference frame index.

[0146] The pixel group unit motion compensator 120 can determine regularization parameters based on the availability of bidirectional prediction from different time directions. Furthermore, the pixel group unit motion compensator 120 can determine regularization parameters based on setting parameters associated with the coded prediction structure. These setting parameters can indicate low latency or random access.

[0147] The pixel group unit motion compensator 120 can determine the regularization parameters differently based on low latency or random access.

[0148] The pixel group unit motion compensator 120 can determine regularization parameters based on the frame rate. The pixel group unit motion compensator 120 can also determine regularization parameters based on the availability of bidirectional prediction with different temporal directions.

[0149] The pixel group unit motion compensator 120 can perform motion compensation on the current block in units of pixel groups by using the displacement vector per unit time for the current pixel and the gradient value of the reference pixel.

[0150] The reference position of the reference block can be an integer pixel position, but optionally, it can be a fractional pixel position.

[0151] When the reference position of the reference block is a fractional pixel position, the gradient value of the reference pixel in the reference block can be determined by using the pixel values ​​of neighboring pixels whose components are integers in a specific direction.

[0152] For example, the pixel group unit motion compensator 120 can determine the gradient value of a reference pixel as the result obtained by filtering the pixel values ​​of neighboring pixels whose components in a particular direction are integers using a gradient filter. Here, the filter coefficients of the gradient filter can be determined by using coefficients predetermined for a DCT-based interpolation filter. The filter coefficients of the gradient filter can be filter coefficients scaled to integer coefficients to reduce real-number operations during filtering.

[0153] Here, the gradient filter can be a 1D gradient filter in the horizontal or vertical direction.

[0154] The pixel group unit motion compensator 120 can determine the gradient value of a reference pixel in the horizontal or vertical direction by filtering neighboring pixels whose components in the corresponding direction are integers using a 1D gradient filter in the horizontal or vertical direction.

[0155] For example, the pixel group unit motion compensator 120 can determine the gradient value of the reference pixel in the horizontal direction by performing filtering on pixels located in the horizontal direction from a plurality of pixels adjacent to the reference pixel whose horizontal direction component is an integer, using a 1D gradient filter in the horizontal direction.

[0156] When the position of the reference pixel is (x+α, y+β) (where x and y are integers, and α and β are fractions), the pixel group unit motion compensator 120 can determine the pixel value at the (x, y+β) position as the result of filtering the pixel at the (x, y) position and the pixels whose vertical components are integers among a plurality of pixels in the vertical direction starting from the pixel at the (x, y) position.

[0157] The pixel group unit motion compensator 120 can determine the gradient value in the horizontal direction at the position (x+α, y+β) as the result of filtering the pixel value at the position (x, y+β) and the pixel value of the pixel among a plurality of pixels in the horizontal direction from the pixel at the position (x, y+β) where the horizontal component is an integer.

[0158] The order in which the 1D gradient filter and the 1D interpolation filter are used is not restricted. In the above description, a vertically interpolated filter value is first generated by filtering pixels at integer positions using a 1D interpolation filter in the vertical direction, and then the vertically interpolated filter value is filtered using a 1D gradient filter in the horizontal direction. However, alternatively, a horizontally gradient filter value can be generated first by filtering pixels at integer positions using a 1D gradient filter in the horizontal direction, and then the horizontally gradient filter value can be filtered using a 1D interpolation filter in the vertical direction.

[0159] The pixel group unit motion compensator 120 has been described in detail above as determining the gradient value in the horizontal direction at the position (x+α, y+β). Since the pixel group unit motion compensator 120 determines the gradient value in the vertical direction at the position (x+α, y+β) in a similar manner to determining the gradient value in the horizontal direction, its details will not be provided again.

[0160] In the preceding text, the pixel group unit motion compensator 120 has been described in detail using a 1D gradient filter and a 1D interpolation filter to determine the gradient value at fractional pixel positions. However, alternatively, the gradient filter and interpolation filter can be used to determine the gradient value at integer pixel positions. While pixel values ​​can be determined without using an interpolation filter in the case of integer pixels, for processing consistent with fractional pixels, the pixel value of an integer pixel can be determined by performing filtering on the integer pixel and its neighboring pixels whose components in a particular direction are integers using an interpolation filter. For example, the interpolation filter coefficients at an integer pixel could be (0,0,64,0,0). Since the interpolation filter coefficients associated with neighboring integer pixels are 0, filtering can be performed using only the pixel value of the current integer pixel. As a result, the pixel value of the current integer pixel can be determined by performing filtering on both the current integer pixel and its neighboring integer pixels using an interpolation filter.

[0161] The pixel group unit motion compensator 120 can perform descaling after filtering pixels at integer positions using a 1D interpolation filter in the vertical direction. Here, descaling may include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the bit depth of the sample. For example, the number of descaling bits can be determined based on specific input data in the block.

[0162] For example, the descaling bit count can be obtained by subtracting 8 from the bit depth of the sample.

[0163] The pixel group unit motion compensator 120 can perform descaling after filtering the values ​​generated by descaling using a gradient filter in the horizontal direction. Similarly, descaling here can include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the number of scaling bits of the 1D interpolation filter in the vertical direction, the number of scaling bits of the 1D gradient filter in the horizontal direction, and the bit depth of the sample. For example, when the number of scaling bits p of the 1D interpolation filter in the vertical direction is 6, the number of scaling bits q of the 1D gradient filter in the horizontal direction is 4, and the bit depth of the sample is b, the number of descaling bits can be p + q + 8 - b, i.e., 18 - b.

[0164] When the pixel group unit motion compensator 120 performs only a right bit shift according to the number of descaling bits on the value generated by filtering after performing filtering, rounding errors may occur. Therefore, the pixel group unit motion compensator 120 can perform descaling after adding the offset value to the value generated by filtering. Here, the offset value can be 2^(number of descaling bits - 1).

[0165] Inter-frame predictor 110 can generate predicted pixel values ​​for the current block using block-level motion compensation values ​​and pixel-group-level motion compensation values ​​for the current block. For example, inter-frame predictor 110 can generate predicted pixel values ​​for the current block by adding block-level motion compensation values ​​and pixel-group-level motion compensation values ​​for the current block. Here, the block-level motion compensation value can represent the value generated by performing motion compensation on a block-by-block basis, and the pixel-group-level motion compensation value can represent the value generated by performing motion compensation on a pixel-group basis. The block-level motion compensation value can be the average or weighted sum of reference pixels, and the pixel-group-level motion compensation value can be a value determined based on the displacement vector per unit time associated with the current pixel and the gradient value of the reference pixel.

[0166] The pixel group unit motion compensator 120 can obtain a shift value for descaling after interpolation or gradient operations based on at least one of the bit depth of the sample, the range of the input of the filter used for interpolation or gradient operations, and the coefficients of the filter. The pixel group unit motion compensator 120 can perform descaling after interpolation or gradient operations for pixels included in the first and second reference blocks by using the shift value for descaling.

[0167] The inter-frame predictor 110 can use and store motion vectors when performing block-unit motion compensation. Here, the motion vector unit can be a block with a 4×4 size. Simultaneously, when the motion vector is stored after block-unit motion compensation, the motion vector storage unit can be a block with various sizes other than 4×4 (e.g., a block with an R×R size, where R is an integer). Here, the motion vector storage unit can be a block larger than 4×4. For example, the motion vector storage unit can be a block with a 16×16 size. When the motion vector unit is a 4×4 size block and the motion vector storage unit is a 16×16 size block, the inter-frame predictor 110 can store data according to the equation (MVx, MVy) = f RXR The motion vector is (MVx, MVy). Here, MVx and MVy are the x and y components of the motion vector used in block unit motion compensation, respectively, and f RXR (MVx, MVy) can represent a function of motion vectors MVx and MVy, taking into account the size of the motion vector storage unit. For example, f RXR(MVx, MVy) can be a function such that the average value of the x-components MVx of the motion vectors of the cells included in the R×R motion vector storage unit is determined as the x-component MVx stored according to the R×R motion vector storage unit, and the average value of the y-components MVy of the motion vectors of the cells included in the R×R motion vector storage unit is determined as the y-component MVy stored according to the R×R motion vector storage unit.

[0168] In other words, when storing motion vectors, the inter-frame predictor 110 can perform storage compression by using larger units. The inter-frame predictor 110 can perform motion compensation not only on a block-by-block basis but also on a pixel-by-pixel basis for the blocks included in the current frame. Therefore, motion vectors that consider both block-by-block motion compensation and pixel-by-pixel motion compensation can be stored. Here, the stored motion vectors can be determined based on the motion vectors used in block-by-block motion compensation, the displacement vectors per unit time in the horizontal or vertical direction used in pixel-by-pixel motion compensation, and the weights of the displacement vectors per unit time in the horizontal or vertical direction.

[0169] Here, the weights can be determined based on the size of the motion vector storage unit, the size of the pixel group, and the scaling factor of the gradient filter or interpolation filter used in motion compensation on a pixel-group basis.

[0170] The inter-frame predictor 110 can determine the motion vector prediction factor for blocks in frames decoded after the current frame by using temporal motion vector prediction factor candidates. The temporal motion vector prediction factor candidates can be motion vectors of co-occurring blocks included in previously decoded frames, and therefore can be motion vectors stored for previously decoded frames. Here, when the stored motion vectors are motion vectors that take into account motion compensation at the pixel group level, the temporal motion vector prediction factor candidates can be determined as motion vectors used in more accurate motion compensation, thus improving predictive coding / decoding efficiency.

[0171] Simultaneously, when pixel group unit motion compensation is performed, the size of the target block used to perform the pixel group unit motion compensation can be increased along with the size of the current block based on the window size and the length of the interpolation filter. Since pixel group unit motion compensation is performed on the current block based on pixels located at the edges of the current block and neighboring pixels at the edges of the current block, the target block is increased in size based on the window size compared to the current block.

[0172] Therefore, during the process of performing pixel group unit motion compensation by using a window, the pixel group unit motion compensator 120 can adjust the position of a pixel outside the current block within the window to the position of a pixel adjacent to the inside of the current block and determine the pixel value and gradient value at the adjusted pixel position in order to reduce the number of memory accesses and multiplication operations.

[0173] The reconstructor 125 can obtain the residual block of the current block from the bitstream and reconstruct the current block using the residual block and the predicted pixel values. For example, the reconstructor 125 can generate the pixel values ​​of the reconstructed block from the bitstream by adding the pixel values ​​of the residual block of the current block to the pixel values ​​of the predicted block of the current block.

[0174] The video decoding device 100 may include an image decoder (not shown), wherein the image decoder may include an acquirer 105, an inter-frame predictor 110, and a reconstructor 125. References will follow below. Figure 1e Describe the image decoder.

[0175] Figure 1b This is a flowchart of a video decoding method according to various embodiments.

[0176] In operation S105, the video decoding device 100 can obtain motion prediction mode information for the current block in the current frame from the bitstream. The video decoding device 100 can receive a bitstream including motion prediction mode information for the current block in the current frame, and obtain the motion prediction mode information for the current block from the received bitstream. The video decoding device 100 can obtain information about the prediction mode of the current block from the bitstream, and determine the prediction mode of the current block based on the information about the prediction mode of the current block. Here, when the prediction mode of the current block is an inter-frame prediction mode, the video decoding device 100 can obtain the motion prediction mode information for the current block.

[0177] For example, video decoding device 100 can determine that the prediction mode of the current block is an inter-frame prediction mode based on information about the prediction mode of the current block. When the prediction mode of the current block is an inter-frame prediction mode, video decoding device 100 can obtain motion prediction mode information for the current block from the bitstream.

[0178] In operation S110, when the motion prediction mode information indicates bidirectional motion prediction mode, the video decoding device 100 can obtain from the bitstream a first motion vector indicating the current block in the first reference frame and a second motion vector indicating the current block in the second reference frame.

[0179] In other words, the video decoding device 100 can obtain a bitstream including information about a first motion vector and a second motion vector, and obtain the first motion vector and the second motion vector from the received bitstream. The video decoding device 100 can obtain a reference frame index from the bitstream, and determine a first reference frame and a second reference frame in a previously decoded frame based on the reference frame index.

[0180] In operation S115, the video decoding device 100 can obtain parameters related to the motion compensation per pixel group of the current block based on information about parameters related to pixel group unit motion compensation obtained from the bitstream and at least one of parameters including the image of the current frame. Here, a pixel group may include at least one pixel.

[0181] During operation S120, the video decoding device 100 can generate a predicted block for the current block by performing motion compensation based on a first motion vector and a second motion vector, as well as pixel group unit motion compensation based on parameters related to pixel group unit motion compensation.

[0182] During operation S125, the video decoding device 100 can obtain the residual block of the current block from the bit stream.

[0183] In operation S130, the video decoding device 100 can reconstruct the current block based on the prediction block and the residual block. In other words, the video decoding device 100 can generate the pixel values ​​of the reconstructed block of the current block by adding the predicted pixel values ​​of the prediction block and the pixel values ​​of the residual block indicated by the residual block associated with the current block.

[0184] Figure 1c It is a block diagram of a video encoding device according to various embodiments.

[0185] Video encoding apparatus 150 according to various embodiments includes an inter-frame predictor 155 and a bitstream generator 170.

[0186] Inter-frame predictor 155 performs inter-frame prediction on the current block by referencing various blocks, based on rate and distortion costs. In other words, inter-frame predictor 155 can generate predicted pixel values ​​for the current block by using at least one of the frames encoded before the current frame that includes the current block.

[0187] The inter-frame predictor 155 may include a block unit motion compensator 160 and a pixel group unit motion compensator 165.

[0188] The block-unit motion compensator 160 can generate block-unit motion compensation values ​​by performing motion compensation on the current block.

[0189] The block unit motion compensator 160 can determine at least one reference frame in a previously encoded frame and determine the reference block of the current block located in the at least one reference frame.

[0190] The block-unit motion compensator 160 can generate block-unit motion compensation values ​​by performing motion compensation on the current block using the pixel values ​​of a reference block. Alternatively, the block-unit motion compensator 160 can generate block-unit motion compensation values ​​by performing motion compensation on the current block using the reference pixel values ​​of a reference block corresponding to the current pixel of the current block.

[0191] The block-unit motion compensator 160 can generate block-based motion compensation values ​​by performing motion compensation on the current block using multiple reference blocks included in multiple reference frames. For example, when the motion prediction mode of the current block is bidirectional prediction mode, the block-unit motion compensator 160 can determine two reference frames in previously encoded frames and determine two reference blocks included in said two reference frames. Here, bidirectional prediction not only means performing inter-frame prediction by using a frame displayed before the current frame and a frame displayed after the current frame, but also means performing inter-frame prediction by using two frames encoded before the current frame regardless of their display order.

[0192] The block-unit motion compensator 160 can generate block-unit motion compensation values ​​by performing motion compensation on the current block using the pixel values ​​of two reference pixels from two reference blocks. Alternatively, the block-unit motion compensator 160 can generate block-unit motion compensation values ​​by performing motion compensation on the current block using the average pixel value or a weighted sum of the two reference pixels.

[0193] The block unit motion compensator 160 can output a reference screen index indicating a reference screen in a previously encoded screen used for motion compensation of the current block.

[0194] The block unit motion compensator 160 determines and outputs a motion vector that takes the current block as the starting point and a reference block of the current block as the ending point. The motion vector can represent a vector that indicates the displacement of the current block's reference coordinates in the current frame and the reference block's reference coordinates in the reference frame. For example, if the coordinates of the top-left corner of the current block are (1,1) and the top-left coordinates of the reference block in the reference frame are (3,3), the motion vector could be (2,2).

[0195] The reference position of the reference block can be an integer pixel position, but optionally, it can be a fractional pixel position. Here, the fractional pixel position can be determined in units of 1 / 4 pel or 1 / 16 pel. Alternatively, the fractional pixel position can be determined in various fractional pel units.

[0196] For example, when the reference position of the reference block is (1.5, 1.5) and the coordinates of the top-left corner of the current block are (1, 1), the motion vector could be (0.5, 0.5). When the motion vector is determined in units of 1 / 4 pel or 1 / 6 pel to indicate the reference position of the reference block (i.e., the position of a fractional pixel), an integer motion vector is determined by scaling the motion vector, and the reference position of the reference block can be determined by using an upwardly scaled motion vector. When the reference position of the reference block is a fractional pixel position, the position of the reference pixel of the reference block can also be a fractional pixel position. Therefore, the pixel value at the fractional pixel position in the reference block can be determined by using the pixel value of a neighboring pixel whose component in a specific direction is an integer.

[0197] For example, the block-unit motion compensator 160 can determine the pixel value of a reference pixel at a fractional pixel position by filtering the pixel values ​​of neighboring pixels whose components in a particular direction are integers using an interpolation filter, and can determine block-level motion compensation values ​​for the current block using the pixel values ​​of the reference pixel. Here, the interpolation filter can be a DCT-based M-tap interpolation filter. The coefficients of the DCT-based M-tap interpolation filter can be derived from DCT and IDCT. Here, the coefficients of the interpolation filter can be filter coefficients scaled to integer coefficients to reduce real-number operations during filtering.

[0198] Here, the interpolation filter can be a 1D interpolation filter in the horizontal or vertical direction.

[0199] The block unit motion compensator 160 can first perform filtering on neighboring integer pixels using a 1D interpolation filter in the vertical direction, and then perform filtering on the filtered value using a 1D interpolation filter in the horizontal direction to determine the pixel value of the reference pixel at the fractional pixel position. When the scaled filter coefficients are used, the block unit motion compensator 160 can perform descaling on the filtered value after performing filtering on the pixel at the integer position using a 1D interpolation filter in the vertical direction. Here, descaling may include bit shifting to the right by a descaling bit number. The descaling bit number can be determined based on the bit depth of the sample. For example, the descaling bit number can be a value obtained by subtracting 8 from the bit depth of the sample.

[0200] Furthermore, the block unit motion compensator 160 can perform filtering on pixels where the horizontal component is an integer using a 1D interpolation filter in the vertical direction, and then perform a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the number of scaling bits of the 1D interpolation filter in the vertical direction, the number of scaling bits of the 1D interpolation filter in the horizontal direction, and the bit depth of the sample.

[0201] When the block unit motion compensator 160 only performs a right bit shift according to the descaling bit number, rounding errors may occur. Therefore, the block unit motion compensator 160 can perform filtering on pixels whose components in a specific direction are integers using a 1D interpolation filter in that specific direction, add the offset value to the filtered value, and then perform descaling on the value with the added offset value. Here, the offset value can be 2^(descaling bit number - 1).

[0202] The preceding text described how the number of descaling bits is determined based on the bit depth of the samples after filtering using a 1D interpolation filter in the vertical direction. However, alternatively, the number of descaling bits can be determined not only based on the bit depth of the samples but also based on the number of bits scaled for the interpolation filter coefficients. In other words, the number of descaling bits can be determined based on both the bit depth of the samples and the number of bits scaled for the interpolation coefficients, while taking into account the size of the registers used during filtering and the size of the buffers storing the values ​​generated during filtering.

[0203] The pixel group unit motion compensator 165 can generate pixel group-based motion compensation values ​​by performing motion compensation on the current block on a pixel group basis. For example, when the motion prediction mode is a bidirectional motion prediction mode, the pixel group unit motion compensator 165 can generate pixel group-based motion compensation values ​​by performing motion compensation on the current block on a pixel group basis.

[0204] The pixel group unit motion compensator 165 can generate pixel group motion compensation values ​​by performing motion compensation on the current block on a pixel group basis using the gradient values ​​of pixels included in the reference block in the current block.

[0205] The pixel group unit motion compensator 165 can generate gradient values ​​of the first pixel in the pixels of the first reference block in the first reference frame and gradient values ​​of the second pixel in the pixels of the second reference block in the second reference frame by applying filters to the first surrounding region of the first pixel and the second surrounding region of the second pixel.

[0206] The pixel group unit motion compensator 165 can determine the pixel values ​​and gradient values ​​of pixels in a first window of a specific size that includes the first reference pixel and surrounds the first reference pixel in a first reference frame, and determine the pixel values ​​and gradient values ​​of pixels in a second window of a specific size that includes the second reference pixel and surrounds the second reference pixel in a second reference frame. The pixel group unit motion compensator 165 can determine the displacement vector per unit time for the current pixel by using the pixel values ​​and gradient values ​​of the pixels in the first window and the pixels in the second window.

[0207] The pixel group unit motion compensator 165 can generate pixel group motion compensation values ​​by performing motion compensation on the current block on a pixel group basis using the displacement vector per unit time and the gradient value of the reference pixel.

[0208] The location of the reference pixel can be an integer pixel location, but optionally, it can be a fraction of a pixel location.

[0209] When the reference position of the reference block is the position of a fractional pixel, the gradient value of the reference pixel in the reference block can be determined by using the pixel values ​​of neighboring pixels whose components are integers in a specific direction.

[0210] For example, the pixel group unit motion compensator 165 can determine the gradient value of a reference pixel as the result obtained by filtering the pixel values ​​of neighboring pixels whose components in a particular direction are integers using a gradient filter. Here, the filter coefficients of the gradient filter can be determined by using coefficients predetermined for a DCT-based interpolation filter.

[0211] The filter coefficients of a gradient filter can be filter coefficients scaled to integer values ​​to reduce real-valued operations during filtering. Here, the gradient filter can be a 1D gradient filter in the horizontal or vertical direction.

[0212] The pixel group unit motion compensator 165 can determine the gradient value of a reference pixel in the horizontal or vertical direction by filtering neighboring pixels whose components in the corresponding direction are integers using a 1D gradient filter in the horizontal or vertical direction.

[0213] For example, the pixel group unit motion compensator 165 can determine the pixel value of a pixel whose vertical component is a fraction by performing filtering on pixels whose vertical component is an integer among pixels in the vertical direction starting from an integer pixel adjacent to the reference pixel.

[0214] For a pixel located in another column adjacent to an integer pixel adjacent to a reference pixel, the pixel group unit motion compensator 165 can determine the pixel value at the fractional pixel position in the other column by performing filtering on the neighboring integer pixels in the vertical direction using a 1D interpolation filter in the vertical direction. Here, the position of the pixel in the other column can be the position of the fractional pixel in the vertical direction and the position of the integer pixel in the horizontal direction.

[0215] In other words, when the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are fractions), the pixel group unit motion compensator 165 can determine the pixel value at position (x, y+β) by performing filtering on the neighboring integer pixels in the vertical direction starting from position (x, y) using an interpolation filter in the vertical direction.

[0216] The pixel group unit motion compensator 165 can determine the gradient value in the horizontal direction at the (x+α, y+β) position by performing filtering on the pixel value at the (x, y+β) position and the pixel values ​​of pixels whose horizontal components are integers among a plurality of pixels in the horizontal direction starting from the pixel value at the (x, y+β) position.

[0217] The order in which the 1D gradient filter and the 1D interpolation filter are used is not restricted. As described above, a vertically interpolated filter value can be generated first by performing filtering on pixels at integer positions using a vertically interpolated filter, and then the vertically interpolated filter value can be filtered using a horizontally interpolated filter. Alternatively, a horizontally gradient filter value can be generated first by performing filtering on pixels at integer positions using a horizontally interpolated filter, and then the horizontally gradient filter value can be filtered using a vertically interpolated filter.

[0218] The pixel group unit motion compensator 165 has been described in detail above, which determines the gradient value in the horizontal direction at the position (x+α, y+β).

[0219] The pixel group unit motion compensator 165 can determine the gradient value in the vertical direction at the position (x+α, y+β) in a manner similar to determining the gradient value in the horizontal direction.

[0220] The pixel group unit motion compensator 165 determines the gradient value of the reference pixel in the vertical direction by performing filtering on the neighboring integer pixels in the vertical direction among the integer pixels adjacent to the reference pixel using a 1D gradient filter in the vertical direction. Furthermore, for pixels adjacent to the reference pixel and located in another column, the pixel group unit motion compensator 165 determines the gradient value in the vertical direction for the pixels adjacent to the reference pixel and located in the other column by performing filtering on the neighboring integer pixels in the vertical direction using a 1D gradient filter in the vertical direction. Here, the position of the pixel can be a fractional pixel position in the vertical direction and an integer pixel position in the horizontal direction.

[0221] In other words, when the position of the reference pixel is (x+α, y+β) (where x and y are integers and α and β are fractions), the pixel group unit motion compensator 165 can determine the gradient value in the vertical direction at the position (x, y+β) by performing filtering on the neighboring integer pixels in the vertical direction starting from the position (x, y).

[0222] The pixel group unit motion compensator 165 can determine the gradient value in the horizontal direction at the position (x+α, y+β) by performing filtering on the gradient value at the position (x, y+β) and the gradient values ​​of the neighboring integer pixels in the horizontal direction from the position (x, y+β).

[0223] The order in which the 1D gradient filter and the 1D interpolation filter are used is not restricted. As described above, a gradient filter value in the vertical direction can be generated first by filtering pixels at integer positions using a gradient filter in the vertical direction, and then the gradient filter value in the vertical direction can be filtered using a 1D interpolation filter in the horizontal direction. Alternatively, an interpolation filter value in the horizontal direction can be generated first by filtering pixels at integer positions using a 1D interpolation filter in the horizontal direction, and then the interpolation filter value in the horizontal direction can be filtered using a 1D gradient filter in the vertical direction.

[0224] The pixel group unit motion compensator 165 has been described in detail above as using gradient filters and interpolation filters to determine gradient values ​​at fractional pixel locations. However, alternatively, gradient filters and interpolation filters can be used to determine gradient values ​​at integer pixel locations.

[0225] In the case of integer pixels, the pixel value can be determined without using an interpolation filter. However, to ensure consistency with the processing in fractional pixels, filtering can be performed on the integer pixel and its neighboring integer pixels using an interpolation filter. For example, the interpolation filter coefficients at the integer pixel could be (0,0,64,0,0). Since the interpolation filter coefficients multiplied by the neighboring integer pixels are 0, filtering can be performed using only the pixel value of the current integer pixel. As a result, the pixel value of the current integer pixel can be equally determined as the value obtained by filtering the current integer pixel and its neighboring integer pixels using an interpolation filter.

[0226] Simultaneously, when the scaled filter coefficients are used, the pixel group unit motion compensator 165 can perform filtering on pixels at integer positions using a 1D gradient filter in the horizontal direction, and then perform descaling on the filtered values. Here, descaling may include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the bit depth of the sample. For example, the number of descaling bits can be a value obtained by subtracting 8 from the bit depth of the sample.

[0227] The pixel group unit motion compensator 165 can perform filtering on pixels whose components in the vertical direction are integers using an interpolation filter in the vertical direction, and then perform descaling. Here, descaling may include a right bit shift according to the number of descaling bits. The number of descaling bits can be determined based on the number of scaling bits of the 1D interpolation filter in the vertical direction, the number of scaling bits of the 1D gradient filter in the horizontal direction, and the bit depth of the sample.

[0228] When the pixel group unit motion compensator 165 only performs a right shift according to the descaling bit count, rounding errors may occur. Therefore, the pixel group unit motion compensator 165 can perform filtering using a 1D interpolation filter, add the offset value to the filtered value, and then perform descaling on the value with the added offset value. Here, the offset value can be 2^(descaling bit count - 1).

[0229] The inter-frame predictor 155 can generate the predicted pixel value of the current block by using block-based motion compensation values ​​and pixel-group-based motion compensation values ​​for the current block. For example, the inter-frame predictor 155 can generate the predicted pixel value of the current block by adding the block-based motion compensation values ​​and pixel-group-based motion compensation values ​​for the current block. Specifically, when the motion prediction mode of the current block is a bidirectional motion prediction mode, the inter-frame predictor 155 can generate the predicted pixel value of the current block by using block-based motion compensation values ​​and pixel-group-based motion compensation values ​​for the current block.

[0230] When the motion prediction mode for the current block is unidirectional motion prediction mode, the inter-frame predictor 155 can generate the predicted pixel values ​​for the current block by using block-based motion compensation values ​​for the current block. Here, unidirectional means using a reference frame from among the previously encoded frames. The reference frame may be a frame displayed before the current frame, but optionally, it may be a frame displayed after the current frame.

[0231] Inter-frame predictor 155 can determine the motion prediction mode of the current block and output information indicating the motion prediction mode of the current block. For example, inter-frame predictor 155 can determine that the motion prediction mode of the current block is a bidirectional motion prediction mode and output information indicating the bidirectional motion prediction mode. Here, the bidirectional motion prediction mode means a mode that predicts motion by using reference blocks in two decoded reference frames.

[0232] The pixel group unit motion compensator 165 can determine parameters related to pixel group unit motion compensation and perform pixel group unit motion compensation on the current block based on these parameters. Here, the parameters related to pixel group unit motion compensation can be obtained from parameters related to the image, including the current frame. Since the process by which the pixel group unit motion compensator 165 obtains the parameters related to pixel group unit motion compensation from image-related parameters is the same as the process by which the pixel group unit motion compensator 120 obtains the parameters related to pixel group unit motion compensation from image-related parameters, its description is omitted.

[0233] Optionally, the pixel group unit motion compensator 165 can determine parameters related to pixel group unit motion compensation while performing pixel group unit motion compensation, and output the determined parameters related to pixel group unit motion compensation. The bitstream generator 170 can generate a bitstream including information related to pixel group unit motion compensation. Since the processing of the pixel group unit motion compensator 165 to output parameters related to pixel group unit motion compensation by performing pixel group unit motion compensation on the current block and the processing of the bitstream generator 170 to generate a bitstream including information about parameters related to pixel group unit motion compensation are the opposite of the processing of the acquirer 105 to obtain parameter information related to pixel group unit motion compensation from the bitstream and the processing of the pixel group unit motion compensator 120 to determine parameters related to pixel group unit motion compensation from the obtained parameter information and perform pixel group unit motion compensation on the current block, their description is omitted.

[0234] Bitstream generator 170 can generate a bitstream including motion vectors indicating reference blocks. Bitstream generator 170 can encode the motion vectors indicating reference blocks and generate a bitstream including the encoded motion vectors. Bitstream generator 170 can encode difference values ​​of the motion vectors indicating reference blocks and generate a bitstream including the encoded difference values ​​of the motion vectors. Here, the difference value of the motion vectors can represent the difference between the motion vector and its prediction factor. Here, the difference value of the motion vectors can represent the difference value of the motion vectors for reference frames respectively associated with prediction directions including L0 and L1 directions. Here, the difference value of the motion vectors for the L0 direction can represent the difference value of the motion vectors indicating reference frames included in the L0 reference frame list, and the difference value of the motion vectors for the L1 direction can represent the difference value of the motion vectors indicating reference frames included in the L1 reference frame list.

[0235] Furthermore, the bitstream generator 170 can generate a bitstream that also includes information indicating the motion prediction mode of the current block. The bitstream generator 170 can encode a reference frame index indicating a reference frame of the current block in a previously encoded frame, and generate a bitstream including the encoded reference frame index. Here, the reference frame index can represent a reference frame index for each of the prediction directions, including the L0 direction and the L1 direction. Specifically, the reference frame index for the L0 direction can represent an index indicating a reference frame among the frames included in the L0 reference frame list, and the reference frame index for the L1 direction can represent an index indicating a reference frame among the frames included in the L1 reference frame list.

[0236] The video encoding device 150 may include an image encoder (not shown), which may include an inter-frame predictor 155 and a bitstream generator 170. References will follow. Figure 1f Describe the video encoder.

[0237] Figure 1d This is a flowchart of a video encoding method according to various embodiments.

[0238] Reference Figure 1d During operation S150, the video encoding device 150 can obtain the prediction block, first motion vector, second motion vector, and parameters related to pixel group unit motion compensation of the current block by performing motion compensation and pixel group unit motion compensation on the current block.

[0239] In operation S155, the video encoding device 150 can generate a bitstream that includes information about a first motion vector and a second motion vector, as well as motion prediction mode information indicating that the motion prediction mode of the current block is a bidirectional motion prediction mode. Here, the first motion vector may be a motion vector of a first reference block corresponding to the current block in the current frame, indicating a first reference frame from the current block, and the second motion vector may be a motion vector of a second reference block corresponding to the current block in the current frame, indicating a second reference frame from the current block.

[0240] When pixel group unit motion compensation is performed on the current block, parameters related to the pixel group unit motion compensation of the current block can be obtained from parameters related to the image including the current frame. However, the embodiments are not limited to this; when pixel group unit motion compensation is performed, parameters related to the pixel group unit motion compensation of the current block can be determined, and information about the determined parameters related to the pixel group unit motion compensation can be included in the bitstream.

[0241] The video encoding apparatus 150 can encode the residual signal of the current block and generate a bitstream that further includes the encoded residual signal, wherein the residual signal indicates the difference between the pixels of the predicted block of the current block and the pixels of the original block of the current block. The video encoding apparatus 150 can encode information about the prediction mode of the current block and a reference frame index, and generate a bitstream that further includes the encoded information about the prediction mode and the encoded reference frame index. For example, the video encoding apparatus 150 can encode information indicating that the prediction mode of the current block is an inter-frame prediction mode, and a reference frame index indicating at least one frame in a previously decoded frame, and generate a bitstream that further includes the encoded information about the prediction mode and the encoded reference frame index.

[0242] Figure 1e This is a block diagram of an image decoder 600 according to various embodiments.

[0243] Image decoding device 600 according to various embodiments performs operations for decoding image data performed by image decoder (not shown) of video decoding device 100.

[0244] Reference Figure 1e The entropy decoder 615 parses the encoded image data to be decoded, as well as the encoding information required for the decoding operation, from the bitstream 605. The encoded image data is the quantized transform coefficients, and the dequantizer 620 and the inverse transform 625 reconstruct the residual data from the quantized transform coefficients.

[0245] Intra-predictor 640 performs intra-prediction for each block. Inter-predictor 635 performs inter-prediction for each block using a reference image obtained from the reconstructed frame buffer 630. Figure 1eThe inter-frame predictor 635 can correspond to Figure 1a Inter-frame predictor 110.

[0246] Data in the spatial domain of a block of the current image can be reconstructed by adding the prediction data and residual data of each block generated by the intra-frame predictor 640 or the inter-frame predictor 635. The deblocking unit 645 and the SAO executor 650 can output a filtered reconstructed image by performing loop filtering on the reconstructed data in the spatial domain. Furthermore, the reconstructed image stored in the reconstructed image buffer 630 can be output as a reference image.

[0247] In order for the decoder (not shown) of the video decoding device 100 to decode the image, the step-by-step operation of the image decoder 600 according to various embodiments can be performed block by block.

[0248] Figure 1f This is a block diagram of an image encoder according to various embodiments.

[0249] The image encoder 700 according to various embodiments performs the operation of encoding image data performed by the image encoder (not shown) of the video encoding device 150.

[0250] In other words, the intra-predictor 720 performs intra-prediction for each block of the current image 705, and the inter-predictor 715 performs inter-prediction using the current image 705 for each block and a reference image obtained from the reconstructed frame buffer 710. Here, Figure 1e Inter-frame predictors can correspond to Figure 1c Inter-frame predictor 155.

[0251] Residual data can be generated by subtracting the prediction data for each block output from the intra-predictor 720 or inter-predictor 715 from the data of the coded blocks of the current image 705. Transformer 725 and quantizer 730 can output the quantized transform coefficients for each block by performing transform and quantization on the residual data. Inverse quantizer 745 and inverse transformer 750 can reconstruct the residual data in the spatial domain by performing inverse quantization and inverse transform on the quantized transform coefficients. The reconstructed spatial domain residual data can be added to the prediction data for each block output from the intra-predictor 720 or inter-predictor 715 to reconstruct the data in the spatial domain for the blocks of the current image 705. Deblocking unit 755 and SAO executor 760 generate a filtered reconstructed image by performing loop filtering on the reconstructed spatial domain data. The resulting filtered reconstructed image can be stored in reconstructed image buffer 710. The reconstructed image stored in reconstructed image buffer 710 can be used as a reference image for inter-frame prediction of another image. The entropy encoder 735 can entropy encode the quantized transform coefficients, and the entropy-encoded coefficients can be output as a bit stream 740.

[0252] In order to apply the image encoder 700 according to various embodiments to the video encoding device 150, the step-by-step operation of the image encoder 700 according to various embodiments can be performed block by block.

[0253] Figure 2 This is a reference diagram illustrating block-based bidirectional motion prediction and compensation processing according to an embodiment.

[0254] Reference Figure 2 The video encoding device 150 performs bidirectional motion prediction, wherein, in the bidirectional motion prediction, it searches in a first reference frame 210 and a second reference frame 220 for a region most similar to the current block 201 of the current frame 200 to be encoded. Here, the first reference frame 210 may be a frame preceding the current frame 200, and the second reference frame 220 may be a frame following the current frame 200. As a result of the bidirectional motion prediction, the video encoding device 150 determines a first corresponding region 212 most similar to the current block 201 from the first reference frame 210, and a second corresponding region 222 most similar to the current block 201 from the second reference frame 220. Here, the first corresponding region 212 and the second corresponding region 222 may be reference regions of the current block 201.

[0255] Furthermore, the video encoding device 150 can determine a first motion vector MV1 based on the positional difference between the first corresponding region 212 and the block 211 of the first reference frame 210 that is located at the same position as the current block 201, and determine a second motion vector MV2 based on the positional difference between the second corresponding region 222 and the block 221 of the second reference frame 220 that is located at the same position as the current block 201.

[0256] The video encoding device 150 performs block-unit bidirectional motion compensation on the current block 201 by using a first motion vector MV1 and a second motion vector MV2.

[0257] For example, when the pixel value at (i,j) in the first reference frame 210 is P0(i,j), the pixel value at (i,j) in the second reference frame 220 is P1(i,j), MV1=(MVx1,MVy1), and MV2=(MVx2,MVy2), the block unit bidirectional motion compensation value P_BiPredBlock(i,j) of the pixel at position (i,j) in the current block 201 can be calculated according to the equation P_BiPredBlock(i,j)={P0(i+MVx1, j+MVy1)+P1(i+MVx2, j+MVy2)} / 2, where i and j are both integers. As a result, the video encoding device 150 can generate block-based motion compensation values ​​by performing motion compensation on the current block 201 using the average or weighted sum of pixels in the first corresponding region 212 and the second corresponding region 222 indicated by the first motion vector MV1 and the second motion vector MV2.

[0258] Figures 3a to 3c This is a reference diagram illustrating the process of performing pixel-unit motion compensation according to an embodiment.

[0259] exist Figure 3a In the middle, the first corresponding region 310 and the second corresponding region 320 are respectively with Figure 2 The first corresponding region 212 and the second corresponding region 222 correspond to each other and can be shifted to overlap with the current block 300 by using bidirectional motion vectors MV1 and MV2.

[0260] Furthermore, P(i,j) represents the pixel of the current block 300 at position (i,j) predicted bidirectionally, P0(i,j) represents the first reference pixel value of the first reference image corresponding to the pixel P(i,j) of the current block 300 predicted bidirectionally, and P1(i,j) represents the second reference pixel value of the second reference image corresponding to the pixel P(i,j) of the current block 300 predicted bidirectionally, where i and j are both integers.

[0261] In other words, the first reference pixel value P0(i,j) is the pixel value of the pixel corresponding to the pixel P(i,j) of the current block 300, determined by the bidirectional motion vector MV1 that indicates the first reference screen, and the second reference pixel value P1(i,j) is the pixel value of the pixel corresponding to the pixel P(i,j) of the current block 300, determined by the bidirectional motion vector MV2 that indicates the second reference screen.

[0262] also, This represents the gradient value of the first reference pixel in the horizontal direction. This represents the gradient value of the first reference pixel in the vertical direction. This represents the gradient value of the second reference pixel in the horizontal direction. This represents the gradient value of the second reference pixel in the vertical direction. Furthermore, τ0 represents the time distance between the current frame to which the current block 300 belongs and the first reference frame to which the first corresponding region 310 belongs, and τ1 represents the time distance between the current frame and the second reference frame to which the second corresponding region 320 belongs. Here, the time distance between frames can be represented by the difference in the frame sequence count (POC) of the frames.

[0263] When there is uniform small motion in the video sequence, the pixel in the first corresponding area 310 of the first reference frame that is most similar to pixel P(i,j) which has undergone bidirectional motion compensation on a pixel-by-pixel basis is not the first reference pixel P0(i,j), but rather the first displacement reference pixel PA after the first reference pixel P0(i,j) has been moved according to a specific displacement vector. As described above, since there is uniform motion in the video sequence, the pixel in the second corresponding area 320 of the second reference frame that is most similar to pixel P(i,j) can be the second displacement reference pixel PB after the second reference pixel P1(i,j) has been moved according to a specific displacement vector.

[0264] The displacement vector may include a displacement vector Vx in the x-axis direction and a displacement vector Vy in the y-axis direction. Therefore, the pixel group unit motion compensator 165 calculates the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction included in the displacement vector, and performs motion compensation on a pixel group basis by using the displacement vector.

[0265] Optical flow represents a pattern of apparent motion on an object or surface caused by the relative motion between the scene and the observer (eye or a video image acquisition device similar to a camera). In a video sequence, optical flow can be represented by calculating the motion between frames acquired at arbitrary times t and t+Δt. The pixel value located at (x,y) in the frame at time t can be I(x,y,t). In other words, I(x,y,t) can be a value that varies in both time and space. I(x,y,t) can be differentiated with respect to time t according to Equation 1.

[0266] [Equation 1]

[0267]

[0268] When the pixel value changes with the motion but not with time for small movements within a block, dI / dt is 0. Furthermore, when the pixel value moves uniformly with time, dx / dt can represent the displacement vector Vx of the pixel value I(x,y,t) in the x-axis direction, and dy / dt can represent the displacement vector Vy of the pixel value I(x,y,t) in the y-axis direction. Therefore, Equation 1 can be expressed as Equation 2.

[0269] [Equation 2]

[0270]

[0271] Here, the magnitudes of the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction can have values ​​smaller than the pixel precision used in bidirectional motion prediction. For example, when the pixel precision is 1 / 4 or 1 / 16 during bidirectional motion prediction, the magnitudes of the displacement vectors Vx and Vy can have values ​​smaller than 1 / 4 or 1 / 16.

[0272] The pixel group unit motion compensator 165 calculates the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction according to Equation 2, and performs motion compensation on a pixel group basis by using the displacement vectors Vx and Vy. In Equation 2, since the pixel value I(x,y,t) is the value of the original signal, using the original pixel value would result in high overhead during encoding. Therefore, the pixel group unit motion compensator 165 can calculate the displacement vectors Vx and Vy according to Equation 2 by using the pixels of the first and second reference frames, which are determined as a result of performing bidirectional motion prediction on a block-by-block basis. In other words, the pixel group unit motion compensator 165 determines the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction, where Δ is minimized in a window Ωij having a specific size and including neighboring pixels around the pixel P(i,j) to which bidirectional motion compensation is performed. △ can be 0, but for all pixels in the window Ωij, the displacement vectors Vx and Vy in the x-axis direction that satisfy △=0 may not exist. Therefore, the displacement vectors Vx and Vy in the x-axis direction with minimum △ are determined. (Refer to...) Figure 8a Describe in detail the process of obtaining displacement vectors Vx and Vy.

[0273] To determine the predicted pixel value of the current pixel, the function P(t) with respect to t can be determined according to Equation 3.

[0274] [Equation 3]

[0275] P(t) = a3*t 3 +a2*t 2 +a1*t+a0

[0276] Here, the frame at t=0 is the current frame including the current block. Therefore, the predicted pixel value of the current pixel included in the current block can be defined as the value of P(t) at t=0.

[0277] When the time distance between the current frame and the first reference frame (which is prior to the current frame in time) is τ0, and the time distance between the current frame and the second reference frame (which is subsequent to the current frame in time) is τ1, the reference pixel value in the first reference frame is equal to P(-τ0), and the reference pixel value in the second reference frame is equal to P(τ1). In the following text, for ease of calculation, it is assumed that both τ0 and τ1 are equal to τ.

[0278] The coefficients of each level of P(t) can be determined according to Equation 4. Here, P0(i,j) can represent the pixel value at position (i,j) of the first reference frame, and P1(i,j) can represent the pixel value at position (i,j) of the second reference frame.

[0279] [Equation 4]

[0280]

[0281] Therefore, the predicted pixel value P(0) of the current pixel in the current block can be determined according to Equation 5.

[0282] [Equation 5]

[0283]

[0284] Considering Equation 2, Equation 5 can be expressed as Equation 6.

[0285] [Equation 6]

[0286]

[0287] Therefore, the predicted pixel value of the current pixel can be determined using displacement vectors Vx and Vy, the gradient values ​​of the first reference pixel in the horizontal and vertical directions, and the gradient values ​​of the second reference pixel in the horizontal and vertical directions. Here, the part unrelated to displacement vectors Vx and Vy (P0(i,j)+P1(i,j)) / 2) can be a block-based motion compensation value, while the part related to displacement vectors Vx and Vy can be a pixel-group-based motion compensation value. As a result, the predicted pixel value of the current pixel can be determined by adding the block-based motion compensation value and the pixel-group-based motion compensation value.

[0288] In the preceding text, for ease of description, the process of determining the predicted pixel value of the current pixel was described when the time distance between the first reference frame and the current frame, and the time distance between the second reference frame and the current frame are both τ and therefore the same. However, the time distance between the first reference frame and the current frame can be τ0, and the time distance between the second reference frame and the current frame can be τ1. Here, the predicted pixel value P(0) of the current pixel can be determined according to Equation 7.

[0289] [Equation 7]

[0290]

[0291] Considering Equation 2, Equation 7 can be expressed as Equation 8.

[0292]

[0293] In the above text, the first reference screen is displayed after the current screen in time, and the second reference screen is displayed before the current screen in time. However, optionally, the first reference screen and the second reference screen may both be displayed before the current screen in time, or both may be displayed after the current screen in time.

[0294] For example, such as Figure 3b As shown, the first reference screen including the first corresponding area 310 and the second reference screen including the second corresponding area 320 can both be displayed in time before the current screen including the current block 300.

[0295] In this case, the predicted pixel value P(0) of the current pixel can be determined according to Equation 9, where -τ1 is replaced by -τ1 in the reference. Figure 3a Equation 8 shows τ1, which indicates the time distance between the second reference frame and the current frame.

[0296] [Equation 9]

[0297]

[0298] For example, such as Figure 3c As shown, the first reference screen including the first corresponding area 310 and the second reference screen including the second corresponding area 320 can both be displayed in time after the current screen including the current block 300.

[0299] In this case, the predicted pixel value P(0) of the current pixel can be determined according to Equation 10, where -τ0 is replaced by -τ0 in the reference. Figure 3a Equation 8 shows τ0, which indicates the time distance between the first reference frame and the current frame.

[0300] [Equation 10]

[0301]

[0302] However, as Figure 3b and Figure 3c As shown, pixel-group unit motion compensation can be performed when the first reference frame and the second reference frame are both displayed before or after the current frame in time. Furthermore, in this case, pixel-group unit motion compensation can be performed only when both bidirectional motion vectors MV1 and MV2 have non-zero components. Additionally, in this case, pixel-group unit motion compensation can be performed only when the ratio of motion vectors MV1 and MV2 is the same as the ratio of the time distance between the first reference frame and the current frame to the time distance between the second reference frame and the current frame. For example, pixel-group unit motion compensation can be performed when the ratio of the x-component of motion vector MV1 to the x-component of motion vector MV2 is the same as the ratio of the y-component of motion vector MV1 to the y-component of motion vector MV2, and is also the same as the ratio of the time distance τ0 between the first reference frame and the current frame to the time distance τ1 between the second reference frame and the current frame.

[0303] Figure 4 This is a reference diagram illustrating the process of calculating gradient values ​​in the horizontal direction and gradient values ​​in the vertical direction, according to an embodiment.

[0304] Reference Figure 4 The gradient value of the first reference pixel P0(i,j) 410 in the horizontal direction can be calculated by obtaining the changes in pixel values ​​at neighboring fractional pixel positions adjacent to the first reference pixel P0(i,j) 410 in the horizontal direction, and the changes in pixel values ​​at neighboring fractional pixel positions adjacent to the first reference pixel P0(i,j) 410 in the vertical direction. The gradient value of the first reference pixel P0(i,j) 410 in the vertical direction In other words, according to Equation 11, the gradient value in the horizontal direction can be calculated by calculating the change in pixel value of fractional pixels P0(ih,j)460 and P0(i+h,j)470 that are h away from pixel P0(i,j) in the horizontal direction. Where h is a fraction less than 1, the gradient value in the vertical direction can be calculated by measuring the change in pixel value of fractional pixels P0(i,jh)480 and P0(i,j+h)490 that are h distances from pixel P0(i,j) in the vertical direction. .

[0305] [Equation 11]

[0306]

[0307] The pixel values ​​of fractional pixels P0(ih,j) 460, P0(i+h,j) 470, P0(i,jh) 480, and P0(i,j+h) 490 can be calculated using general interpolation. Furthermore, the gradient values ​​in the horizontal and vertical directions of the second reference pixels in the second reference frame can be calculated similarly to Equation 11.

[0308] According to an embodiment, instead of calculating the gradient value by calculating the change in pixel value at a fractional pixel location as in Equation 11, the gradient value at a reference pixel can be calculated using a specific filter. Considering the linearity of the filter, the filter coefficients of the specific filter can be determined based on the coefficients of the interpolation filter used to obtain the pixel value at the fractional pixel location.

[0309] Figure 5 This is a reference diagram according to another embodiment for describing the process of calculating gradient values ​​in the horizontal direction and gradient values ​​in the vertical direction.

[0310] According to another embodiment, gradient values ​​can be determined by applying a specific filter to pixels of a reference image. (See reference...) Figure 5 The video decoding device 100 can apply a specific filter to |M based on the reference pixel P0 500 to which the current horizontal gradient value will be obtained. Min | 520 pixels on the left and M Max The rightmost pixel 510 is used to calculate the gradient value of the reference pixel P0 500 in the horizontal direction. For example... Figures 7a to 7d As shown, the M value used to determine the window size can be determined according to the instructions. Max and M Min The value α, representing the interpolation position (fractional pel position) between integer pixels, determines the filter coefficients used here. For example, refer to Figure 7a When M is used to determine the window size Min and M Max They are -2 and 3 respectively, and M Max and M Min When the distance from the reference pixel P0 500 is 1 / 4, that is, when α = 1 / 4, Figure 7a The filter coefficients {4, -17, -36, 60, -15, 4} in the second row are applied to the neighboring pixel P. -2 P -1 P0, P1, P2, and P3. In this case, the gradient value of the reference pixel P0 500 in the horizontal direction can be calculated by using the filter coefficients and the weighted sum of neighboring pixels. Such as equations =4*P -2- 17*P -1 + -36*P0+60*P1-15*P2+4*P3+ 32>>6. Similarly, it can also be done by determining the interpolation position and M used to determine the window size. Max and M Min ,Will Figures 7a to 7e The filter coefficients shown are applied to neighboring pixels to calculate the gradient value in the vertical direction.

[0311] Figure 6a and Figure 6b This is a diagram illustrating a process for determining gradient values ​​in the horizontal and vertical directions using a 1D filter, according to an embodiment.

[0312] Reference Figure 6a Filtering can be performed by using multiple 1D filters for integer pixels to determine the gradient value of the reference pixel in the horizontal direction within the reference frame. Motion compensation at the pixel group level is additional motion compensation performed after block-level motion compensation. Therefore, the reference position of the reference block of the current block, indicated by the motion vector during block-level motion compensation, can be a fractional pixel position, and pixel group-level motion compensation can be performed for the reference pixel in the reference block at the fractional pixel position. Thus, filtering can be performed considering that the gradient value of the pixel at the fractional pixel position is determined.

[0313] Reference Figure 6a First, the video decoding device 100 can perform filtering on pixels located in the horizontal or vertical direction starting from the nearest integer pixels of the reference pixel in the reference frame using a first 1D filter. Similarly, the video decoding device 100 can perform filtering on nearest integer pixels in rows or columns different from the reference pixel using the first 1D filter. The video decoding device 100 can generate the gradient value of the reference pixel in the horizontal direction by performing filtering on the values ​​generated by the filtering using a second 1D filter.

[0314] For example, when the reference pixel is located at the position of a fractional pixel at (x+α, y+β) (where x and y are integers, and α and β are fractions), it can be calculated by targeting the integer pixels (x,y), (x-1,y), (x+1,y), to (x+M) in the horizontal direction. Min (,y) and (x+M) Max ,y) uses a 1D vertical interpolation filter, and the filtering is performed according to Equation 12, where M Min and M Mmax All are integers.

[0315] [Equation 12]

[0316]

[0317] Here, fracFilter β FracFilter can represent an interpolation filter used to determine the pixel value at a fractional pixel position β in the vertical direction. β [j'] can represent the coefficients of the interpolation filter applied to the pixel at position (i,j'). I[i,j'] can represent the pixel value at position (i,j').

[0318] In other words, the first 1D filter can be an interpolation filter used to determine fractional pixel values ​​in the vertical direction. `offset1` can represent the offset used to prevent rounding errors, and `shift1` can represent the descaling bit count. `Temp[i,j+β]` can represent the pixel value at the fractional pixel position (i,j+β). `Temp[i',j+β]` can also be determined by replacing `i` with `i'` according to Equation 12, where `i'` is from `i+M`. min To i+M max Integers excluding i.

[0319] Then, the video decoding device 100 can perform filtering on the pixel value at fractional pixel position (i,j+β) and the pixel value at fractional pixel position (i',j+β) by using a second 1D filter.

[0320] [Equation 13]

[0321]

[0322] Here, gradFilter α This can be a gradient filter used to determine the gradient value at a fractional pixel position α in the horizontal direction. (gradFilter) α [i'] can represent the coefficients of the gradient filter applied to the pixel at position (i', j+β). In other words, the second 1D filter can be a gradient filter used to determine the gradient value in the horizontal direction. offset2 can represent the offset used to prevent rounding errors, and shift2 can represent the descaling bit count.

[0323] In other words, according to Equation 13, the video decoding device 100 can decode the video by using a gradient filter (gradFilter). α Filter the pixel value (Temp[i,j+β]) at pixel position (i, j+β) and the pixel value (Temp[i',j+β]) in the vertical direction starting from pixel position (i, j+β) to determine the gradient value in the horizontal direction at (i+α,j+β). .

[0324] In the preceding text, the gradient value in the horizontal direction was determined by first applying an interpolation filter and then applying a gradient filter. Alternatively, the gradient value in the horizontal direction can be determined by first applying a gradient filter and then applying an interpolation filter. Below, an embodiment of determining the gradient value in the horizontal direction by applying a gradient filter and then applying an interpolation filter will be described.

[0325] For example, when the reference pixel is located at the position of a fractional pixel at (x+α, y+β) (where x and y are integers, and α and β are fractions), it can be determined by targeting the integer pixels (x,y), (x-1,y), (x+1, y), to (x+M) in the horizontal direction. Min (,y) and (x+M) Max ,y) Using the first 1D filter, filtering is performed according to Equation 14, where M Min and M Mmax All are integers.

[0326] [Equation 14]

[0327]

[0328] Here, gradFilter α Gradient filters, gradFilter, can be used to determine the gradient value at a fractional pixel position α in the vertical direction. α [i'] can represent the coefficients of the gradient filter applied to the pixel at position (i',j). I[i',j] can represent the pixel value at position (i',j).

[0329] In other words, the first 1D filter can be a gradient filter used to determine the gradient value of a pixel in the horizontal direction, where the horizontal component of the pixel position is a fractional position. `offset3` can represent the offset used to prevent rounding errors, and `shift3` can represent the descaling bit count. `Temp[i+α,j]` can represent the gradient value at the pixel position (i+α,j) in the horizontal direction. `Temp[i+α,j']` can also be determined by replacing `j` with `j'` according to Equation 14, where `j'` is from `j+M`. min to j+M max The integers that do not include j.

[0330] Then, the video decoding device 100 can perform filtering on the gradient value at the pixel position (i+α,j) in the horizontal direction and the gradient value at the pixel position (i+α,j') in the horizontal direction by using a second 1D filter according to Equation 15.

[0331] [Equation 15]

[0332]

[0333] Here, fracFilter β It can be an interpolation filter used to determine the pixel value at a fractional pixel position β in the vertical direction. fracFilter β [j'] can represent the coefficients of the interpolation filter applied to the pixel at position (i+β, j'). In other words, the second 1D filter can be an interpolation filter used to determine the pixel value at the fractional pixel position β in the vertical direction. offset4 can represent the offset used to prevent rounding errors, and shift4 can represent the descaling bit count.

[0334] In other words, according to Equation 15, the video decoding device 100 can decode the video by using an interpolation filter fracFilter. β The gradient value in the horizontal direction at pixel position (i+α, j) is determined by filtering the gradient value (Temp[i+α, j]) at the horizontal position (i+α, j) and the gradient value in the horizontal direction of the pixel located in the vertical direction from pixel position (i+α, j) at the horizontal position (Temp[i+α, j']). .

[0335] Reference Figure 6b Filtering can be performed by using multiple 1D filters for integer pixels to determine the gradient value of the reference pixel in the vertical direction within the reference frame. Motion compensation at the pixel group level is an additional motion compensation performed after block-level motion compensation. Therefore, the reference position of the reference block of the current block, indicated by the motion vector during block-level motion compensation, can be a fractional pixel position, and pixel group-level motion compensation can be performed for the reference pixel in the reference block at the fractional pixel position. Thus, filtering can be performed considering that the gradient value of the pixel at the fractional pixel position is determined.

[0336] Reference Figure 6b First, the video decoding device 100 can perform filtering on pixels located in the horizontal or vertical direction starting from the nearest integer pixels of the reference pixel in the reference frame using a first 1D filter. Similarly, the video decoding device 100 can perform filtering on nearest integer pixels in rows or columns different from the reference pixel using the first 1D filter. The video decoding device 100 can generate the gradient value of the reference pixel in the vertical direction by performing filtering on the values ​​generated by the filtering using a second 1D filter.

[0337] For example, when the reference pixel is located at the position of a fractional pixel at (x+α, y+β) (where x and y are integers, and α and β are fractions), it can be determined by targeting the integer pixels (x, y), (x-1, y-1), (x+1, y+1), to (x+M) in the horizontal direction. Min ,y+M Min ) and (x+M Max ,y+M Max Using the first 1D filter, filtering is performed according to Equation 16, where M Min and M Mmax All are integers.

[0338] [Equation 16]

[0339]

[0340] Here, fracFilter α FracFilter can represent an interpolation filter used to determine the pixel value at a fractional pixel position α in the horizontal direction. α [i'] can represent the coefficients of the interpolation filter applied to the pixel at position (i',j). I[i',j] can represent the pixel value at position (i',j).

[0341] In other words, the first 1D filter can be an interpolation filter used to determine the pixel value at a fractional pixel position α in the horizontal direction. offset5 can represent the offset used to prevent rounding errors, and shift5 can represent the descaling bit count.

[0342] Temp[i+α,j] can represent the pixel value at the fractional pixel position (i+α,j). Alternatively, Temp[i+α,j'] can be determined according to Equation 16 by replacing j with j', where j' is from j+M. min to j+M max The integers that do not include j.

[0343] Then, the video decoding device 100 can perform filtering on the pixel value at pixel position (i+α,j) and the pixel value at pixel position (i+α,j') according to Equation 17 using a second 1D filter.

[0344] [Equation 17]

[0345]

[0346] Here, gradFilter β This can be a gradient filter used to determine the gradient value at a fractional pixel position β in the vertical direction. (gradFilter) β[j'] can represent the coefficients of the gradient filter applied to the pixel at position (i+α,j'). In other words, the second 1D filter can be a gradient filter used to determine the gradient value in the vertical direction at the fractional pixel position β. offset6 can represent the offset used to prevent rounding errors, and shift6 can represent the descaling bit count.

[0347] In other words, according to Equation 17, the video decoding device 100 can decode the video by using a gradient filter (gradFilter). β Filter the pixel value (Temp[i+α,j]) at pixel position (i+α,j) and the pixel value (Temp[i+α,j']) in the vertical direction starting from pixel position (i+α,j). .

[0348] In the preceding text, the gradient value in the vertical direction was determined by first applying an interpolation filter and then applying a gradient filter. Alternatively, the gradient value in the vertical direction can be determined by first applying a gradient filter and then applying an interpolation filter. Below, an embodiment of determining the gradient value in the vertical direction by applying a gradient filter and then applying an interpolation filter will be described.

[0349] For example, when the reference pixel is located at the position of a fractional pixel at (x+α, y+β) (where x and y are integers, and α and β are fractions), it can be determined by targeting integer pixels (x,y), (x,y-1), (x, y+1), and up to (x,y+M) in the vertical direction. Min ) and (x,y+M Max Using the first 1D filter, filtering is performed according to Equation 18, where M Min and M Mmax All are integers.

[0350] [Equation 18]

[0351]

[0352] Here, gradFilter β Gradient filters, gradFilter, can be used to determine the gradient value at a fractional pixel position β in the vertical direction. β [j'] can represent the coefficients of the gradient filter applied to the pixel at position (i,j'). I[i,j'] can represent the pixel value at position (i,j').

[0353] In other words, the first 1D filter can be a gradient filter used to determine the gradient value of a pixel in the vertical direction, where the vertical component of the pixel position is a fractional position. offset7 can represent an offset used to prevent rounding errors, and shift7 can represent descaling the number of bits.

[0354] Temp[i,j+β] can represent the gradient value at pixel position (i,j+β) in the vertical direction. Alternatively, Temp[i',j+β] can be determined according to Equation 18 by replacing i with i', where i' is the gradient value from i+M. min To i+M max Integers excluding i.

[0355] Then, the video decoding device 100 can perform filtering on the gradient value at the pixel position (i, j+β) in the vertical direction and the gradient value at the pixel position (i', j+β) in the vertical direction by using a second 1D filter according to Equation 19.

[0356] [Equation 19]

[0357]

[0358] Here, fracFilter α It can be an interpolation filter used to determine the pixel value at a fractional pixel position α in the horizontal direction. fracFilter α [i'] can represent the coefficients of the interpolation filter applied to the pixel at position (i', j+β). In other words, the second 1D filter can be an interpolation filter used to determine the pixel value at a fractional pixel position α in the horizontal direction. offset8 can represent the offset used to prevent rounding errors, and shift8 can represent the descaling bit count.

[0359] In other words, according to Equation 19, the video decoding device 100 can decode the video by using an interpolation filter fracFilter. α The gradient value in the vertical direction at pixel position (i, j+β) is determined by filtering the gradient value (Temp[i,j+β]) at the vertical position (i, j+β) and the gradient value in the vertical direction of the pixels located in the horizontal direction from pixel position (i, j+β) at the horizontal position (Temp[i',j+β)]). .

[0360] According to an embodiment, in the video decoding device 100, the gradient values ​​in the horizontal and vertical directions at (i+α, j+β) can be determined based on combinations of various filters described above. For example, to determine the gradient value in the horizontal direction, an interpolation filter for determining the pixel value in the vertical direction can be used as a first 1D filter, and a gradient filter for determining the gradient value in the horizontal direction can be used as a second 1D filter. Optionally, a gradient filter for determining the gradient value in the vertical direction can be used as a first 1D filter, and an interpolation filter for determining the pixel value in the horizontal direction can be used as a second 1D filter.

[0361] Figures 7a to 7e This is a table showing filter coefficients for a filter used to determine pixel values ​​at fractional pixel positions in fractional pixel units, as well as gradient values ​​in the horizontal and vertical directions, according to an embodiment.

[0362] Figure 7a and Figure 7b This is a table showing the filter coefficients of a filter used to determine the gradient value at a fractional pixel position of 1 / 4 pel unit in the horizontal or vertical direction.

[0363] As described above, 1D gradient filters and 1D interpolation filters can be used to determine gradient values ​​in the horizontal or vertical directions. (See reference...) Figure 7a The diagram illustrates the filter coefficients of a 1D gradient filter. Here, a 6-tap filter can be used as a 1D gradient filter. The filter coefficients of the 1D gradient filter can be coefficients scaled by 2^4. Mmin represents the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied to the filter in the negative direction based on the center integer pixel, and Mmax represents the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied to the filter in the positive direction based on the center integer pixel. For example, the gradient filter coefficients for obtaining the gradient value of a pixel in the horizontal direction, where the fractional pixel position α is 1 / 4 in the horizontal direction, can be {4, -17, -36, 60, -15, -4}. (Refer to...) Figure 7a It can also determine the gradient filter coefficients used to obtain the gradient value of the pixel in the horizontal direction, where the fractional pixel position α is 0, 1 / 2, or 3 / 4 in the horizontal direction.

[0364] Reference Figure 7bThe diagram illustrates the filter coefficients of a 1D interpolation filter. Here, a 6-tap filter can be used as a 1D interpolation filter. The filter coefficients of the 1D interpolation filter can be coefficients scaled by 2^6. Mmin represents the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied to the filter in the negative direction based on the center integer pixel, and Mmax represents the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied to the filter in the positive direction based on the center integer pixel.

[0365] Figure 7c This is a table showing the filter coefficients of a 1D interpolation filter used to determine the pixel value at a fractional pixel location of 1 / 4 pel unit.

[0366] As described above, two identical 1D interpolation filters can be used in the horizontal and vertical directions to determine the pixel value at the fractional pixel location.

[0367] Reference Figure 7c The diagram illustrates the filter coefficients of a 1D interpolation filter. Here, a 6-tap filter can be used as a 1D interpolation filter. The filter coefficients of the 1D interpolation filter can be coefficients scaled by 2^6. Mmin represents the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied to the filter in the negative direction based on the center integer pixel, and Mmax represents the difference between the position of the center integer pixel and the position of the farthest pixel among the integer pixels applied to the filter in the positive direction based on the center integer pixel.

[0368] Figure 7d This is a table showing the filter coefficients of a filter used to determine the gradient value in the horizontal or vertical direction at a fractional pixel position of 1 / 16 pel unit.

[0369] As described above, 1D gradient filters and 1D interpolation filters can be used to determine gradient values ​​in the horizontal or vertical directions. (See reference...) Figure 7d The diagram illustrates the filter coefficients of a 1D gradient filter. Here, a 6-tap filter can be used as a 1D gradient filter. The filter coefficients of a 1D gradient filter can be coefficients scaled by 2^4. For example, the gradient filter coefficients for obtaining the gradient value of a pixel in the horizontal direction, where the fractional pixel position α is 1 / 16 in the horizontal direction, could be {8, -32, -13, 50, -18, 5}. (Refer to...) Figure 7dFurthermore, gradient filter coefficients for obtaining the horizontal gradient value of a pixel can be determined when the fractional pixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, or 1 / 2 in the horizontal direction. Simultaneously, gradient filter coefficients for obtaining the horizontal gradient value of a pixel can be determined by using the symmetry of the filter coefficients based on α=1 / 2 when the fractional pixel position α is 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, or 15 / 16 in the horizontal direction. In other words, gradient filter coefficients for obtaining the horizontal gradient value of a pixel can be determined by using... Figure 7d The filter coefficients at the right fractional pixel position based on α=1 / 2 are shown in the diagram. For example, the filter coefficients at α=15 / 16 can be determined using the filter coefficients {8, -32, -13, 50, -18, 5} at α=1 / 16, which is a symmetrical position based on α=1 / 2. In other words, the filter coefficients at α=15 / 16 can be determined as {5, -18, 50, -13, -32, 8} by reversing the order of {8, -32, -13, 50, -18, 5}.

[0370] Reference Figure 7e The diagram illustrates the filter coefficients of a 1D interpolation filter. Here, a 6-tap filter can be used as a 1D interpolation filter. The filter coefficients of a 1D interpolation filter can be coefficients scaled by 2^6. For example, the 1D interpolation filter coefficients for obtaining the pixel value in the horizontal direction, where the fractional pixel position α is 1 / 16 in the horizontal direction, could be {1, -3, 64, 4, -2, 0}. (Refer to...) Figure 7e This also allows us to determine interpolation filter coefficients for obtaining the pixel value in the horizontal direction when the fractional pixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, or 1 / 2 in the horizontal direction. Simultaneously, we can determine interpolation filter coefficients for obtaining the pixel value in the horizontal direction when the fractional pixel position α is 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, or 15 / 16 in the horizontal direction by using the symmetry of the filter coefficients based on α=1 / 2. In other words, we can use... Figure 7eThe filter coefficients at the right fractional pixel position based on α=1 / 2 are shown in the diagram. For example, the filter coefficients at α=15 / 16 can be determined using the filter coefficients at α=1 / 16, which are symmetrical to α=1 / 2, as {1, -3, 64, 4, -2, 0}. In other words, the filter coefficients at α=15 / 16 can be determined as {0, -2, 4, 64, -3, 1} by reversing the order of {1, -3, 64, 4, -2, 0}.

[0371] Figure 8a This is a reference diagram according to an embodiment for describing the process of determining the horizontal and vertical displacement vectors of a pixel.

[0372] Reference Figure 8a The size of the window Ωij 800 with a specific size is (2M+1)*(2N+1) based on the bidirectionally predicted pixels P(i,j) in the current block, where M and N are both integers.

[0373] When P(i',j') represents the pixel of the current block in window Ωij 800 that is bidirectionally predicted (where iM≤i'≤i+M and jN≤j'≤j+N, (i',j')∈Ωij), P0(i',j') represents the pixel value of the first reference pixel in the first reference frame 810 corresponding to the pixel P(i',j') of the current block that is bidirectionally predicted, and P1(i',j') represents the pixel value of the second reference pixel in the second reference frame 820 corresponding to the pixel P(i',j') of the current block that is bidirectionally predicted. This represents the gradient value of the first reference pixel in the horizontal direction. This represents the gradient value of the first reference pixel in the vertical direction. This represents the gradient value of the second reference pixel in the horizontal direction, and When representing the gradient value of the second reference pixel in the vertical direction, the first displacement-corresponding pixel PA' and the second displacement-corresponding pixel PB' can be determined according to Equation 20. Here, PA' and PB' can be determined by using the first linear term of the local Taylor expansion.

[0374] [Equation 20]

[0375]

[0376] In Equation 20, the displacement vector Vx in the x-axis direction and the displacement vector Vy in the y-axis direction can be changed according to the position of pixel P(i,j) (i.e., depending on (i,j)), and the displacement vectors Vx and Vy can be represented as Vx(i,j) and Vy(i,j).

[0377] The difference △i'j' between the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can be determined according to Equation 21.

[0378] [Equation 21]

[0379]

[0380] The displacement vectors Vx in the x-axis direction and Vy in the y-axis direction that minimize the difference △i'j' between the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can be determined by using the sum of squares of the differences △i'j' as in Equation 22, Φ(Vx,Vy).

[0381] [Equation 22]

[0382]

[0383] In other words, the displacement vectors Vx and Vy can be determined using the local maximum or local minimum of Φ(Vx,Vy). Φ(Vx,Vy) represents a function with the displacement vectors Vx and Vy as parameters, and the local maximum or local minimum can be determined by calculating the value of Φ(Vx,Vy) rearranged with respect to τVx and τVy according to Equation 23, which becomes 0. In the following text, for ease of calculation, τ0 and τ1 are the same, i.e., both are τ.

[0384] [Equation 23]

[0385]

[0386] By using equations Sum of equations This yields two linear equations, as shown in Equation 24, using Vx(i,j) and Vy(i,j) as variables.

[0387] [Equation 24]

[0388]

[0389] In Equation 24, s1 to s6 can be calculated according to Equation 25.

[0390] [Equation 25]

[0391]

[0392] By solving the simultaneous equations in Equation 24, the values ​​of Vx(i,j) and Vy(i,j) can be obtained using the Cramer formula: τ*Vx(i,j)=-det1 / det and τ*Vy(i,j)=-det2 / det. Here, det1=s3*s5-s2*s6, det2=s1*s6-s3*s4, and det=s1*s5-s2*s2.

[0393] The simplified solution to the above equations can be determined by first performing minimization in the horizontal direction and then minimization in the vertical direction. In other words, when only the displacement vector in the horizontal direction changes, Vy=0 in the first equation of Equation 24, thus the equation τVx=s3 / s1 can be determined.

[0394] Then, when the second equation of equation 24 is rearranged using the equation τVx=s3 / s1, the equation τVy=(s6-τVx*S2) / s5 can be determined.

[0395] Here, the gradient values ​​can be adjusted without changing the resulting values ​​Vx(i,j) and Vy(i,j). , , and Scaling is performed. However, this is contingent on preventing overflow and rounding errors.

[0396] The regularization parameters r and m are introduced to prevent division by zero or very small values ​​from being performed when calculating Vx(i,j) and Vy(i,j).

[0397] For convenience, we assume that Vx(i,j) and Vy(i,j) are the same as... Figure 3a The directions shown are opposite. For example, apart from the symbols, Equation 24 is based on... Figure 3a The directions of Vx(i,j) and Vy(i,j) derived from them can have the same properties as those determined by the given directions. Figure 3a Vx(i,j) and Vy(i,j) are opposite in direction and have the same size.

[0398] The first displacement-corresponding pixel PA' and the second displacement-corresponding pixel PB' can be determined according to Equation 26. Here, the first displacement-corresponding pixel PA' and the second displacement-corresponding pixel PB' can be determined by using the first linear term of the local Taylor expansion.

[0399] [Equation 26]

[0400]

[0401] The difference △i'j' between the first displacement corresponding pixel PA' and the second displacement corresponding pixel PB' can be determined according to Equation 27.

[0402] [Equation 27]

[0403]

[0404] The displacement vectors Vx and Vy in the x-axis direction that minimize the difference Δi'j' between the first displacement-corresponding pixel PA' and the second displacement-corresponding pixel PB' can be determined by using the sum of squares of the differences Δ as shown in Equation 28, Φ(Vx,Vy). In other words, the displacement vectors Vx and Vy can be determined when Φ(Vx,Vy) is minimized as shown in Equation 29, and can be determined by using the local maximum or local minimum of Φ(Vx,Vy).

[0405] [Equation 28]

[0406]

[0407] [Equation 29]

[0408]

[0409] Φ(Vx,Vy) is a function that takes displacement vectors Vx and Vy as parameters, and the local maximum or local minimum can be determined by calculating the value that makes Φ(Vx,Vy) equal to 0 by partial differentiation of displacement vectors Vx and Vy as shown in Equation 30.

[0410] [Equation 30]

[0411]

[0412] In other words, the displacement vectors Vx and Vy that minimize Φ(Vx,Vy) can be determined. To solve the optimization problem, minimization can be performed first in the vertical direction and then in the horizontal direction. Based on the minimization, the displacement vector Vx can be determined according to Equation 31.

[0413] [Equation 31]

[0414]

[0415] Here, the function clip3(x,y,z) is a function that outputs x when z < x, outputs y when z > y, and outputs z when x < z < y. According to Equation 31, when s1 + r > m, the displacement vector Vx can be clip3(-thBIO,thBIO,-s3 / (s1 + r)), and when s1 + r is not greater than m, the displacement vector Vx can be 0.

[0416] Based on minimization, the displacement vector Vy can be determined according to Equation 32.

[0417] [Equation 32]

[0418]

[0419] Here, the function clip3(x,y,z) is a function that outputs x when z < x, outputs y when z > y, and outputs z when x < z < y. According to Equation 32, when s5 + r > m, the displacement vector Vy can be clip3(-thBIO,thBIO,-(s6-Vx*s2) / 2 / (s5 + r), and when s5 + r is not greater than m, the displacement vector Vy can be 0.

[0420] Here, s1, s2, s3 and s5 can be determined according to Equation 33.

[0421] [Equation 33]

[0422]

[0423] As mentioned above, r and m can be regularization parameters introduced to avoid division results in values ​​of 0 or less than 0 and determined according to Equation 34 based on the internal bit depth d of the input video. In other words, the regularization parameter m is the minimum allowed denominator, and the regularization parameter r can be introduced to avoid division where 0 is used as the denominator when the gradient value is 0.

[0424] [Equation 34]

[0425]

[0426] The displacement vectors Vx and Vy can have upper and lower limits ±thBIO. Since motion compensation at the pixel group level may be unreliable due to noise or irregular motion, the displacement vectors Vx and Vy can be capped by a specific threshold thBIO. The regularization parameter thBIO can be determined based on whether the orientations of all reference frames are the same. For example, when the orientations of all reference frames are the same, the regularization parameter thBIO can be determined as 12^(d-8-1) or 12*2^(14-d). When the orientations of all reference frames are different, thBIO can be determined as 12^(d-8-1) / 2 or 12*2^(13-d).

[0427] However, the embodiments are not limited to this; the values ​​of regularization parameters r, m, and thBIO can be determined based on information about the regularization parameters obtained from the bitstream. Here, information about the regularization parameters can be included in a high-level syntax carrier in a strip header, a frame parameter set, a sequence parameter set, or other various other forms.

[0428] Furthermore, the regularization parameters r, m, and thBIO can be determined based on image-related parameters. For example, the regularization parameters r, m, and thBIO can be determined based on at least one of the following: bit depth of the sample, size of the GOP, distance to the reference frame, motion vector, index of the reference frame, availability of bidirectional prediction in different time directions, frame rate, and setting parameters related to the coding prediction structure.

[0429] For example, regularization parameters can be determined based on the GOP size. For instance, when the GOP size is 8 and the encoding prediction structure is random access, thBIO can be 12^(d-8-1). When the GOP size is 16 (i.e., twice 8), thBIO can be determined as 2*2^(d-8-1).

[0430] Furthermore, the video decoding device 100 can determine regularization parameters based on the distance to the reference frame. Here, the distance to the reference frame can represent the point of interest (POC) difference between the current frame and the reference frame. For example, when the distance to the reference frame is small, thBIO can be determined to be small, and when the distance to the reference frame is large, thBIO can be determined to be large.

[0431] The video decoding device 100 can determine the regularization parameters based on the motion vector of the block. For example, when the magnitude of the block's motion vector is small, thBIO can be determined to be small, and when the magnitude of the block's motion vector is large, thBIO can be determined to be large. Furthermore, for example, when the angle of the block's motion vector is close to 0 and therefore has only a horizontal component (typically, the horizontal component is greater than the vertical component), thBIO for the vertical displacement vector can be determined to be small, and thBIO for the horizontal displacement vector can be determined to be large.

[0432] The video decoding device 100 can determine the regularization parameters based on a reference frame index. The reference frame index indicates the frame closer to the current frame when the value of the reference frame index is small. Therefore, when the reference frame index is small, thBIO can be determined to be small, and when the reference frame index is large, thBIO can be determined to be large.

[0433] Furthermore, regularization parameters can be determined based on the availability of bidirectional forecasts at different times. For example, when bidirectional forecasts at different times are available, thBIO... diff It can be greater than thBIO when bidirectional predictions are available at the same time. same And thBIO diff The size can be thBIO same Twice the size.

[0434] The video decoding device 100 can determine the regularization parameters based on the frame rate. Even when the GOP size is the same, the time distance between frames is shorter at a higher frame rate, so the video decoding device 100 can determine that thBIO has a smaller value.

[0435] The video decoding device 100 can determine regularization parameters based on settings associated with the coding prediction structure. For example, settings associated with the coding prediction structure may indicate random access or low latency. When the settings associated with the coding prediction structure indicate low latency, the thBIO value can be determined to be small because future frames are not referenced in time. When the settings associated with the coding prediction structure indicate random access, the thBIO value can be determined to be relatively large.

[0436] The video decoding device 100 can determine the regularization parameters r and m based on the bit depth of the sample. The regularization parameters r and m can be proportional to s1 and s5 in Equation 25. Since the regularization parameters r and m consist of gradient multiplication, r and m increase as the gradient value increases. For example, when the bit depth d of the sample increases, the gradient value can increase, and therefore the magnitudes of the regularization parameters r and m can increase. Figure 8b This is a reference diagram according to an embodiment for describing the process of determining the horizontal and vertical displacement vectors of a pixel group.

[0437] Reference Figure 8b The window Ωij 800 with a specific size has a size of (2M+K+1)*(2N+K+1), where (2M+K+1)*(2N+K+1) is based on pixel group 820 rather than on the pixels of the current block to which bidirectional prediction is performed, where pixel group 820 has a size of K×K and includes multiple pixels, where M and N are both integers.

[0438] Here, with Figure 8a The difference lies in the larger window size, and apart from that difference, the horizontal and vertical displacement vectors of the pixel group can be determined in the same way.

[0439] Figure 9a This is a diagram illustrating, according to an embodiment, the process of adding an offset value after filtering is performed and the process of determining gradient values ​​in the horizontal or vertical direction by performing descaling.

[0440] Reference Figure 9aThe video decoding device 100 can determine gradient values ​​in the horizontal or vertical direction by performing filtering on pixels whose components in a specific direction are integer positions using a first 1D filter and a second 1D filter. However, the values ​​obtained by performing filtering on pixels whose components in a specific direction are integer positions using a first 1D filter and a second 1D filter can exceed a certain range. This phenomenon is called overflow. The coefficients of the 1D filter can be determined as integers for integer operations rather than inaccurate and complex fractional operations. The coefficients of the 1D filter can be scaled to be determined as integers. When filtering is performed using the scaled coefficients of the 1D filter, integer operations can be performed, but the magnitude of the filtered value may be higher than that of filtering using the unscaled coefficients of the 1D filter, and overflow may occur. Therefore, to prevent overflow, descaling can be performed after filtering is performed using the 1D filter. Here, descaling may include right bit shifting by the descaling bit number. The descaling bit number can be determined by considering the maximum number of bits in the register used for the filtering operation and the maximum number of bits in the temporary buffer storing the filtering result while maximizing the accuracy of the calculation. Specifically, the number of descaling bits can be determined based on the internal bit depth, the number of scaling bits of the interpolation filter, and the number of scaling bits used for the gradient filter.

[0441] The following describes a descaling process performed during the process of first using a vertical interpolation filter to filter pixels at integer positions to determine gradient values ​​in the horizontal direction to generate vertical interpolation filtered values, and then using a horizontal gradient filter to filter the vertical interpolation filtered values.

[0442] According to Equation 12 above, the video decoding device 100 can first perform filtering on the pixels at integer positions using an interpolation filter in the vertical direction to determine the gradient value in the horizontal direction. Here, shift1 can be b-8. Here, b can represent the internal bit depth of the input image. In the following, the bit depth of the register (Reg Bitdepth) and the bit depth of the temporary buffer (Temp Bitdepth) when descaling is actually performed based on shift1 will be described with reference to Table 1.

[0443] [Table 1]

[0444]

[0445] Here, the values ​​of the variables in Table 1 can be determined according to Equation 35.

[0446] [Equation 35]

[0447]

[0448] Here, Min(I) represents the minimum pixel value I determined by the internal bit depth, and Max(I) represents the maximum pixel value I determined by the internal bit depth. FilterSumPos represents the maximum sum of positive filter coefficients, and FilterSumNeg represents the minimum sum of negative filter coefficients.

[0449] For example, when using Figure 7c When using a gradient filter FracFilter with units of 1 / 4 pel, FilterSumPos can be 88 and FilterSumNeg can be -24.

[0450] The function Ceiling(x) can be a function of the smallest integer equal to or greater than x in terms of the output of the real number x. offset1 is an offset value added to the value being filtered to prevent rounding errors that may occur when descaling is performed using shift1, and offset1 can be determined as 2^(shift1-1).

[0451] Referring to Table 1, when the internal bit depth b is 8, the register bit depth can be 16; when the internal bit depth b is 9, the register bit depth can be 17; and when the internal bit depth b is 10, 11, 12, or 16, the register bit depth can be 18, 19, 20, or 24, respectively. When the register used for filtering is a 32-bit register, overflow will not occur because the bit depth of all registers in Table 1 does not exceed 32.

[0452] Similarly, when the internal bit depth b is 8, 9, 10, 11, 12, and 16, the bit depth (TempBitDepth) of the temporary buffer is always 16. When the temporary buffer used to store the value that has been filtered and then descaled is a 16-bit buffer, overflow will not occur because the bit depth of all temporary buffers in Table 1 is 16 and therefore does not exceed 16.

[0453] According to Equation 12, the video decoding device 100 can first generate a vertically interpolated filter value by performing filtering on pixels at integer positions using vertical interpolation filtering to determine gradient values ​​in the horizontal direction, and then perform filtering on the vertically interpolated filter value using a gradient filter in the horizontal direction according to Equation 13. Here, shift2 can be determined as p + q - shift1. Here, p can represent the values ​​for including... Figure 7c The interpolation filter coefficients shown in the figure are scaled by the number of bits, q can represent the number of bits for the interpolation filter including... Figure 7aThe gradient filter coefficients shown are scaled by the number of bits. For example, p can be 6, q can be 4, therefore, shift2 = 18 - b.

[0454] shift2 is determined in this way because shift1 + shift2 (i.e., the sum of the number of bits scaled) should be the same as sum(p + q) for the number of bits upscaled for the filter, so that the final filtered value is the same whether the filter coefficients are upscaled or not.

[0455] In the following text, the register bit depth and temporary buffer bit depth will be described with reference to Table 2 when descaling is actually performed based on shift2.

[0456] [Table 2]

[0457]

[0458] Here, the values ​​of the variables in Table 2 can be determined according to Equation 36.

[0459] [Equation 36]

[0460]

[0461] Here, TempMax can represent TempMax from Table 1, and TempMin can represent TempMin from Table 1. FilterSumPos represents the maximum sum of the positive filter coefficients, and FilterSumNeg represents the minimum sum of the negative filter coefficients. For example, when... Figure 7c When the gradient filter gradFilter shown in the figure is used in units of 1 / 4 pel, FilterSumPos can be 68 and FilterSumNeg can be -68.

[0462] offset2 is an offset value added to the value being filtered to prevent rounding errors that may occur when descaling is performed using shift2, and offset1 can be determined as 2^(shift2-1).

[0463] Shift1 and shift2 can be determined in this way, but alternatively, shift1 and shift2 can be determined differently, as long as the sum of shift1 and shift2 equals the sum of the scaling bits. Here, the values ​​of shift1 and shift2 can be determined based on the premise that no overflow occurs. Shift1 and shift2 can be determined based on the internal bit depth of the input image and the scaling bits with respect to the filter.

[0464] However, shift1 and shift2 need not be determined such that their sum equals the sum of the descaling bits of the filter. For example, shift1 could be determined to be d-8, but shift2 could be determined to be a fixed number.

[0465] When shift1 remains the same as before and shift2 is a fixed number of 7, the OutMax, OutMin, and Temp Bitdepth described in Table 2 can be changed. The Temp Bitdepth of the temporary buffer will now be described in Table 3 below.

[0466] [Table 3]

[0467]

[0468] Unlike Table 2, in Table 3, the temporary buffer's temp bit depth is the same (16) for all b values. When storing the result data using a 16-bit temporary buffer, the temp bit depth is less than 16, therefore no overflow occurs for the internal bit depth of any input image. However, referring to Table 2, when the internal bit depth of the input image is 12 or 16, and the result data is stored using a 16-bit temporary buffer, the temp bit depth is greater than 16, potentially leading to overflow.

[0469] When shift2 is a fixed number, the scaling filter coefficients are not used, and the result of performing filtering may differ from the result of performing filtering followed by descaling. In this case, it will be obvious to those skilled in the art that descaling needs to be performed separately.

[0470] The above has described a process of generating vertically interpolated filtered values ​​by first filtering pixels at integer positions using a vertical interpolation filter to determine gradient values ​​in the horizontal direction, and then filtering the vertically interpolated filtered values ​​using a horizontal gradient filter. However, it will be apparent to those skilled in the art that descaling can be performed in a similar manner when filtering pixels whose components in a particular direction are integers via a combination of various 1D filters to determine gradient values ​​in the horizontal and vertical directions.

[0471] Figure 9b This is a diagram used to describe the range required to determine the horizontal and vertical displacement vectors during the processing of pixel-unit motion compensation for the current block.

[0472] Reference Figure 9b When performing pixel-unit motion compensation on a reference block 910 corresponding to the current block, the video decoding device 100 can determine the displacement vector per unit time in the horizontal direction and the displacement vector per unit time in the vertical direction at pixel 915 by using a window 920 located near pixel 915 in the upper left of the reference block 910. Here, the displacement vector per unit time in the horizontal or vertical direction can be determined by using the pixel values ​​and gradient values ​​of pixels located outside the reference block 910. Similarly, when determining the horizontal and vertical displacement vectors of pixels located on the boundary of the reference block 910, the video decoding device 100 determines the pixel values ​​and gradient values ​​of pixels located outside the reference block 910. Therefore, the video decoding device 100 can determine the horizontal displacement vector and the displacement vector per unit time in the vertical direction by using a block 925 with a range larger than the reference block 910. For example, when the size of the current block is A×B and the size of the window for each pixel is (2M+1)x(2N+1), the size of the range used to determine the horizontal displacement vector and the vertical displacement vector can be (A+2M)x(B+2N).

[0473] Figure 9c and Figure 9d This is a diagram illustrating the range of regions used during motion compensation processing performed on a pixel-by-pixel basis, according to various embodiments.

[0474] Reference Figure 9cWhen performing motion compensation on a pixel-by-pixel basis, the video decoding device 100 can determine the horizontal displacement vector and the vertical displacement vector per unit time for each pixel included in the reference block 930 based on block 935, which is extended within the range of the window size of pixels located on the boundary of the reference block 930. However, when determining the horizontal displacement vector and the vertical displacement vector per unit time, the video decoding device 100 obtains the pixel value and gradient value of the pixel located in block 935, and at this time, an interpolation filter or gradient filter can be used to obtain the pixel value and gradient value. When using an interpolation filter or gradient filter on the boundary pixels of block 935, the pixel values ​​of neighboring pixels can be used, and therefore, pixels located outside the block boundary can be used. Therefore, pixel-by-pixel motion compensation can be performed by using block 940, which is further extended to the range of values ​​obtained by subtracting 1 from the number of taps of the interpolation filter or gradient filter. Therefore, when the block size is N×N, the window size of each pixel is (2M+1)x(2M+1), and the length of the interpolation filter or gradient filter is T, the block size in the expanded range can be (N+2M+T-1)x(N+2M+T-1).

[0475] Reference Figure 9d When performing motion compensation on a pixel-by-pixel basis, the video decoding device 100 can determine the horizontal displacement vector and the vertical displacement vector per unit time for each pixel by using the pixel values ​​and gradient values ​​of the pixels located in the reference block 945, without expanding the reference block according to the size of the window of pixels located on the boundary of the reference block 945. Specifically, referring to... Figure 9e The video decoding device 100 describes the process of determining the displacement vector per unit time in the horizontal direction and the displacement vector per unit time in the vertical direction without expanding the reference block. However, the interpolation filter or gradient filter of the reference block 945 is used to obtain the pixel value or gradient value of the pixel, and pixel group unit motion compensation can be performed by using the expanded block 950. Therefore, when the block size is N×N, the window size of each pixel is (2M+1)x(2M+1), and the length of the interpolation filter or gradient filter is T, the size of the expanded block can be (N+T-1)x(N+T-1).

[0476] Figure 9e This is a diagram illustrating the process of determining the horizontal and vertical displacement vectors without expanding the reference block.

[0477] Reference Figure 9eFor a pixel located outside the boundary of reference block 955, video decoding device 100 can adjust the position of that pixel to the position of the nearest available pixel among the pixels located on the boundary of reference block 955, so that the pixel value and gradient value of the pixel located outside the boundary of reference block 955 are determined as the pixel value and gradient value of the available pixel at the nearest position. Here, video decoding device 100 can determine the pixel value and gradient value of the pixel located outside the boundary of reference block 955 according to the equation Sum of equations The position of the pixel located outside the reference block 955 is adjusted to the position of the available pixel at the nearest position.

[0478] Here, i' represents the x-coordinate of the pixel, j' represents the y-coordinate of the pixel, and H and W represent the height and width of the reference block. We assume the top-left position of the reference block is (0,0). When the top-left position of the reference block is (xP, yP), the final pixel position can be (i'+xP, j'+yP).

[0479] Reference Figure 9c In block 935, which expands according to the size of the window for each pixel, the position of pixels located outside the boundary of reference block 930 is adjusted to the position of pixels adjacent to the boundary of reference block 930. The video decoding device 100 can then use methods such as... Figure 9d The pixel values ​​and gradient values ​​of the reference block 945 shown in the diagram determine the horizontal displacement vector of each pixel and the vertical displacement vector of each pixel per unit time in block 945.

[0480] Therefore, since the video decoding device 100 performs pixel-unit motion compensation without expanding the reference block 945 according to the window size of each pixel, the number of memory accesses for pixel value reference and the number of multiplication operations are reduced, thereby reducing computational complexity.

[0481] The video decoding device 100 performs memory access operations and multiplication operations according to the number of memory accesses and multiplication operations shown in Table 4 below, depending on whether it performs block-unit motion compensation (according to HEVC standard operation), performs pixel-unit motion compensation using block expansion based on the window size, or performs pixel-unit motion compensation without block expansion. Here, it is assumed that the gradient filter length T is 7, the block size is N×N, and the window size 2M+1 for each pixel is 5.

[0482] [Table 4]

[0483]

[0484] In block-unit motion compensation according to the HEVC standard, since an 8-tap interpolation filter is used for one sample, 8 neighboring samples are required. Therefore, when the size of the reference block is N×N, (N+7)x(N+7) reference samples are needed according to 8-tap interpolation. Furthermore, since bidirectional motion prediction compensation is performed, two reference blocks are used. Thus, as shown in Table 4, memory accesses are performed 2 x (N+7)x(N+7) times in block-unit motion compensation according to the HEVC standard. When pixel-unit motion compensation is performed using block expansion, M=2, and pixel-unit motion compensation is performed by using an 8-tap interpolation filter or gradient filter for blocks with an expanded size of (N+4)x(N+4), (N+4+7)x(N+4+7) reference samples are required. Since bidirectional motion prediction compensation is performed, two reference blocks are used, so as shown in Table 4, memory accesses are performed 2 x (N+4+7)x(N+4+7) times in pixel-unit motion compensation performed using block expansion.

[0485] However, when pixel-unit motion compensation is performed without block expansion, (N+7) x (N+7) reference samples are required in block-unit motion compensation according to the HEVC standard because the blocks are not expanded. Furthermore, since bidirectional motion prediction compensation is performed, two reference blocks are used, resulting in memory accesses being performed 2 x (N+7) x (N+7) times in pixel-unit motion compensation performed without block expansion, as shown in Table 4.

[0486] Figure 9f This is a diagram illustrating the process for obtaining candidates for time motion vector predictors, wherein pixel group unit motion compensation is considered in the process.

[0487] The video decoding device 100 can perform inter-frame prediction on the current block 965 in the current frame 960. Here, the video decoding device 100 can obtain the motion vector 980 of the co-position block 975 in the previously decoded frame 970 as a candidate temporal motion vector prediction factor for the current block 965, determine one of the obtained candidate temporal motion vector prediction factor for the current block and another candidate motion vector prediction factor as the motion vector prediction factor for the current block 965, and perform inter-frame prediction on the current block 965 by using the motion vector prediction factor.

[0488] The video decoding device 100 can perform block-unit motion compensation and pixel-group-unit motion compensation on the co-occurrence block 975 included in the previously decoded frame 970 while performing inter-frame prediction on the co-occurrence block 975. The video decoding device 100 can perform block-unit motion compensation by using motion vector 980, and can perform pixel-group-unit motion compensation by using displacement vectors per unit time in the horizontal direction and displacement vectors per unit time in the vertical direction for each pixel group.

[0489] Considering that the motion vector 980 of the corresponding block 975 can be used as a candidate for temporal motion vector predictor after the previously decoded frame 970, the video decoding device 100 can store the motion vector 980 of the corresponding block 975. Here, the video decoding device 100 can store the motion vector 980 based on a motion vector storage unit. Specifically, the video decoding device 100 can store the motion vector 980 according to the equation (MVx, MVy) = f RXR (MVx+μVx,MVy+μVy) is used to store the motion vector 980.

[0490] Here, MVx and MVy can represent the x and y components of the motion vector used in block-unit motion compensation, respectively, and Vx and Vy can represent the x and y components of the displacement vector of each pixel used in pixel-group-unit motion compensation, respectively. Furthermore, μ indicates a weight. Here, the weight μ is determined based on the size R of the motion vector storage unit, the size K of the pixel group, and the scaling factor of the gradient filter or interpolation filter used in pixel-group-based motion compensation. For example, the weight μ can decrease when the value of the pixel group size K increases, and the weight μ can decrease when the size R of the motion vector storage unit increases. Furthermore, the weight μ can decrease when the value of the scaling factor of the gradient filter or interpolation filter increases. Here, f RXR (MVx, MVy) can represent a function of motion vectors MVx and MVy, taking into account the size R×R of the motion vector storage unit. For example, f RXR (MVx, MVy) can be a function in which the average value of the x-components MVx of the motion vectors of the cells included in the R×R motion vector storage unit is determined as the x-component MVx stored in the R×R motion vector storage unit, and the average value of the y-components MVy of the motion vectors of the cells included in the R×R motion vector storage unit is determined as the y-component MVy stored in the R×R motion vector storage unit.

[0491] Since the stored motion vector 980 is a motion vector that takes into account motion compensation on a pixel-by-pixel basis, the temporal motion vector predictor candidate for the current block 965 can be determined as the motion vector used in more accurate motion compensation when performing inter-frame prediction on the current block 965, thus improving predictive coding / decoding efficiency.

[0492] In the following text, reference will be made to Figures 10 to 23 A method for determining data units that can be used when decoding an image using a video decoding apparatus 100 according to an embodiment is described. The operation of the video encoding apparatus 150 may be similar to or opposite to various embodiments of the operation of the video decoding apparatus 100 described below.

[0493] Figure 10 The illustration shows the process of determining at least one coding unit when the video decoding device 100 divides the current coding unit, according to an embodiment.

[0494] According to an embodiment, the video decoding device 100 can determine the shape of the coding unit by using block shape information, and can determine the shape into which the coding unit is divided by using partition shape information. In other words, the partitioning method of the coding unit indicated by the partition shape information can be determined based on the block shape indicated by the block shape information used by the video decoding device 100.

[0495] According to an embodiment, the video decoding device 100 can use block shape information indicating that the current coding unit has a square shape. For example, the video decoding device 100 can determine whether to not divide the square coding unit, whether to divide the square coding unit vertically, whether to divide the square coding unit horizontally, or whether to divide the square coding unit into four coding units based on the division shape information. (See also...) Figure 10 When the block shape information of the current encoding unit 1000 indicates a square shape, the video decoding device 100 may not divide the encoding unit 1010a, which has the same size as the current encoding unit 1000, according to the division shape information indicating that it should not be divided, or it may determine the encoding units 1010b, 1010c or 1010d based on the division shape information indicating a specific division method.

[0496] Reference Figure 10According to an embodiment, the video decoding device 100 can determine two coding units 1010b by dividing the current coding unit 1000 vertically based on division shape information indicating division in the vertical direction. The video decoding device 100 can determine two coding units 1010c by dividing the current coding unit 1000 horizontally based on division shape information indicating division in the horizontal direction. The video decoding device 100 can determine four coding units 1010d by dividing the current coding unit 1000 vertically and horizontally based on division shape information indicating division in both vertical and horizontal directions. However, the division shape into which a square coding unit can be divided is not limited to the shapes described above and can include any shape indicated by the division shape information. Specific division shapes into which a square coding unit can be divided will now be described in detail through various embodiments.

[0497] Figure 11 The illustration shows the process of determining at least one coding unit when the image decoding device 100 divides coding units having a non-square shape, according to an embodiment.

[0498] According to an embodiment, the video decoding device 100 can use block shape information indicating that the current coding unit has a non-square shape. The video decoding device 100 can determine whether to not divide the non-square current coding unit, or whether to divide the non-square current coding unit via a specific method, based on the division shape information. Referring to FIG11, when the block shape information of the current coding unit 1100 or 1150 indicates a non-square shape, the video decoding device 100 can, based on the division shape information indicating no division, not divide the coding unit 1110 or 1160 with the same size as the current coding unit 1100 or 1150, or can determine coding units 1120a, 1120b, 1130a, 1130b, 1130c, 1170a, 1170b, 1180a, 1180b, and 1180c based on the division shape information indicating a specific division method. Specific division methods for dividing non-square coding units will now be described in detail through various embodiments.

[0499] According to an embodiment, the video decoding device 100 can determine the shape into which the coding unit is divided by using division shape information. In this case, the division shape information can indicate the number of at least one coding unit generated when the coding unit is divided. (Refer to...) Figure 11When the partition shape information indicates that the current coding unit 1100 or 1150 is divided into two coding units, the video decoding device 100 can determine the two coding units 1120a and 1120b or 1170a and 1170b included in the current coding unit 1100 or 1150 by partitioning the current coding unit 1100 or 1150 based on the partition shape information.

[0500] According to an embodiment, when the video decoding device 100 divides a current coding unit 1100 or 1150 with a non-square shape based on division shape information, the video decoding device 100 may divide the current coding unit 1100 or 1150 considering the position of the longer side of the non-square shape. For example, the video decoding device 100 may determine multiple coding units by dividing the current coding unit 1100 or 1150 according to the direction of dividing the longer side of the current coding unit 1100 or 1150, taking into account the shape of the current coding unit 1100 or 1150.

[0501] According to an embodiment, when the partitioning shape information indicates that the coding unit is divided into an odd number of blocks, the video decoding device 100 can determine an odd number of coding units included in the current coding unit 1100 or 1150. For example, when the partitioning shape information indicates that the current coding unit 1100 or 1150 is divided into three coding units, the video decoding device 100 can divide the current coding unit 1100 or 1150 into three coding units 1130a to 1130c or 1180a to 1180c. According to an embodiment, the video decoding device 100 can determine an odd number of coding units included in the current coding unit 1100 or 1150, and the sizes of the determined coding units may not all be the same. For example, the size of coding unit 1130b or 1180b in the determined odd number of coding units 1130a to 1130c or 1180a to 1180c may be different from the sizes of coding units 1130a and 1130c or 1180a and 1180c. In other words, the coding units that can be determined when the current coding unit 1100 or 1150 is divided can have different types of sizes, and in some cases, coding units 1130a to 1130c or 1180a to 1180c can have different sizes.

[0502] According to an embodiment, when the partitioning shape information indicates that the coding unit is divided into an odd number of blocks, the video decoding device 100 can determine an odd number of coding units included in the current coding unit 1100 or 1150, and furthermore, can set specific limitations on at least one coding unit among the odd number of coding units generated by the partitioning. (Refer to...) Figure 11The video decoding device 100 may cause the decoding process performed on the middle coding unit 1130b or 1180b among the three coding units 1130a to 1130c or 1180a to 1180c generated when the current coding unit 1100 or 1150 is divided to be different from the decoding process performed on the other coding units 1130a and 1130c or 1180a and 1180c. For example, the video decoding device 100 may limit the middle coding unit 1130b or 1180b to be further divided like the other coding units 1130a and 1130c or 1180a and 1180c, or to be divided only a certain number of times.

[0503] Figure 12 The illustration shows a process by which a video decoding device 100 divides a coding unit based on at least one of block shape information and partition shape information, according to an embodiment.

[0504] According to an embodiment, the video decoding device 100 can determine whether a first coding unit 1200 having a square shape is divided into coding units based on at least one of block shape information and partition shape information. According to an embodiment, when the partition shape information indicates that the first coding unit 1200 is divided horizontally, the video decoding device 100 can determine a second coding unit 1210 by dividing the first coding unit 1200 horizontally. The terms "first coding unit," "second coding unit," and "third coding unit" used according to the embodiment are terms used to indicate the relationship between coding units before and after partitioning. For example, a second coding unit can be determined by partitioning a first coding unit, and a third coding unit can be determined by partitioning a second coding unit. In the following, it should be understood that the relationship between the first coding unit and the third coding unit conforms to the characteristics described above.

[0505] According to an embodiment, the video decoding device 100 can determine whether the determined second coding unit 1210 is divided into coding units based on at least one of block shape information and partition shape information. (Refer to...) Figure 12The video decoding device 100 may divide a second coding unit 1210, which has a non-square shape and is determined by dividing the first coding unit 1200, into at least one third coding unit 1220a, 1220b, 1220c, or 1220d based on at least one of block shape information and partition shape information, or it may not divide the second coding unit 1210. The video decoding device 100 may obtain at least one of block shape information and partition shape information, and may obtain a plurality of second coding units (e.g., second coding units 1210) with various shapes by dividing the first coding unit 1200 based on at least one of the obtained block shape information and partition shape information, wherein the second coding unit 1210 may be divided according to the method of dividing the first coding unit 1200 based on at least one of the block shape information and partition shape information. According to an embodiment, when the first coding unit 1200 is divided into a second coding unit 1210 based on at least one of block shape information and partition shape information for the first coding unit 1200, the second coding unit 1210 can also be divided into a third coding unit, for example, third coding units 1220a to 1220d, based on at least one of the block shape information and partition shape information for the second coding unit 1210. In other words, coding units can be recursively divided based on at least one of block shape information and partition shape information associated with each coding unit. Therefore, square coding units can be determined from non-square coding units, and such square coding units can be recursively divided such that non-square coding units are determined. (Refer to...) Figure 12 When the second coding unit 1210 with a non-square shape is divided, a specific coding unit (e.g., the middle coding unit or a square coding unit) among an odd number of third coding units 1220b to 1220d can be recursively divided. According to an embodiment, the third coding unit 1220c with a square shape among the third coding units 1220b to 1220d can be divided horizontally into a plurality of fourth coding units. The fourth coding unit 1240 with a non-square shape among the plurality of fourth coding units can be further divided into a plurality of coding units. For example, the fourth coding unit 1240 with a non-square shape can be divided into an odd number of coding units 1250a to 1250c.

[0506] The following describes, through various embodiments, methods that can be used to recursively divide coding units.

[0507] According to an embodiment, the video decoding device 100 can determine, based on at least one of block shape information and partition shape information, whether to partition each of the third coding units 1220a to 1220d into a coding unit or not to partition the second coding unit 1210. According to an embodiment, the video decoding device 100 can partition the second coding unit 1210, which has a non-square shape, into an odd number of third coding units 1220b to 1220d. The video decoding device 100 can set specific limitations on certain third coding units among the third coding units 1220b to 1220d. For example, the video decoding device 100 can limit the middle third coding unit 1220c among the third coding units 1220b to 1220d to no longer be partitioned or to be partitioned a set number of times. (See also...) Figure 12 The video decoding device 100 may be limited in that the middle third coding unit 1220c, which is included in the third coding units 1220b to 1220d of the second coding unit 1210 having a non-square shape, is no longer divided, but is divided into a specific division shape (e.g., divided into four coding units or divided into a shape corresponding to the shape into which the second coding unit 1210 is divided), or is divided only a specific number of times (e.g., divided only n times, where n > 0). However, such limitation on the middle third coding unit 1220c is merely an example and should not be construed as being limited to these examples, but rather as including various limitations, as long as the middle third coding unit 1220c is decoded differently from the other third coding units 1220b and 1220d.

[0508] According to an embodiment, the video decoding device 100 can obtain at least one of block shape information and partition shape information for dividing the current coding unit from a specific position in the current coding unit.

[0509] Figure 13 This illustrates a method by which a video decoding device 100 determines a specific coding unit from an odd number of coding units according to an embodiment. (See also...) Figure 13 At least one of block shape information and partition shape information of the current encoding unit 1300 can be obtained from a sample located at a specific position (e.g., sample 1340 located in the middle) among a plurality of samples included in the current encoding unit 1300. However, the specific position in the current encoding unit 1300 used to obtain at least one of the block shape information and partition shape information is not limited to... Figure 13The position shown is not the middle position, but can be any position included in the current coding unit 1300 (e.g., top position, bottom position, left position, right position, upper left position, lower left position, upper right position, or lower right position). The video decoding device 100 can determine whether the current coding unit is divided into coding units with various shapes and sizes or not by obtaining at least one of block shape information and partition shape information from a specific position.

[0510] According to an embodiment, when the current coding unit is divided into a specific number of coding units, the video decoding device 100 may select one coding unit. The method for selecting one of multiple coding units may vary, and details thereof will be described below through various embodiments.

[0511] According to an embodiment, the video decoding device 100 can divide the current encoding unit into multiple encoding units and determine the encoding unit at a specific location.

[0512] Figure 13 This illustrates a method by which a video decoding device 100 determines a coding unit at a specific location from an odd number of coding units, according to an embodiment.

[0513] According to an embodiment, the video decoding device 100 can use information indicating the position of each of an odd number of coding units to determine the middle coding unit among the odd number of coding units. (See also...) Figure 13 The video decoding device 100 can determine an odd number of coding units 1320a to 1320c by dividing the current coding unit 1300. The video decoding device 100 can determine the intermediate coding unit 1320b by using information about the positions of the odd number of coding units 1320a to 1320c. For example, the video decoding device 100 can determine the intermediate coding unit 1320b by determining the positions of coding units 1320a to 1320c based on information indicating the positions of specific samples included in coding units 1320a to 1320c. Specifically, the video decoding device 100 can determine the intermediate coding unit 1320b by determining the positions of coding units 1320a to 1320c based on information indicating the positions of samples 1330a to 1330c located to the upper left of coding units 1320a to 1320c.

[0514] According to an embodiment, the information indicating the position of the upper left samples 1330a to 1330c respectively included in the encoding units 1320a to 1320c may include information about the position or coordinates of the encoding units 1320a to 1320c in the frame. According to an embodiment, the information indicating the position of the upper left samples 1330a to 1330c respectively included in the encoding units 1320a to 1320c may include information indicating the width or height of the encoding units 1320a to 1320c included in the current encoding unit 1300, and such width or height may correspond to information indicating the difference between the coordinates of the encoding units 1320a to 1320c in the frame. In other words, the video decoding device 100 can determine the middle encoding unit 1320b by directly using information about the position or coordinates of the encoding units 1320a to 1320c in the frame or by using information about the width or height of the encoding units 1320a to 1320c corresponding to the difference between their coordinates.

[0515] According to an embodiment, information indicating the position of the upper left sample 1330a of the upper encoding unit 1320a can indicate coordinates (xa, ya), information indicating the position of the upper left sample 1330b of the middle encoding unit 1320b can indicate coordinates (xb, yb), and information indicating the position of the upper left sample 1330c of the lower encoding unit 1320c can indicate coordinates (xc, yc). The video decoding device 100 can determine the middle encoding unit 1320b by using the coordinates of the upper left samples 1330a to 1330c respectively included in the encoding units 1320a to 1320c. For example, when the coordinates of the upper left samples 1330a to 1330c are arranged in ascending or descending order, the encoding unit 1320b located in the middle, including the coordinates (xb, yb) of sample 1330b, can be determined as the middle encoding unit among the encoding units 1320a to 1320c determined when the current encoding unit 1300 is divided. However, the coordinates indicating the positions of the upper left samples 1330a to 1330c can be coordinates indicating absolute positions within the image. Alternatively, (dxb, dyb) coordinates (i.e., information indicating the relative position of the upper left sample 1330b of the middle encoding unit 1320b) and (dxc, dyc) coordinates (i.e., information indicating the relative position of the upper left sample 1330c of the lower encoding unit 1320c) can be used based on the position of the upper left sample 1330a of the upper encoding unit 1320a. Furthermore, the method of determining the encoding unit at a specific position by using the coordinates of the samples included in the encoding unit as information indicating the position of the samples is not limited to the above method, and various arithmetic methods capable of using the coordinates of the samples can be used.

[0516] According to an embodiment, the video decoding device 100 can divide the current encoding unit 1300 into a plurality of encoding units 1320a to 1320c, and select an encoding unit from the encoding units 1320a to 1320c according to a specific criterion. For example, the video decoding device 100 can select an encoding unit 1320b with a different size from the encoding units 1320a to 1320c.

[0517] According to an embodiment, the video decoding device 100 can determine the width or height of the encoding units 1320a to 1320c by using (xa, ya) coordinates (i.e., information indicating the position of the upper left sample 1330a of the upper encoding unit 1320a), (xb, yb) coordinates (i.e., information indicating the position of the upper left sample 1330b of the middle encoding unit 1320b), and (xc, yc) coordinates (i.e., information indicating the position of the upper left sample 1330c of the lower encoding unit 1320c), respectively. The video decoding device 100 can also determine the dimensions of the encoding units 1320a to 1320c by using the coordinates (xa, ya), (xb, yb), and (xc, yc) indicating the positions of the encoding units 1320a to 1320c, respectively.

[0518] According to an embodiment, the video decoding device 100 can determine the width of the upper encoding unit 1320a as xb-xa and the height of the upper encoding unit 1320a as yb-ya. According to an embodiment, the video decoding device 100 can determine the width of the middle encoding unit 1320b as xc-xb and the height of the middle encoding unit 1320b as yc-yb. According to an embodiment, the video decoding device 100 can determine the width or height of the lower encoding unit 1320c by using the width and height of the current encoding unit 1300, and the width and height of the upper encoding unit 1320a and the middle encoding unit 1320b. The video decoding device 100 can determine encoding units with different dimensions from other encoding units based on the determined width and height of the encoding units 1320a to 1320c. (Refer to...) Figure 13 The video decoding device 100 can identify an intermediate encoding unit 1320b, which has a different size than the upper encoding unit 1320a and the lower encoding unit 1320c, as an encoding unit at a specific location. However, the process by which the video decoding device 100 identifies an encoding unit with a different size than the other encoding units is merely an example of identifying an encoding unit at a specific location by using the size of the encoding unit determined based on sample coordinates. Therefore, various processes can be used to identify an encoding unit at a specific location by comparing the sizes of encoding units determined according to specific sample coordinates.

[0519] However, the position of the sample point considered to determine the position of the coding unit is not limited to the upper left as described above, and information about the position of any sample point included in the coding unit can be used.

[0520] According to an embodiment, while considering the shape of the current coding unit, the video decoding device 100 can select a coding unit at a specific position from an odd number of coding units determined when the current coding unit was divided. For example, when the current coding unit has a non-square shape with a width greater than its height, the video decoding device 100 can determine a coding unit at a specific position in the horizontal direction. In other words, the video decoding device 100 can determine one coding unit among the coding units that has a different position in the horizontal direction and set a limitation on that coding unit. When the current coding unit has a non-square shape with a height greater than its width, the video decoding device 100 can determine a coding unit at a specific position in the vertical direction. In other words, the video decoding device 100 can determine one coding unit among the coding units that has a different position in the vertical direction and set a limitation on that coding unit.

[0521] According to an embodiment, the video decoding device 100 can use information indicating the position of each of an even number of coding units to determine the coding unit at a specific position from the even number of coding units. The video decoding device 100 can determine the even number of coding units by dividing the current coding unit, and determine the coding unit at the specific position by using information about the positions of the even number of coding units. Detailed processing can be described in... Figure 13 The process of determining the coding unit at a specific position (e.g., the middle position) from an odd number of coding units is described accordingly, and therefore its details are not provided further.

[0522] According to an embodiment, when a current coding unit having a non-square shape is divided into multiple coding units, specific information about coding units at a particular location during the partitioning process can be used to determine the coding unit at that particular location among the multiple coding units. For example, the video decoding device 100 can use at least one of block shape information and partition shape information stored in samples (which are included in the intermediate coding unit) during the partitioning process to determine the intermediate coding unit from the multiple coding units obtained by partitioning the current coding unit.

[0523] Reference Figure 13The video decoding device 100 can divide the current coding unit 1300 into a plurality of coding units 1320a to 1320c based on at least one of block shape information and partition shape information, and determine the coding unit 1320b located in the middle from the plurality of coding units 1320a to 1320c. Furthermore, the video decoding device 100 can determine the coding unit 1320b located in the middle by considering the position used to obtain at least one of the block shape information and partition shape information. In other words, at least one of the block shape information and partition shape information of the current coding unit 1300 can be obtained from a sample point 1340 located in the middle of the current coding unit 1300, and when the current coding unit 1300 is divided into a plurality of coding units 1320a to 1320c based on at least one of the block shape information and partition shape information, the coding unit 1320b including the sample point 1340 can be determined as the coding unit located in the middle. However, the information used to determine the intermediate coding unit is not limited to at least one of block shape information and partition shape information, and various types of information can be used when determining the intermediate coding unit.

[0524] According to an embodiment, specific information for identifying a coding unit at a specific location can be obtained from specific samples included in the coding unit to be identified. (Refer to...) Figure 13 The video decoding device 100 can use at least one of block shape information and partition shape information obtained from a sample point located at a specific position in the current coding unit 1300 (e.g., a sample point located in the middle of the current coding unit 1300) to determine a coding unit at a specific position (e.g., the middle coding unit among the multiple coding units) from a plurality of coding units 1320a to 1320c determined when the current coding unit 1300 is partitioned. In other words, the video decoding device 100 can determine a sample point at a specific position considering the block shape of the current coding unit 1300, and can determine and specifically define coding unit 1320b from a plurality of coding units 1320a to 1320c determined when the current coding unit 1300 is partitioned, wherein coding unit 1320b includes a sample point from which specific information (e.g., at least one of block shape information and partition shape information) can be obtained. (Refer to...) Figure 13 According to an embodiment, the video decoding device 100 may determine a sample 1340 located in the middle of the current encoding unit 1300 as a sample from which specific information can be obtained, and may set specific limitations on the encoding unit 1320b including such a sample 1340 during the decoding process. However, the position of the sample from which specific information can be obtained is not limited to the above position, and the sample may be a sample included at any position in the encoding unit 1320b that is determined to be limited.

[0525] According to an embodiment, the location of a sample from which specific information can be obtained can be determined based on the shape of the current coding unit 1300. According to an embodiment, block shape information can determine whether the shape of the current coding unit is square or non-square, and can determine the location of a sample from which specific information can be obtained based on that shape. For example, the video decoding device 100 can determine a sample located on a boundary that divides at least one of the width and height of the current coding unit into two equal parts using at least one of information about the width of the current coding unit and information about the height of the current coding unit as a sample from which specific information can be obtained. As another example, when the block shape information associated with the current coding unit indicates a non-square shape, the video decoding device 100 can determine one of the samples adjacent to the boundary that divides the long side of the current coding unit into two equal parts as a sample from which predetermined information can be obtained.

[0526] According to an embodiment, when a current coding unit is divided into multiple coding units, the video decoding device 100 can use at least one of block shape information and partition shape information to determine the coding unit at a specific location from the multiple coding units. According to an embodiment, the video decoding device 100 can obtain at least one of block shape information and partition shape information from samples included in the coding unit at a specific location, and can divide the multiple coding units by using at least one of the block shape information and partition shape information obtained from samples included in each of the multiple coding units generated when the current coding unit is divided. In other words, the coding unit can be recursively divided by using at least one of the block shape information and partition shape information obtained from samples included in each coding unit at a specific location. Since reference has been made... Figure 12 The process of recursively dividing the coding unit is described, so its details are not provided here.

[0527] According to an embodiment, the video decoding device 100 can determine at least one coding unit by dividing the current coding unit, and determine the order in which the at least one coding unit is decoded according to a specific block (e.g., the current coding unit).

[0528] Figure 14 This illustrates the order in which multiple coding units are processed when multiple coding units are determined during the division of the current coding unit in the video decoding device 100, according to an embodiment.

[0529] According to an embodiment, the video decoding device 100 can determine the second coding units 1410a and 1410b by dividing the first coding unit 1400 in the vertical direction, the second coding units 1430a and 1430b by dividing the first coding unit 1400 in the horizontal direction, or the second coding units 1450a to 1450d by dividing the first coding unit 1400 in both the horizontal and vertical directions, based on block shape information and division shape information.

[0530] Reference Figure 14 The video decoding device 100 can determine second coding units 1410a and 1410b, which are determined by dividing the first coding unit 1400 vertically, to be processed in a horizontal direction 1410c. The video decoding device 100 can determine second coding units 1430a and 1430b, which are determined by dividing the first coding unit 1400 horizontally, to be processed in a vertical direction 1430c. The video decoding device 100 can determine second coding units 1450a to 1450d, which are determined by dividing the first coding unit 1400 vertically and horizontally, to be processed according to a specific order (e.g., raster scan order or z-scan order 1450e), wherein coding units in one row are processed, followed by coding units in the next row.

[0531] According to an embodiment, the video decoding device 100 can recursively divide the encoding units. (Refer to...) Figure 14 The video decoding device 100 can determine a plurality of second coding units 1410a and 1410b, 1430a and 1430b, or 1450a to 1450d by dividing a first coding unit 1400, and recursively divide each of the plurality of second coding units 1410a and 1410b, 1430a and 1430b, or 1450a to 1450d. The method of dividing the plurality of second coding units 1410a and 1410b, 1430a and 1430b, or 1450a to 1450d can correspond to the method of dividing the first coding unit 1400. Therefore, each of the plurality of coding units 1410a and 1410b, 1430a and 1430b, or 1450a to 1450d can be independently divided into a plurality of coding units. (Refer to...) Figure 14 The video decoding device 100 can determine the second coding units 1410a and 1410b by dividing the first coding unit 1400 in a vertical direction. In addition, it can determine whether to divide each of the second coding units 1410a and 1410b independently.

[0532] According to an embodiment, the video decoding device 100 may divide the second encoding unit 1410a on the left side into third encoding units 1420a and 1420b in the horizontal direction, and may not divide the second encoding unit 1410b on the right side.

[0533] According to an embodiment, the order in which the coding units are processed can be determined based on the division of the coding units. In other words, the order in which the divided coding units are processed can be determined based on the order in which the coding units were processed before being divided. The video decoding device 100 can determine the order in which the third coding units 1420a and 1420b, which are determined when the second coding unit 1410a on the left is divided independently of the second coding unit 1410b on the right, are processed. Since the third coding units 1420a and 1420b are determined when the second coding unit 1410a on the left is divided in a horizontal direction, the third coding units 1420a and 1420b can be processed in a vertical direction 1420c. In addition, since the order in which the second coding unit 1410a on the left and the second coding unit 1410b on the right are processed corresponds to the horizontal direction 1410c, the second coding unit 1410b on the right can be processed after the third coding units 1420a and 1420b, which are included in the second coding unit 1410a on the left, are processed in a vertical direction 1420c. The above description relates to a process that determines the order of processing coding units based on the coding units before they are divided. However, such a process is not limited to the above embodiments, and any method that processes coding units divided into various shapes independently in a specific order can be used.

[0534] Figure 15 The illustration shows the process of determining that the current coding unit is divided into an odd number of coding units when the coding units cannot be processed by the video decoding device 100 in a specific order, according to an embodiment.

[0535] According to an embodiment, the video decoding device 100 can determine, based on the obtained block shape information and partition shape information, that the current coding unit is divided into an odd number of coding units. (Refer to...) Figure 15 A first coding unit 1500 having a square shape can be divided into second coding units 1510a and 1510b having non-square shapes. The second coding units 1510a and 1510b can be independently divided into third coding units 1520a and 1520b, and further divided into 1520c and 1520e, respectively. According to an embodiment, the video decoding device 100 can divide the second coding units 1510a on the left side of the second coding units 1510a and 1510b in a horizontal direction to determine a plurality of third coding units 1520a and 1520b, and divide the second coding units 1510b on the right side into an odd number of third coding units 1520c to 1520e.

[0536] According to an embodiment, the video decoding device 100 can determine whether an odd number of coding units exist by determining whether the third coding units 1520a to 1520e can be processed in a specific order. (See also...) Figure 15 The video decoding device 100 can determine the third coding units 1520a to 1520e by recursively dividing the first coding unit 1500. The video decoding device 100 can determine, based on at least one of block shape information and division shape information, whether the coding units are divided into an odd number of shapes within the shapes into which the first coding unit 1500, second coding units 1510a and 1510b, or third coding units 1520a to 1520e will be divided. For example, the rightmost coding unit 1510b of the second coding units 1510a and 1510b can be divided into an odd number of third coding units 1520c to 1520e. The order in which the multiple coding units included in the first coding unit 1500 are processed can be a specific order (e.g., z-scan order 1530), and the video decoding device 100 can determine whether the third coding units 1520c to 1520e determined when the rightmost second coding unit 1510b is divided into an odd number satisfy the condition that they can be processed according to the specific order.

[0537] According to an embodiment, the video decoding device 100 can determine whether third encoding units 1520a to 1520e, included in the first encoding unit 1500, satisfy a condition that allows them to be processed in a specific order. This condition is related to whether at least one of the width and height of each of the second encoding units 1510a and 1510b is divided into two equal parts according to the boundaries of the third encoding units 1520a to 1520e. For example, third encoding units 1520a and 1520b, determined when the height of the left-hand second encoding unit 1510a, which has a non-square shape, is divided into two equal parts, satisfy the condition. However, since the boundaries of the third encoding units 1520c to 1520e, determined when the right-hand second encoding unit 1510b is divided into three encoding units, do not divide the width or height of the right-hand second encoding unit 1510b into two equal parts, the third encoding units 1520c to 1520e do not satisfy the condition. When the aforementioned conditions are not met, the video decoding device 100 can determine that the scanning sequence is broken, and based on the determination result, determine that the second coding unit 1510b on the right is divided into an odd number of coding units. According to an embodiment, the video decoding device 100 can set specific limitations on coding units at specific positions among the odd number of coding units obtained by dividing the coding units. Since such limitations or specific positions have been described above through various embodiments, their details will not be provided again.

[0538] Figure 16This illustration shows the process of determining at least one coding unit when the video decoding device 100 divides the first coding unit 1600 according to an embodiment. According to the embodiment, the video decoding device 100 may divide the first coding unit 1600 based on at least one of block shape information and division shape information obtained by the acquirer 105. The first coding unit 1600 having a square shape may be divided into four coding units having square shapes or multiple coding units having non-square shapes. For example, refer to… Figure 16 When the block shape information indicates that the first coding unit 1600 is square and the division shape information indicates that it is divided into non-square coding units, the video decoding device 100 can divide the first coding unit 1600 into multiple non-square coding units. Specifically, when the division shape information indicates that an odd number of coding units are determined by dividing the first coding unit 1600 according to a horizontal or vertical direction, the video decoding device 100 can determine the second coding units 1610a to 1610c by dividing the first coding unit 1600, which has a square shape, according to a vertical direction, or by dividing the first coding unit 1600 according to a horizontal direction, as an odd number of coding units.

[0539] According to an embodiment, the video decoding device 100 can determine whether the second encoding units 1610a to 1610c and 1620a to 1620c included in the first encoding unit 1600 satisfy a condition that they can be processed in a specific order, wherein the condition is related to whether at least one of the width and height of the first encoding unit 1600 is divided into two equal parts according to the boundaries of the second encoding units 1610a to 1610c and 1620a to 1620c. (Refer to...) Figure 16Since the boundaries of the second encoding units 1610a to 1610c, determined when the first encoding unit 1600, having a square shape, is divided vertically, do not divide the width of the first encoding unit 1600 into two equal parts, it can be determined that the first encoding unit 1600 does not meet the condition that it can be processed in a specific order. Furthermore, since the boundaries of the second encoding units 1620a to 1620c, determined when the first encoding unit 1600, having a square shape, is divided horizontally, do not divide the height of the first encoding unit 1600 into two equal parts, it can be determined that the first encoding unit 1600 does not meet the condition that it can be processed in a specific order. The video decoding device 100 can determine a break in the scanning order when the conditions are not met, and can determine, based on the determination result, that the first encoding unit 1600 is divided into an odd number of encoding units. According to an embodiment, the video decoding device 100 can set specific limitations on encoding units at specific positions among the odd number of encoding units obtained by dividing the encoding units; since such limitations or specific positions have been described above through various embodiments, their details will not be provided further.

[0540] According to an embodiment, the video decoding device 100 can determine coding units with various shapes by dividing a first coding unit.

[0541] Reference Figure 16 The video decoding device 100 can divide the first encoding unit 1600 with a square shape and the first encoding unit 1630 or 1650 with a non-square shape into encoding units with various shapes.

[0542] Figure 17 The embodiment shows that when a second coding unit with a non-square shape, determined when the first coding unit 1700 is divided, satisfies certain conditions, the shape into which the second coding unit can be divided by the video decoding device 100 is defined.

[0543] According to an embodiment, the video decoding device 100 can determine, based on at least one of block shape information and partition shape information obtained by the acquirer 105, that a first coding unit 1700 having a square shape is divided into second coding units 1710a and 1710b or 1720a and 1720b having a non-square shape. The second coding units 1710a and 1710b or 1720a and 1720b can be divided independently. Therefore, the video decoding device 100 can determine, based on at least one of the block shape information and partition shape information associated with each of the second coding units 1710a and 1710b or 1720a and 1720b, that the second coding units 1710a and 1710b or 1720a and 1720b are divided into multiple coding units or are not divided at all. According to an embodiment, the video decoding device 100 can determine the third coding units 1712a and 1712b by dividing the second coding unit 1710a on the left side, which has a non-square shape and is determined when the first coding unit 1700 is divided in a vertical direction, in a horizontal direction. However, when the second coding unit 1710a on the left side is divided in a horizontal direction, the video decoding device 100 may set the following limitation: the second coding unit 1710b on the right side is not divided in a horizontal direction like the second coding unit 1710a on the left side. When the third coding units 1714a and 1714b are determined when the second coding unit 1710b on the right side is divided in the same direction (i.e., horizontal direction), the third coding units 1712a, 1712b, 1714a, and 1714b may be determined when the second coding unit 1710a on the left side and the second coding unit 1710b on the right side are divided independently in a horizontal direction. However, this is the same result as dividing the first coding unit 1700 into four second coding units 1730a to 1730d with square shapes based on at least one of block shape information and partition shape information, and therefore may be inefficient in terms of image decoding.

[0544] According to an embodiment, the video decoding device 100 can determine the third encoding units 1722a and 1722b or 1724a and 1724b by dividing the second encoding units 1720a or 1720b, which have a non-square shape and are determined when the first encoding unit 1700 is divided in a horizontal direction, in a vertical direction. However, when one of the second encoding units (e.g., the top second encoding unit 1720a) is divided in a vertical direction, for the reasons mentioned above, the video decoding device 100 may set the following limitation: other second encoding units (e.g., the bottom second encoding unit 1720b) ​​are not divided in a vertical direction like the top second encoding unit 1720a.

[0545] Figure 18The illustration shows the process by which a video decoding device 100 divides encoding units with square shapes when the division shape information fails to indicate that the encoding unit is divided into four square shapes, according to an embodiment.

[0546] According to an embodiment, the video decoding device 100 can determine second coding units 1810a and 1810b or 1820a and 1820b by dividing the first coding unit 1800 based on at least one of block shape information and partition shape information. The partition shape information may include information about various shapes into which the coding unit can be divided, but such information about various shapes may not include information for dividing the coding unit into four square coding units. Based on such partition shape information, the video decoding device 100 cannot divide the first coding unit 1800, which has a square shape, into four second coding units 1830a to 1830d, which also have square shapes. Based on the partition shape information, the video decoding device 100 can determine second coding units 1810a and 1810b or 1820a and 1820b that have non-square shapes.

[0547] According to an embodiment, the video decoding device 100 can independently divide each of the second coding units 1810a and 1810b or 1820a and 1820b having a non-square shape. Each of the second coding units 1810a and 1810b or 1820a and 1820b can be divided in a specific order using a recursive method, wherein the recursive method can be a division method corresponding to a method for dividing the first coding unit 1800 based on at least one of block shape information and partition shape information.

[0548] For example, the video decoding device 100 can determine the third coding units 1812a and 1812b with square shapes by dividing the second coding unit 1810a on the left side in a horizontal direction, or by dividing the second coding unit 1810b on the right side in a horizontal direction. Furthermore, the video decoding device 100 can determine the third coding units 1816a to 1816d with square shapes by dividing both the second coding unit 1810a on the left side and the second coding unit 1810b on the right side in a horizontal direction. In this case, the coding units can be determined in the same manner as when the first coding unit 1810 is divided into four second coding units 1830a to 1830d with square shapes.

[0549] As another example, the video decoding device 100 can determine third coding units 1822a and 1822b with square shapes by dividing the top second coding unit 1820a vertically, and third coding units 1824a and 1824b with square shapes by dividing the bottom second coding unit 1820b vertically. Furthermore, the video decoding device 100 can determine third coding units 1826a to 1826d with square shapes by dividing both the top second coding unit 1820a and the bottom second coding unit 1820b vertically. In this case, the coding units can be determined in the same manner as when the first coding unit 1800 is divided into four second coding units 1830a to 1830d with square shapes.

[0550] Figure 19 The order in which multiple coding units are processed according to an embodiment can be changed depending on the process of dividing the coding units.

[0551] According to an embodiment, the video decoding device 100 can divide the first encoding unit 1900 based on block shape information and partition shape information. When the block shape information indicates a square shape and the partition shape information indicates that the first encoding unit 1900 is divided according to at least one of the horizontal and vertical directions, the video decoding device 100 can divide the first encoding unit 1900 to determine second encoding units 1910a and 1910b or 1920a and 1920b. (Refer to...) Figure 19 The second coding units 1910a and 1910b, or 1920a and 1920b, which have non-square shapes and are determined when the first coding unit 1900 is divided in a horizontal or vertical direction, can be independently divided based on block shape information and division shape information. For example, the video decoding device 100 can determine the third coding units 1916a to 1916d by dividing each of the second coding units 1910a and 1910b in a horizontal direction, wherein the second coding units 1910a and 1910b are generated when the first coding unit 1900 is divided in a vertical direction, or the video decoding device 100 can determine the third coding units 1926a to 1926d by dividing the second coding units 1920a and 1920b in a vertical direction, wherein the second coding units 1920a and 1920b are generated when the first coding unit 1900 is divided in a horizontal direction. The above has been referred to... Figure 17 The process of dividing the second coding units 1910a and 1910b or 1920a and 1920b is described, and therefore its details are not provided.

[0552] According to an embodiment, the video decoding device 100 can process the encoding units in a specific order. (The above has been referred to...) Figure 14 The characteristics of processing coding units according to a specific order are described, therefore their details are not provided further. (See reference...) Figure 19 The video decoding device 100 can determine four third coding units 1916a to 1916d or 1926a to 1926d, each having a square shape, by dividing a first coding unit 1900 having a square shape. According to an embodiment, the video decoding device 100 can determine the order in which the third coding units 1916a to 1916d or 1926a to 1926d are processed based on how the first coding unit 1900 is divided.

[0553] According to an embodiment, the video decoding device 100 can determine the third coding units 1916a to 1916d by dividing the second coding units 1910a to 1910b in a horizontal direction, wherein the second coding units 1910a and 1910b are generated when the first coding unit 1900 is divided in a vertical direction, and the video decoding device 100 can process the third coding units 1916a to 1916d according to a sequence 1917 of first processing the third coding units 1916a and 1916b included in the second coding unit 1910a on the left side in a vertical direction and then processing the third coding units 1916c and 1916d included in the second coding unit 1910b on the right side in a vertical direction.

[0554] According to an embodiment, the video decoding device 100 can determine the third coding units 1926a to 1926d by dividing the second coding units 1920a to 1920b in a vertical direction, wherein the second coding units 1920a and 1920b are generated when the first coding unit 1900 is divided in a horizontal direction, and the video decoding device 100 can process the third coding units 1926a to 1926d according to a sequence 1927 of first processing the third coding units 1926a and 1926b included in the top second coding unit 1920a in a horizontal direction and then processing the third coding units 1926c and 1926d included in the bottom second coding unit 1920b in a horizontal direction.

[0555] Reference Figure 19The third coding units 1916a to 1916d or 1926a to 1926d, which have square shapes, can be determined when both second coding units 1910a and 1910b or 1920a and 1920b are divided. The second coding units 1910a and 1910b, determined when the first coding unit 1900 is divided vertically, and the second coding units 1920a and 1920b, determined when the first coding unit 1900 is divided horizontally, are divided into different shapes. However, based on the subsequently determined third coding units 1916a to 1916d and 1926a to 1926d, the first coding unit 1900 is divided into coding units with the same shape. Therefore, even when coding units with the same shape are determined by recursively dividing the coding units based on at least one of block shape information and division shape information through different processes, the video decoding device 100 can process multiple coding units determined to have the same shape in different orders.

[0556] Figure 20 The illustration shows the process of determining the depth of a coding unit when the shape and size of the coding unit are changed, as multiple coding units are determined during recursive division according to an embodiment.

[0557] According to an embodiment, the video decoding device 100 can determine the depth of a coding unit based on a specific criterion. For example, the specific criterion could be the length of the long side of the coding unit. When the length of the long side of the current coding unit is divided into segments that are 2n times shorter than the length of the long side of the coding unit before the division, it can be determined that the depth of the current coding unit is increased to n times the depth of the coding unit before the division, where n > 0. Hereinafter, the coding unit with the increased depth will be referred to as the coding unit of the lower depth.

[0558] Reference Figure 20According to an embodiment, the video decoding device 100 can determine a second encoding unit 2002 and a third encoding unit 2004 of lower layer depth by dividing a first encoding unit 2000 having a square shape based on block shape information indicating a square shape (e.g., the block shape information may indicate "0: SQUARE"). When the size of the first encoding unit 2000 having a square shape is 2N×2N, the second encoding unit 2002, determined by dividing the width and height of the first encoding unit 2000 by 1 / 2^1, may have a size of N×N. Furthermore, the third encoding unit 2004, determined by dividing the width and height of the second encoding unit 2002 by 1 / 2, may have a size of N / 2×N / 2. In this case, the width and height of the third encoding unit 2004 correspond to 1 / 2^2 times the width and height of the first encoding unit 2000. When the depth of the first coding unit 2000 is D, the depth of the second coding unit 2002 can be D+1, wherein the width and height of the second coding unit 2002 are 1 / 2^1 times the width and height of the first coding unit 2000, and the depth of the third coding unit 2004 can be D+2, wherein the width and height of the third coding unit 2004 are 1 / 2^2 times the width and height of the first coding unit 2000.

[0559] According to an embodiment, the video decoding device 100 can determine the second encoding unit 2012 or 2022 and the third encoding unit 2014 or 2024 by dividing the first encoding unit 2010 or 2020 with a non-square shape based on block shape information indicating a non-square shape (e.g., the block shape information may indicate "1: NS_VER" indicating a non-square shape with a height greater than its width, or "2: NS_HOR" indicating a non-square shape with a width greater than its height).

[0560] The video decoding device 100 can determine a second encoding unit (e.g., a second encoding unit 2002, 2012, or 2022) by dividing at least one of the height and width of a first encoding unit 2010 having a size of N×2N. In other words, the video decoding device 100 can determine a second encoding unit 2002 having a size of N×N or a second encoding unit 2022 having a size of N×N / 2 by dividing the first encoding unit 2010 in a horizontal direction, or by dividing the first encoding unit 2010 in both horizontal and vertical directions.

[0561] The video decoding device 100 can determine a second encoding unit (e.g., a second encoding unit 2002, 2012, or 2022) by dividing at least one of the width and height of a first encoding unit 2020 having a size of 2N×N. In other words, the video decoding device 100 can determine a second encoding unit 2002 having a size of N×N or a second encoding unit 2012 having a size of N / 2×N by dividing the first encoding unit 2020 in a vertical direction, or by dividing the first encoding unit 2010 in both horizontal and vertical directions.

[0562] According to an embodiment, the video decoding device 100 can determine a third encoding unit (e.g., a third encoding unit 2004, 2014, or 2024) by dividing the width and height of a second encoding unit 2002 having a size of N×N. In other words, the video decoding device 100 can determine a third encoding unit 2004 having a size of N / 2×N / 2, a third encoding unit 2014 having a size of N / 4×N / 2, or a third encoding unit 2024 having a size of N / 2×N / 4 by dividing the second encoding unit 2002 in both horizontal and vertical directions.

[0563] According to an embodiment, the video decoding device 100 can determine a third encoding unit (e.g., a third encoding unit 2004, 2014, or 2024) by dividing at least one of the width and height of a second encoding unit 2022 having a size of N / 2 × N. In other words, the video decoding device 100 can determine a third encoding unit 2004 having a size of N / 2 × N / 2 or a third encoding unit 2024 having a size of N / 2 × N / 4 by dividing the second encoding unit 2022 in a horizontal direction, or determine a third encoding unit 2014 having a size of N / 4 × N / 2 by dividing the second encoding unit 2012 in both vertical and horizontal directions.

[0564] According to an embodiment, the video decoding device 100 can determine a third encoding unit (e.g., a third encoding unit 2004, 2014, or 2024) by dividing at least one of the width and height of a second encoding unit 2022 having a size of N×N / 2. In other words, the video decoding device 100 can determine a third encoding unit 2004 having a size of N / 2×N / 2 or a third encoding unit 2014 having a size of N / 4×N / 2 by dividing the second encoding unit 2022 in a vertical direction, or determine a third encoding unit 2024 having a size of N / 2×N / 4 by dividing the second encoding unit 2022 in both vertical and horizontal directions.

[0565] According to an embodiment, the video decoding device 100 can divide square-shaped coding units (e.g., first coding unit 2000, second coding unit 2002, or third coding unit 2004) in a horizontal or vertical direction. For example, a first coding unit 2010 with an N×2N size can be determined by dividing the first coding unit 2000 with a 2N×2N size vertically, or a first coding unit 2020 with a 2N×N size can be determined by dividing the first coding unit 2000 with a 2N×2N size horizontally. According to an embodiment, when the depth is determined based on the length of the longest side of the coding unit, the depth of the coding unit determined when the first coding unit 2000 with a 2N×2N size is divided in a horizontal or vertical direction can be the same as the depth of the first coding unit 2000.

[0566] According to an embodiment, the width and height of the third encoding unit 2014 or 2024 can be 1 / 2^2 of the width and height of the first encoding unit 2010 or 2020. When the depth of the first encoding unit 2010 or 2020 is D, the depth of the second encoding unit 2012 or 2022 can be D+1, wherein the width and height of the second encoding unit 2012 or 2022 are 1 / 2 of the width and height of the first encoding unit 2010 or 2020, and the depth of the third encoding unit 2014 or 2024 can be D+2, wherein the width and height of the third encoding unit 2014 or 2024 are 1 / 2^2 of the width and height of the first encoding unit 2010 or 2020.

[0567] Figure 21 The diagram illustrates a partial index (PID) for distinguishing depth and coding units, which can be determined based on the shape and size of the coding unit, according to an embodiment.

[0568] According to an embodiment, the video decoding device 100 can determine second coding units of various shapes by dividing a first coding unit 1400 having a square shape. (Refer to...) Figure 21 The video decoding device 100 can determine the second coding units 2102a and 2102b, 2104a and 2104b, or 2106a to 2106d by dividing the first coding unit 2100 according to at least one of the horizontal and vertical directions based on the division shape information. In other words, the video decoding device 100 can determine the second coding units 2102a and 2102b, 2104a and 2104b, or 2106a to 2106d based on the division shape information of the first coding unit 2100.

[0569] According to an embodiment, the depth of the second coding units 2102a and 2102b, 2104a and 2104b, or 2106a to 2106d, determined based on the division shape information of the first coding unit 2100 having a square shape, can be determined based on the length of the longer side. For example, since the length of one side of the first coding unit 2100 having a square shape is the same as the length of the longer side of the second coding units 2102a and 2102b or 2104a and 2104b having a non-square shape, the depth of the first coding unit 2100 and the second coding units 2102a and 2102b or 2104a and 2104b having a non-square shape can be the same, i.e., D. On the other hand, when the video decoding device 100 divides the first encoding unit 2100 into four second encoding units 2106a to 2106d with square shapes based on the division shape information, the length of one side of the second encoding units 2106a to 2106d with square shapes is 1 / 2 the length of one side of the first encoding unit 2100, and the depth of the second encoding units 2106a to 2106d can be D+1, that is, a depth lower than the depth D of the first encoding unit 2100.

[0570] According to an embodiment, the video decoding device 100 can divide the first encoding unit 2110, whose height is longer than its width, into a plurality of second encoding units 2112a and 2112b or 2114a to 2114c in the horizontal direction according to the division shape information. According to an embodiment, the video decoding device 100 can also divide the first encoding unit 2120, whose length is longer than its height, into a plurality of second encoding units 2122a and 2122b or 2124a to 2124c in the vertical direction according to the division shape information.

[0571] According to an embodiment, the depths of the second coding units 2112a and 2112b, 2114a to 2114c, 2122a and 2122b, or 2124a to 2124c, determined based on the division shape information of the first coding unit 2110 or 2120 having a non-square shape, can be determined based on the length of the long side. For example, since the length of one side of the second coding units 2112a and 2112b having a square shape is half the length of the long side of the first coding unit 2110 having a non-square shape (the height is longer than the width), the depth of the second coding units 2112a and 2112b is D+1, that is, a depth lower than the depth D of the first coding unit 2110 having a non-square shape.

[0572] Furthermore, the video decoding device 100 can divide the first coding unit 2110 with a non-square shape into an odd number of second coding units 2114a to 2114c based on the division shape information. The odd number of second coding units 2114a to 2114c may include second coding units 2114a and 2114c with non-square shapes and second coding units 2114b with square shapes. In this case, since the length of the longer side of the second coding units 2114a and 2114c with non-square shapes and the length of one side of the second coding unit 2114b with square shapes are half the length of one side of the first coding unit 2110, the depth of the second coding units 2114a to 2114c can be D+1, that is, a depth lower than the depth D of the first coding unit 2110. The video decoding device 100 can determine the depth of the coding unit associated with the first coding unit 2120 with a non-square shape (whose width is longer than its height) in the same manner as determining the depth of the coding unit associated with the first coding unit 2110.

[0573] According to an embodiment, for the operation of determining the PID for distinguishing coding units, when an odd number of coding units do not have the same size, the video decoding device 100 may determine the PID based on the size ratio of the coding units. (Refer to...) Figure 21 A second coding unit 2114b located in the middle of an odd number of second coding units 2114a to 2114c may have the same width as second coding units 2114a and 2114c, but its height is twice that of second coding units 2114a and 2114c. In this case, the middle second coding unit 2114b may include two second coding units 2114a and 2114c. Therefore, when the PID of the middle second coding unit 2114b is 1 according to the scan order, the PID of the next sequential second coding unit 2114c may be 3, which is increased by 2. In other words, the PID values ​​may be discontinuous. According to an embodiment, the video decoding device 100 may determine whether an odd number of coding units have the same size based on the discontinuity of the PID used to distinguish coding units.

[0574] According to an embodiment, the video decoding device 100 can determine, based on the value of the PID, whether a plurality of coding units determined when the current coding unit is divided have a specific division shape. (See also...) Figure 21The video decoding device 100 can determine an even number of second coding units 2112a and 2112b or an odd number of second coding units 2114a to 2114c by dividing a first coding unit 2110 having a rectangular shape with a height greater than its width. The video decoding device 100 can use a PID indicating each coding unit to distinguish multiple coding units. According to an embodiment, the PID can be obtained from a sample point at a specific location (e.g., the upper left sample point) of each coding unit.

[0575] According to an embodiment, the video decoding device 100 can determine a coding unit at a specific position from a determined set of coding units by using a PID for distinguishing coding units. According to an embodiment, when the partitioning shape information of a first coding unit 2110 having a rectangular shape with a height greater than its width indicates that the first coding unit 2110 is divided into three coding units, the video decoding device 100 can divide the first coding unit 2110 into three second coding units 2114a to 2114c. The video decoding device 100 can assign a PID to each of the three second coding units 2114a to 2114c. The video decoding device 100 can compare the PIDs of an odd number of coding units to determine an intermediate coding unit from among the odd number of coding units. The video decoding device 100 can determine, based on the PID of the coding unit, a second coding unit 2114b having a PID corresponding to the middle value among a plurality of PIDs as the coding unit located at the middle position among the plurality of coding units determined when the first coding unit 2110 was partitioned. According to an embodiment, while determining the PID for distinguishing coding units, when the coding units do not have the same size, the video decoding device 100 can determine the PID based on the size ratio of the coding units. (Refer to...) Figure 21When the first coding unit 2110 is divided, the resulting second coding unit 2114b may have the same width as the second coding units 2114a and 2114c, but its height may be twice the height of the second coding units 2114a and 2114c. In this case, when the PID of the middle second coding unit 2114b is 1, the PID of the next sequential second coding unit 2114c may be 3, which is increased by 2. Thus, when the PID increases consistently while the range of PID increase differs, the video decoding device 100 can determine that the current coding unit is divided into multiple coding units including coding units with different sizes than other coding units. According to an embodiment, when the division shape information indicates division into an odd number of coding units, the video decoding device 100 may divide the current coding unit into multiple coding units, wherein a coding unit at a specific position (e.g., an intermediate coding unit) among the multiple coding units has a different size than other coding units. In this case, the video decoding device 100 can determine the intermediate coding unit with a different size by using the PID of the coding unit. However, the PIDs and sizes or positions of the encoding units at specific locations described above are specified to describe embodiments and should not be interpreted as limiting. Various PIDs, positions, and sizes of encoding units can be used.

[0576] According to an embodiment, the video decoding device 100 may use a specific data unit, wherein the recursive partitioning of the encoding unit begins from that specific data unit.

[0577] Figure 22 The multiple encoding units shown in the embodiment are determined based on multiple specific data units included in the image.

[0578] According to an embodiment, a specific data unit can be defined as a data unit that is recursively divided into coding units using at least one of block shape information and partition shape information, starting from that data unit. In other words, a specific data unit may correspond to the highest-depth coding unit used while determining multiple coding units by partitioning the current frame. Hereinafter, for ease of description, the specific data unit will be referred to as a reference data unit.

[0579] According to an embodiment, the reference data unit may indicate a specific size and shape. According to an embodiment, the reference data unit may include M×N sample points. Here, M and N can be the same and can be integers representing multiples of 2. In other words, the reference data unit may indicate a square shape or a non-square shape and may subsequently be divided into an integer number of coded units.

[0580] According to an embodiment, the video decoding device 100 can divide the current frame into a plurality of reference data units. According to an embodiment, the video decoding device 100 can divide the plurality of reference data units by using division shape information about each of the plurality of reference data units obtained by dividing the current frame. Such reference data unit division processing can correspond to division processing using a quadtree structure.

[0581] According to an embodiment, the video decoding device 100 may predetermine the minimum size of a reference data unit that can be included in the current frame. Therefore, the video decoding device 100 may determine reference data units of various sizes that are equal to or greater than the minimum size, and determine at least one encoding unit based on the determined reference data units using block shape information and partition shape information.

[0582] refer to Figure 22 The video decoding device 100 may use a reference coding unit 2200 having a square shape, or it may use a reference coding unit 2202 having a non-square shape. According to an embodiment, the shape and size of the reference coding unit may be determined based on various data units (e.g., sequences, frames, stripes, strip segments, and maximum coding units) that may include at least one reference coding unit.

[0583] According to an embodiment, the acquirer 105 of the video decoding device 100 can obtain at least one of information about the shape of a reference coding unit and information about the size of a reference coding unit from the bitstream based on various data units. This has already been demonstrated above through... Figure 10 The process of dividing the current coding unit 1000 describes the process of determining at least one coding unit included in the reference coding unit 2200 having a square shape, and the above has been described by... Figure 11 The process of dividing the current coding unit 1100 or 1150 describes the process of determining at least one coding unit included in the reference coding unit 2200 which has a non-square shape, and therefore its details will not be provided again.

[0584] According to an embodiment, in order to determine the size and shape of a reference coding unit based on some data units predetermined based on predetermined conditions, the video decoding device 100 can use a PID for distinguishing the size and shape of the reference coding unit. In other words, the acquirer 105 can obtain only the PID for distinguishing the size and shape of the reference coding unit from the bitstream based on stripes, strip segments, and maximum coding units, wherein the reference coding unit is a data unit among various data units (e.g., sequences, frames, stripes, strip segments, and maximum coding units) that meets predetermined conditions (e.g., a data unit with a size equal to or smaller than the stripe). The video decoding device 100 can determine the size and shape of the reference data unit based on the data unit that meets the predetermined conditions by using the PID. When information about the shape and size of the reference coding unit is obtained from the bitstream and used based on data units with relatively small sizes, the utilization efficiency of the bitstream is insufficient, so the information about the shape and size of the reference coding unit is not obtained directly, but only the PID is obtained and used. In this case, at least one of the size and shape of the reference coding unit corresponding to the PID indicating the size and shape of the reference coding unit can be predetermined. In other words, the video decoding device 100 can select at least one of the dimensions and shapes of a predetermined reference coding unit based on the PID to determine at least one of the dimensions and shapes of the reference coding unit included in the data unit as a standard for obtaining the PID.

[0585] According to embodiments, the video decoding device 100 may use at least one reference coding unit included in a maximum coding unit. In other words, the maximum coding unit for dividing an image may include at least one reference coding unit, and a coding unit can be determined when each of the reference coding units is recursively divided. According to embodiments, at least one of the width and height of the maximum coding unit may be an integer multiple of at least one of the width and height of the reference coding unit. According to embodiments, the size of the reference coding unit may be equal to the size of the maximum coding unit, wherein the maximum coding unit is divided n times according to a quadtree structure. In other words, according to various embodiments, the video decoding device 100 may determine a reference coding unit by dividing the maximum coding unit n times according to a quadtree structure, and divide the reference coding unit based on at least one of block shape information and partition shape information.

[0586] Figure 23 The processing block shown is a standard for determining the order of reference coding units included in screen 2300, according to an embodiment.

[0587] According to an embodiment, the video decoding device 100 can determine at least one processing block for dividing a screen. A processing block is a data unit comprising at least one reference coding unit for dividing the image, and the at least one reference coding unit included in the processing block can be determined in a specific order. In other words, the determination order of the at least one reference coding unit determined in each processing block may correspond to one of various orders used to determine the reference coding units, and may change depending on the processing block. The determination order of the reference coding units determined in each processing block may be one of various orders (such as raster scan order, Z-scan order, N-scan order, upper right diagonal scan order, horizontal scan order, and vertical scan order), but should not be interpreted restrictively regarding the scan order.

[0588] According to an embodiment, the video decoding device 100 can determine the size of at least one processing block included in an image by obtaining information about the size of the processing block. The video decoding device 100 can obtain information about the size of the processing block from the bitstream to determine the size of the at least one processing block included in the image. The size of the processing block can be a specific size of a data unit indicated by information about the size of the processing block.

[0589] According to an embodiment, the acquirer 105 of the video decoding device 100 can obtain information about the size of a processing block from the bitstream based on specific data units. For example, the information about the size of the processing block can be obtained from the bitstream according to data units of images, sequences, frames, stripes, and strip segments. In other words, the acquirer 105 can obtain information about the size of the processing block from the bitstream based on several such data units, and the video decoding device 100 can determine the size of at least one processing block for dividing a frame by using the obtained information about the size of the processing block, wherein the size of the processing block can be an integer multiple of the size of a reference coding unit.

[0590] According to an embodiment, the video decoding device 100 can determine the sizes of processing blocks 2302 and 2312 included in the frame 2300. For example, the video decoding device 100 can determine the size of the processing block based on information about the size of the processing block obtained from the bitstream. (See also...) Figure 23 According to an embodiment, the video decoding device 100 may determine the horizontal dimensions of processing blocks 2302 and 2312 to be four times the horizontal dimensions of the reference coding unit, and the vertical dimensions of processing blocks 2302 and 2312 to be four times the vertical dimensions of the reference coding unit. The video decoding device 100 may determine the determination order of at least one reference coding unit in at least one processing block.

[0591] According to an embodiment, the video decoding device 100 may determine each of the processing blocks 2302 and 2312 included in the frame 2300 based on the size of the processing blocks, and may determine the determination order of at least one reference coding unit included in each of the processing blocks 2302 and 2312. According to an embodiment, the step of determining the reference coding unit may include determining the size of the reference coding unit.

[0592] According to an embodiment, the video decoding device 100 can obtain information from a bitstream regarding the determined order of at least one reference coding unit included in at least one processing block, and determine the determined order of the at least one reference coding unit based on the obtained information. The information regarding the determined order can be defined as the order or direction of the reference coding units within the processing block. In other words, the determined order of the reference coding units can be determined independently for each processing block.

[0593] According to an embodiment, the video decoding device 100 can obtain information about the determined order of reference coding units from the bitstream based on specific data units. For example, the acquirer 105 can obtain information about the determined order of reference coding units from the bitstream based on data units (such as images, sequences, frames, stripes, strip segments, and processing blocks). Since the information about the determined order of reference coding units indicates the determined order of reference coding units in a processing block, the information about the determined order can be obtained for each specific data unit (including an integer number of processing blocks).

[0594] According to an embodiment, the video decoding device 100 may determine at least one reference coding unit based on a determined order.

[0595] According to an embodiment, the acquirer 105 can obtain information about the determined order of reference coding units from the bitstream as information related to processing blocks 2302 and 2312, and the video decoding device 100 can determine the order of at least one reference coding unit included in processing blocks 2302 and 2312, and determine at least one reference coding unit included in frame 2300 based on the determined order of the coding units. (Refer to...) Figure 23The video decoding device 100 can determine determination orders 2304 and 2314 of at least one reference coding unit associated with processing blocks 2302 and 2312, respectively. For example, when information regarding the determination order of reference coding units is obtained for each processing block, the determination orders of the reference coding units associated with processing blocks 2302 and 2312 may differ from each other. When the determination order 2304 associated with processing block 2302 is a raster scan order, the reference coding units included in processing block 2302 can be determined according to the raster scan order. On the other hand, when the determination order 2314 associated with processing block 2312 is the reverse of the raster scan order, the reference coding units included in processing block 2312 can be determined according to the reverse of the raster scan order.

[0596] According to an embodiment, the video decoding device 100 can decode at least one determined reference coding unit. The video decoding device 100 can decode an image based on the reference coding unit determined through the above embodiments. Examples of methods for decoding the reference coding unit may include various methods for decoding images.

[0597] According to an embodiment, the video decoding device 100 can obtain and use block shape information indicating the shape of the current coding unit or partition shape information indicating the method of partitioning the current coding unit from the bitstream. The block shape information or partition shape information can be included in the bitstream associated with various data units. For example, the video decoding device 100 can use block shape information or partition shape information included in sequence parameter sets, picture parameter sets, video parameter sets, strip headers, and strip segment headers. Furthermore, the video decoding device 100 can obtain and use syntax corresponding to the block shape information or partition shape information from the bitstream based on the maximum coding unit, reference coding unit, and processing block.

[0598] While this disclosure has been specifically shown and described with reference to embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this disclosure as defined by the appended claims. The embodiments are to be understood in a descriptive sense only and not for limiting purposes. Therefore, the scope of this disclosure is not limited by the detailed description thereof, but by the appended claims, and all distinctions within that scope shall be construed as included in this disclosure.

[0599] The embodiments of this disclosure can be written as computer programs and implemented in a general-purpose digital computer that executes the programs using a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.), optical recording media (e.g., CD-ROM or DVD), etc.

Claims

1. A video decoding method, comprising: Information related to a first motion vector and information related to a second motion vector are obtained from the bit stream, wherein the first motion vector indicates the first reference block of the current block in the first reference frame, and the second motion vector indicates the second reference block of the current block in the second reference frame; Whether to perform optical flow-based compensation is determined by using at least one of the following flags: whether the current block is bidirectionally predicted and whether optical flow-based compensation is available. If optical flow-based compensation is to be performed, a first displacement vector in the horizontal direction and a second displacement vector in the vertical direction are obtained for a group of pixels in the current block, wherein the group of pixels includes at least one pixel; Based on the pixel values ​​of the first reference block, the pixel values ​​of the second reference block, and the values ​​obtained by using the first and second displacement vectors for the pixel group, a predicted pixel value corresponding to the pixel group is obtained; and Reconstruct the current block based on the predicted pixel values. The flag information is obtained from the bit stream. The size of the pixel group is 4×4. The current frame is divided into multiple maximum coding units, and the current block is obtained by dividing the maximum coding unit among the multiple maximum coding units.

2. A video encoding method, comprising: Whether to perform optical flow-based compensation is determined by using at least one of the following flags: whether the current block is bidirectionally predicted and whether optical flow-based compensation is available. If optical flow-based compensation is to be performed, a first displacement vector in the horizontal direction and a second displacement vector in the vertical direction are obtained for a group of pixels in the current block, wherein the group of pixels includes at least one pixel; Based on the pixel values ​​of the first reference block, the pixel values ​​of the second reference block, and the values ​​obtained by using the first and second displacement vectors for the pixel group, a predicted pixel value corresponding to the pixel group is obtained. Encode the current block based on the predicted pixel values; and Generate a bitstream, wherein the bitstream includes an encoding result, information related to a first motion vector, and information related to a second motion vector, wherein the first motion vector indicates a first reference block of the current block in a first reference frame, and the second motion vector indicates a second reference block of the current block in a second reference frame; The flag information is included in the bitstream. The size of the pixel group is 4×4. The current frame is divided into multiple maximum coding units, and the current block is obtained by dividing the maximum coding unit among the multiple maximum coding units.

3. A method for transmitting a bit stream, comprising: Perform the video encoding method according to claim 2 to generate a bitstream; as well as Send the bit stream.

Images