Video decoding method and apparatus and video encoding method and apparatus
Patent Information
- Application Number
- IN202328053811
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2017-01-04
- Filing Date
- 2023-08-10
- Publication Date
- 2026-10-05
AI Technical Summary
Existing video codecs face inefficiencies in encoding and decoding high-resolution video content due to limitations in inter prediction methods, particularly in bi-directional motion prediction modes, which require additional memory accesses and complex multiplication operations when handling pixels at block boundaries.
A video decoding and encoding method that uses pixel values and gradient values from reference blocks to determine displacement vectors for motion compensation, minimizing memory accesses and operations by applying interpolation and gradient filters to fractional pixel locations without referencing pixels outside the reference block boundaries.
Enhances encoding and decoding efficiency by predicting pixel values using gradient values from reference blocks, reducing memory accesses and operations while maintaining accurate displacement vector calculations.
Abstract
Description
TECHNICAL FIELD
[0001] The disclosure relates to a video decoding method and apparatus and a videoencoding method and apparatus. More particularly, the disclosure relates to videodecoding and video encoding methods and apparatuses for performing inter prediction ina bi-directional motion prediction mode.BACKGROUND ART
[0002] With the development and spread of hardware capable of reproducing andstoring high-resolution or high-definition video content, the need for a video codec thateffectively encodes or decodes high-resolution or high-definition video content isincreasing. According to an existing video codec, video is encoded according to a limitedencoding scheme based on coding units having a tree structure.
[0003] Image data in a spatial domain is transformed into coefficients in a frequencydomain by using a frequency transform. To perform a fast calculation in the frequencytransform, a video codec splits an image into blocks having a predetermined size,performs discrete cosine transform (DCT) transformation on each of the blocks, and thusencodes frequency coefficients in each block unit. Compared to the image data in thespatial domain, the coefficients in the frequency domain are easily compressed.Because an image pixel value in the spatial domain is expressed as a prediction error viainter prediction or intra prediction by the video codec, when the frequency transform isperformed on prediction errors, much data may be transformed into 0. The video codecdecreases an amount of data by changing data, which occurs sequentially andrepeatedly, into small-sized data.DESCRIPTION OF EMBODIMENTSTECHNICAL PROBLEM
[0004] According to various embodiments, a predicted pixel value of a current blockmay be generated by using not only a pixel value of a first reference block of a firstreference picture and a pixel value of a second reference block of a second referencepicture in a bi-directional motion prediction mode but also using a first gradient value of afirst reference block and a second gradient value of a second reference block. Therefore,a prediction block similar to an original block may be generated such that encoding anddecoding efficiency may be enhanced.
[0005] The pixel value of the first reference block, the pixel value of the secondreference block, the first gradient value of the first reference block, and the secondgradient value of the second reference block may be used to determine a displacementvector of the current block in a horizontal direction or a vertical direction when pixelgroup unit motion compensation is performed. In particular, to determine a displacementvector of a current pixel in the current block in a horizontal direction or a vertical direction,not only a pixel value and a gradient value of a first reference pixel in the first referenceblock, the first reference pixel corresponding to the current pixel, and a pixel value and agradient value of a second reference pixel in the second reference block but also pixelvalues and gradient values of neighboring pixels included in a window having apredetermined size with respect to the first reference pixel and the second referencepixel are used. Therefore, when the current pixel is located at a boundary, there is aproblem that an additional access to a memory is required because, when a neighboringpixel of a reference pixel corresponding to the current pixel is located outside a referenceblock, a pixel value and a gradient value of a pixel located outside the reference blockhave to be referenced.
[0006] According to various embodiments, the displacement vector of the current blockin the horizontal direction or the vertical direction is determined by referring to only apixel value and a gradient value of a pixel located in the reference block, withoutreferring to the pixel value and the gradient value that are stored with respect to the pixellocated outside the reference block, such that the number of memory accesses may beminimized.
[0007] According to various embodiments, to determine a gradient value of thereference pixel in the horizontal direction or the vertical direction, a gradient filter and aninterpolation filter in the horizontal direction or the vertical direction that use an input of apixel value of an integer pixel are not used, but instead, a pixel value of a pixel at alocation of a fractional pixel unit is determined by applying the interpolation filter to thepixel value of the integer pixel, and the gradient value of the reference pixel in thehorizontal direction or the vertical direction is determined by applying the gradient filter inthe horizontal direction or the vertical direction to the pixel value of the pixel at thelocation of the fractional pixel unit, the gradient filter having a relatively less length, suchthat the number of complicated multiplication operations may be minimized.
[0008] According to various embodiments, motion compensation is performed in a pixelgroup unit, such that the number of complicated multiplication operations may beminimized, compared to motion compensation performed in a pixel unit.
[0009] Provided is a computer-readable recording medium having recorded thereon aprogram for executing a method according to various embodiments.
[0010] The technical problems of various embodiments are not limited to theaforementioned features, and other unstated technical problems will be clearlyunderstood by one of ordinary skill in the art in view of descriptions below.SOLUTION TO PROBLEM
[0011] The technical problems of the present disclosure are not limited to theaforementioned features, and other unstated technical problems will be clearlyunderstood by one of ordinary skill in the art in view of descriptions below.
[0012] According to various embodiments, a video decoding method may includeobtaining, from a bitstream, motion prediction mode information about a current block ina current picture;
[0013] when the obtained motion prediction mode information indicates a bi-directionalmotion prediction mode, obtaining, from the bitstream, information about a first motionvector indicating a first reference block of a current block in a first reference picture andinformation about a second motion vector indicating a second reference block of thecurrent block in a second reference picture;
[0014] determining a displacement vector per unit time of pixels of the current block in ahorizontal direction or a vertical direction, the pixels including a pixel adjacent to aninside of a boundary of the current block, by using values about reference pixels includedin the first reference block and the second reference block, without using a stored valueabout a pixel located outside boundaries of the first reference block and the secondreference block;
[0015] obtaining a prediction block of the current block by performing block-unitmotion compensation and pixel group unit motion compensation on the current block byusing a gradient value in the horizontal direction or the vertical direction of a firstcorresponding reference pixel in the first reference block which corresponds to a currentpixel included in a current pixel group in the current block, a gradient value in thehorizontal direction or the vertical direction of a second corresponding reference pixel inthe second reference block which corresponds to the current pixel, a pixel value of thefirst corresponding reference pixel, a pixel value of the second corresponding referencepixel, and a displacement vector per unit time of the current pixel in the horizontaldirection or the vertical direction;
[0016] obtaining, from the bitstream, information about a residual block of the currentblock; and
[0017] reconstructing the current block based on the prediction block and the residualblock,
[0018] wherein the current pixel group includes at least one pixel.
[0019] In the video decoding method according to various embodiments,
[0020] the obtaining of the prediction block of the current block may further includecalculating the gradient value in the horizontal direction or the vertical direction of the firstcorresponding reference pixel or the second corresponding reference pixel by applying agradient filter in the horizontal direction or the vertical direction to pixel values of pixels atfractional locations in the first reference block or the second reference block,
[0021] the gradient filter may a 5-tap filter, and
[0022] the pixels at the fractional locations may be each a pixel in which at least one ofa horizontal component or a vertical component of coordinates indicating a location ofthe pixel has a fractional value.
[0023] In the video decoding method according to various embodiments, pixel values ofthe pixels at the fractional locations in the first reference block or the second referenceblock may be calculated by applying an interpolation filter in the horizontal direction orthe vertical direction to pixel values of pixels at integer locations.
[0024] In the video decoding method according to various embodiments, a size of thecurrent pixel group may be determined based on a minimum value from among values ofa height and a width of the current block.
[0025] In the video decoding method according to various embodiments, thedisplacement vector per unit time in the horizontal direction or the vertical direction withrespect to the current pixel group may be a displacement vector per unit time determinedby using pixel values of first corresponding reference pixels and neighboring pixels of thefirst corresponding reference pixels in a first corresponding reference pixel group in thefirst reference picture, the first corresponding reference pixel group corresponding to thecurrent pixel group, pixel values of second corresponding reference pixels andneighboring pixels of the second corresponding reference pixels in a secondcorresponding reference pixel group in the second reference picture, gradient values, afirst picture order count (POC) difference between the first reference picture and thecurrent picture, and a second POC difference between the second reference picture andthe current picture.
[0026] In the video decoding method according to various embodiments, thedetermining of the displacement vector per unit time of the pixels of the current block inthe horizontal direction or the vertical direction may include,
[0027] when the first corresponding reference pixel or the second correspondingreference pixel is a boundary neighboring pixel adjacent to an inside of a boundary of thefirst reference block or the second reference block,
[0028] deriving, by using a pixel value of the boundary neighboring pixel, a pixel valueand a gradient value of a pixel located outside the boundary of the first reference block orthe second reference block, and
[0029] determining the displacement vector per unit time of the current pixel in thehorizontal direction or the vertical direction, based on the pixel value and a gradientvalue of the boundary neighboring pixel and a pixel value and a gradient value of a pixellocated outside the boundary of the current block, the pixel value and the gradient valueof the pixel being derived by using the pixel value of the boundary neighboring pixel.
[0030] In the video decoding method according to various embodiments, thedetermining of the displacement vector per unit time of the pixels of the current block inthe horizontal direction or the vertical direction may include:
[0031] calculating a value about the current pixel by using a pixel value of the firstcorresponding reference pixel included in the first reference block, a pixel value of thesecond corresponding reference pixel included in the second reference block, thegradient value of the first corresponding reference pixel, the gradient value of the secondcorresponding reference pixel, a first POC difference between the first reference pictureand the current picture, and a second POC difference between the second referencepicture and the current picture;
[0032] calculating values about neighboring pixels by using pixel values of firstcorresponding neighboring pixels of the first corresponding reference pixel, pixel valuesof second corresponding neighboring pixels of the second corresponding reference pixel,gradient values of the first corresponding neighboring pixels, gradient values of thesecond corresponding neighboring pixels, the first POC difference between the firstreference picture and the current picture, and the second POC difference between thesecond reference picture and the current picture;
[0033] calculating a weighted average value for the current pixel, the weighted averagevalue being required to calculate a displacement vector per unit time in the horizontaldirection or the vertical direction, by using the value about the current pixel, values aboutneighboring pixels, and a weight; and
[0034] determining the displacement vector per unit time of the current pixel in thehorizontal direction or the vertical direction by using the calculated weighted averagevalue for the current pixel.
[0035] In the video decoding method according to various embodiments, the weightedaverage value for the current pixel may be a value calculated by applying an exponentialsmoothing technique in the vertical and horizontal directions to values about pixelsincluded in the first reference block and the second reference block.
[0036] According to various embodiment, a video decoding apparatus may include
[0037] an obtainer configured to obtain, from a bitstream, motion prediction modeinformation about a current block in a current picture, when the obtained motionprediction mode information indicates a bi-directional motion prediction mode, obtain,from the bitstream, information about a first motion vector indicating a first referenceblock of a current block in a first reference picture and information about a secondmotion vector indicating a second reference block of the current block in a secondreference picture, and obtain, from the bitstream, information about a residual block ofthe current block;
[0038] an inter predictor configured to determine a displacement vector per unit time ofpixels of the current block in a horizontal direction or a vertical direction, the pixelsincluding a pixel adjacent to an inside of a boundary of the current block, by using valuesabout reference pixels included in the first reference block and the second referenceblock, without using a stored value about a pixel located outside boundaries of the firstreference block and the second reference block, and obtain a prediction block of thecurrent block by performing block-unit motion compensation and pixel group unit motioncompensation on the current block by using a gradient value in the horizontal direction orthe vertical direction of a first corresponding reference pixel in the first reference blockwhich corresponds to a current pixel included in a current pixel group in the current block,a gradient value in the horizontal direction or the vertical direction of a secondcorresponding reference pixel in the second reference block which corresponds to thecurrent pixel, a pixel value of the first corresponding reference pixel, a pixel value of thesecond corresponding reference pixel, and a displacement vector per unit time of thecurrent pixel in the horizontal direction or the vertical direction; and
[0039] a decoder configured to reconstruct the current block based on the predictionblock and the residual block,
[0040] wherein the current pixel group includes at least one pixel.
[0041] In the video decoding apparatus according to various embodiments,
[0042] the inter predictor may be configured to calculate the gradient value in thehorizontal direction or the vertical direction of the first corresponding reference pixel orthe second corresponding reference pixel by applying a gradient filter in the horizontaldirection or the vertical direction to pixel values of pixels at fractional locations in the firstreference block or the second reference block,
[0043] the gradient filter may be a 5-tap filter, and
[0044] the pixels at the fractional locations may be each a pixel in which at least one ofa horizontal component or a vertical component of coordinates indicating a location ofthe pixel has a fractional value.
[0045] In the video decoding apparatus according to various embodiments,
[0046] the displacement vector per unit time in the horizontal direction or the verticaldirection with respect to the current pixel group may be a displacement vector per unittime determined by the inter predictor using pixel values of first corresponding referencepixels and neighboring pixels of the first corresponding reference pixels in a firstcorresponding reference pixel group in the first reference picture, the first correspondingreference pixel group corresponding to the current pixel group, pixel values of secondcorresponding reference pixels and neighboring pixels of the second correspondingreference pixels in a second corresponding reference pixel group in the secondreference picture, gradient values, a first picture order count (POC) difference betweenthe first reference picture and the current picture, and a second POC difference betweenthe second reference picture and the current picture.
[0047] In the video decoding apparatus according to various embodiments, the interpredictor may be configured to
[0048] calculate a value about the current pixel by using a pixel value of the firstcorresponding reference pixel included in the first reference block, a pixel value of thesecond corresponding reference pixel included in the second reference block, thegradient value of the first corresponding reference pixel, the gradient value of the secondcorresponding reference pixel, a first POC difference between the first reference pictureand the current picture, and a second POC difference between the second referencepicture and the current picture,
[0049] calculate values about neighboring pixels by using pixel values of firstcorresponding neighboring pixels of the first corresponding reference pixel, pixel valuesof second corresponding neighboring pixels of the second corresponding reference pixel,gradient values of the first corresponding neighboring pixels, gradient values of thesecond corresponding neighboring pixels, the first POC difference between the firstreference picture and the current picture, and the second POC difference between thesecond reference picture and the current picture,
[0050] calculate a weighted average value for the current pixel, the weighted averagevalue being required to calculate a displacement vector per unit time in the horizontaldirection or the vertical direction, by using the value about the current pixel, the valuesabout neighboring pixels, and a weight, and
[0051] determine the displacement vector per unit time of the current pixel in thehorizontal direction or the vertical direction by using the calculated weighted averagevalue for the current pixel.
[0052] According to various embodiments, a video encoding method may includeobtaining a prediction block, a first motion vector, and a second motion vector of acurrent block by performing motion compensation and pixel group unit motioncompensation on the current block; and
[0053] generating a bitstream including information about a first motion vector and asecond motion vector, and motion prediction mode information indicating that a motionprediction mode of the current block is a bi-directional motion prediction mode,
[0054] wherein a pixel group for the pixel group unit motion compensation includes atleast one pixel,
[0055] wherein the first motion vector is a motion vector indicating, from the currentblock, a first reference block of a first reference picture which corresponds to the currentblock in a current picture,
[0056] wherein the second motion vector is a motion vector indicating, from the currentblock, a second reference block of a second reference picture which corresponds to thecurrent block in the current picture,
[0057] wherein a prediction block of the current block is obtained by performingblock-unit motion compensation and pixel group unit motion compensation on thecurrent block by using a gradient value in the horizontal direction or the vertical directionof a first corresponding reference pixel in the first reference block which corresponds to acurrent pixel included in a current pixel group in the current block, a gradient value in thehorizontal direction or the vertical direction of a second corresponding reference pixel inthe second reference block which corresponds to the current pixel, a pixel value of thefirst corresponding reference pixel, a pixel value of the second corresponding referencepixel, and a displacement vector per unit time of the current pixel in the horizontaldirection or the vertical direction, and
[0058] wherein a displacement vector per unit time of pixels of the current block in thehorizontal direction or the vertical direction, the pixels including a pixel adjacent to aninside of a boundary of the current block, is determined by using values about referencepixels included in the first reference block and the second reference block, without usinga stored value about a pixel located outside boundaries of the first reference block andthe second reference block.
[0059] According to various embodiments, a video encoding apparatus may include aninter predictor configured to obtain a prediction block, a first motion vector, and a secondmotion vector of a current block by performing motion compensation and pixel group unitmotion compensation on the current block; and
[0060] a bitstream generator configured to generate a bitstream including informationabout a first motion vector and a second motion vector, and motion prediction modeinformation indicating that a motion prediction mode of the current block is abi-directional motion prediction mode,
[0061] wherein a pixel group for the pixel group unit motion compensation includes atleast one pixel,
[0062] wherein the first motion vector is a motion vector indicating, from the currentblock, a first reference block of a first reference picture which corresponds to the currentblock in a current picture, and the second motion vector is a motion vector indicating,from the current block, a second reference block of a second reference picture whichcorresponds to the current block in the current picture,
[0063] wherein a prediction block of the current block is obtained by performingblock-unit motion compensation and pixel group unit motion compensation on thecurrent block by using a gradient value in the horizontal direction or the vertical directionof a first corresponding reference pixel in the first reference block which corresponds to acurrent pixel included in a current pixel group in the current block, a gradient value in thehorizontal direction or the vertical direction of a second corresponding reference pixel inthe second reference block which corresponds to the current pixel, a pixel value of thefirst corresponding reference pixel, a pixel value of the second corresponding referencepixel, and a displacement vector per unit time of the current pixel in the horizontaldirection or the vertical direction, and
[0064] wherein a displacement vector per unit time of pixels of the current block in thehorizontal direction or the vertical direction, the pixels including a pixel adjacent to aninside of a boundary of the current block, is determined by using values about referencepixels included in the first reference block and the second reference block, without usinga stored value about a pixel located outside boundaries of the first reference block andthe second reference block.
[0065] Provided is a computer-readable recording medium having recorded thereon aprogram for executing a method according to various embodiments.ADVANTAGEOUS EFFECTS OF DISCLOSURE
[0066] According to various embodiments, a value similar to a value of an original blockof a current block is predicted by performing inter prediction on the current block by usinga gradient value of a reference block of a reference picture in a bi-directional motionprediction mode, such that encoding and decoding efficiency may be enhanced.BRIEF DESCRIPTION OF DRAWINGS
[0067] FIG. 1A illustrates a block diagram of a video decoding apparatus, according tovarious embodiments.
[0068] FIG. 1B illustrates a flowchart of a video decoding method, according to variousembodiments.
[0069] FIG. 1C illustrates a block diagram of a video encoding apparatus, according tovarious embodiments.
[0070] FIG. 1D illustrates a flowchart of a video encoding method, according to variousembodiments.
[0071] FIG. 1E illustrates a block diagram of an image decoder according to variousembodiments.
[0072] FIG. 1F illustrates a block diagram of an image encoder according to variousembodiments.
[0073] FIG. 2 is a reference diagram for describing block-based bi-directional motionprediction and compensation processes, according to an embodiment.
[0074] FIGS. 3A to 3C are reference diagrams for describing processes of performingpixel group unit motion compensation, according to an embodiment.
[0075] FIG. 4 is a reference diagram for describing processes of calculating gradientvalues in horizontal and vertical directions, according to an embodiment.
[0076] FIG. 5 is a reference diagram for describing processes of calculating gradientvalues in horizontal and vertical directions, according to another embodiment.
[0077] FIGS. 6A to 6B are diagrams for describing processes of determining gradientvalues in horizontal and vertical directions by using 1D filters, according to anembodiment.
[0078] FIGS. 7A to 7E are Tables showing filter coefficients of filters used to determinea pixel value at a fractional pixel location of a fractional pixel unit, and gradient values inhorizontal and vertical directions, according to an embodiment.
[0079] FIG. 8A is a reference diagram for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to a pixel, according to an embodiment.
[0080] FIG. 8B is a reference diagram for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to a pixel group, according to an embodiment.
[0081] FIG. 8C is a reference diagram for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to a pixel, according to an embodiment.
[0082] FIG. 8D is a reference diagram for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to a pixel group, according to an embodiment.
[0083] FIG. 9A is a diagram for describing processes of adding an offset value afterfiltering is performed, and determining a gradient value in a horizontal or verticaldirection by performing de-scaling, according to an embodiment.
[0084] FIG. 9B is a diagram for describing processes of adding an offset value afterfiltering is performed, and determining a gradient value in a horizontal or verticaldirection by performing de-scaling, according to another embodiment.
[0085] FIG. 9C is a diagram for describing a necessary input range for determining ahorizontal direction displacement vector and a vertical direction displacement vectorduring processes of performing pixel-unit motion compensation on a current block.
[0086] FIGS. 9D to 9E are diagrams for determining an input range of a region used inprocesses of performing pixel-unit motion compensation, according to variousembodiments.
[0087] FIG. 9F is a diagram for describing processes of determining a horizontaldirection displacement vector and a vertical direction displacement vector, withoutenlarging a reference block.
[0088] FIGS. 9G to 9I are diagrams for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vectorwithout enlarging a reference block, according to other embodiments.
[0089] FIG. 9J is a diagram for describing processes of determining a horizontaldirection displacement vector and a vertical direction displacement vector with respect toeach pixel by applying an exponential smoothing technique in vertical and horizontaldirections by referring to a pixel value and a gradient value of a reference block, withoutenlargement of a block, according to an embodiment.
[0090] FIG. 9K is a diagram for describing processes of determining a pixel value andgradient values in horizontal and vertical directions of a reference pixel in a referenceblock by using a filter so as to perform motion compensation on a current block,according to an embodiment.
[0091] FIG. 9I is a diagram for describing processes of determining a pixel value andgradient values in horizontal and vertical directions of a reference pixel in a referenceblock by using a filter so as to perform motion compensation on a current block,according to another embodiment.
[0092] FIG. 10 illustrates a process of determining at least one coding unit by splitting acurrent coding unit, according to an embodiment.
[0093] FIG. 11 illustrates a process of determining at least one coding unit by splitting anon-square coding unit, according to an embodiment.
[0094] FIG. 12 illustrates a process of splitting a coding unit based on at least one ofblock shape information and split shape information, according to an embodiment.
[0095] FIG. 13 illustrates a method of determining a predetermined coding unit fromamong an odd number of coding units, according to an embodiment.
[0096] FIG. 14 illustrates an order of processing a plurality of coding units when theplurality of coding units are determined by splitting a current coding unit, according to anembodiment.
[0097] FIG. 15 illustrates a process of determining that a current coding unit is to besplit into an odd number of coding units, when the coding units are not processable in apredetermined order, according to an embodiment.
[0098] FIG. 16 illustrates a process of determining at least one coding unit by splitting afirst coding unit, according to an embodiment.
[0099] FIG. 17 illustrates that, when a second coding unit having a non-square shape isdetermined by splitting a first coding unit and satisfies a predetermined condition, ashape into which the second coding unit is split table is restricted, according to anembodiment.
[00100] FIG. 18 illustrates a process of splitting a square coding unit when split shapeinformation indicates that the square coding unit is not to be split into four square codingunits, according to an embodiment.
[00101] FIG. 19 illustrates that a processing order between a plurality of coding unitsmay be changed depending on a process of splitting a coding unit, according to anembodiment.
[00102] FIG. 20 illustrates a process of determining a depth of a coding unit as a shapeand size of the coding unit change, when the coding unit is recursively split such that aplurality of coding units are determined, according to an embodiment.
[00103] FIG. 21 illustrates depths determinable based on shapes and sizes of codingunits, and part indexes (PIDs) for distinguishing the coding units, according to anembodiment.
[00104] FIG. 22 illustrates that a plurality of coding units are determined based on aplurality of predetermined data units included in a picture, according to an embodiment.
[00105] FIG. 23 illustrates a processing block serving as a unit for determining adetermination order of reference coding units included in a picture, according to anembodiment.BEST MODE
[00106] According to various embodiments, a video decoding method may includeobtaining, from a bitstream, motion prediction mode information about a current block ina current picture;
[00107] when the obtained motion prediction mode information indicates abi-directional motion prediction mode, obtaining, from the bitstream, information about afirst motion vector indicating a first reference block of a current block in a first referencepicture and information about a second motion vector indicating a second referenceblock of the current block in a second reference picture;
[00108] determining a displacement vector per unit time of pixels of the current block ina horizontal direction or a vertical direction, the pixels including a pixel adjacent to aninside of a boundary of the current block, by using values about reference pixels includedin the first reference block and the second reference block, without using a stored valueof a pixel located outside boundaries of the first reference block and the secondreference block;
[00109] obtaining a prediction block of the current block by performing block-unitmotion compensation and pixel group unit motion compensation on the current block byusing a gradient value in the horizontal direction or the vertical direction of a firstcorresponding reference pixel in the first reference block which corresponds to a currentpixel included in a current pixel group in the current block, a gradient value in thehorizontal direction or the vertical direction of a second corresponding reference pixel inthe second reference block which corresponds to the current pixel, a pixel value of thefirst corresponding reference pixel, a pixel value of the second corresponding referencepixel, and a displacement vector per unit time of the current pixel in the horizontaldirection or the vertical direction;
[00110] obtaining, from the bitstream, information about a residual block of the currentblock; and
[00111] reconstructing the current block based on the prediction block and the residualblock,
[00112] wherein the current pixel group includes at least one pixel.
[00113] According to various embodiments, a video decoding apparatus includes
[00114] an obtainer configured to obtain, from a bitstream, motion prediction modeinformation about a current block in a current picture, when the obtained motionprediction mode information indicates a bi-directional motion prediction mode, obtain,from the bitstream, information about a first motion vector indicating a first referenceblock of a current block in a first reference picture and information about a secondmotion vector indicating a second reference block of the current block in a secondreference picture, and obtain, from the bitstream, information about a residual block ofthe current block;
[00115] an inter predictor configured to determine a displacement vector per unit timeof pixels of the current block in a horizontal direction or a vertical direction, the pixelsincluding a pixel adjacent to an inside of a boundary of the current block, by using valuesabout reference pixels included in the first reference block and the second referenceblock, without using a stored value about a pixel located outside boundaries of the firstreference block and the second reference block, and obtain a prediction block of thecurrent block by performing block-unit motion compensation and pixel group unit motioncompensation on the current block by using a gradient value in the horizontal direction orthe vertical direction of a first corresponding reference pixel in the first reference blockwhich corresponds to a current pixel included in a current pixel group in the current block,a gradient value in the horizontal direction or the vertical direction of a secondcorresponding reference pixel in the second reference block which corresponds to thecurrent pixel, a pixel value of the first corresponding reference pixel, a pixel value of thesecond corresponding reference pixel, and a displacement vector per unit time of thecurrent pixel in the horizontal direction or the vertical direction; and
[00116] a decoder configured to reconstruct the current block based on the predictionblock and the residual block,
[00117] wherein the current pixel group includes at least one pixel.
[00118] According to various embodiments, a video encoding method includesobtaining a prediction block, a first motion vector, and a second motion vector of acurrent block by performing motion compensation and pixel group unit motioncompensation on the current block; and
[00119] generating a bitstream including information about a first motion vector and asecond motion vector, and motion prediction mode information indicating that a motionprediction mode of the current block is a bi-directional motion prediction mode,
[00120] wherein a pixel group for the pixel group unit motion compensation includes atleast one pixel,
[00121] wherein the first motion vector is a motion vector indicating, from the currentblock, a first reference block of a first reference picture which corresponds to the currentblock in a current picture,
[00122] wherein the second motion vector is a motion vector indicating, from thecurrent block, a second reference block of a second reference picture which correspondsto the current block in the current picture,
[00123] wherein a prediction block of the current block is obtained by performingblock-unit motion compensation and pixel group unit motion compensation on thecurrent block by using a gradient value in the horizontal direction or the vertical directionof a first corresponding reference pixel in the first reference block which corresponds to acurrent pixel included in a current pixel group in the current block, a gradient value in thehorizontal direction or the vertical direction of a second corresponding reference pixel inthe second reference block which corresponds to the current pixel, a pixel value of thefirst corresponding reference pixel, a pixel value of the second corresponding referencepixel, and a displacement vector per unit time of the current pixel in the horizontaldirection or the vertical direction, and
[00124] wherein a displacement vector per unit time of pixels of the current block in thehorizontal direction or the vertical direction, the pixels including a pixel being adjacent toan inside of a boundary of the current block, is determined by using values aboutreference pixels included in the first reference block and the second reference block,without using a stored value about a pixel located outside boundaries of the firstreference block and the second reference block.
[00125] According to various embodiments, a video encoding apparatus includes aninter predictor configured to obtain a prediction block, a first motion vector, and a secondmotion vector of a current block by performing motion compensation and pixel group unitmotion compensation on the current block; and
[00126] a bitstream generator configured to generate a bitstream including informationabout a first motion vector and a second motion vector, and motion prediction modeinformation indicating that a motion prediction mode of the current block is abi-directional motion prediction mode,
[00127] wherein a pixel group for the pixel group unit motion compensation includes atleast one pixel,
[00128] wherein the first motion vector is a motion vector indicating, from the currentblock, a first reference block of a first reference picture which corresponds to the currentblock in a current picture, and the second motion vector is a motion vector indicating,from the current block, a second reference block of a second reference picture whichcorresponds to the current block in the current picture,
[00129] wherein a prediction block of the current block is obtained by performingblock-unit motion compensation and pixel group unit motion compensation on thecurrent block by using a gradient value in the horizontal direction or the vertical directionof a first corresponding reference pixel in the first reference block which corresponds to acurrent pixel included in a current pixel group in the current block, a gradient value in thehorizontal direction or the vertical direction of a second corresponding reference pixel inthe second reference block which corresponds to the current pixel, a pixel value of thefirst corresponding reference pixel, a pixel value of the second corresponding referencepixel, and a displacement vector per unit time of the current pixel in the horizontaldirection or the vertical direction, and
[00130] wherein a displacement vector per unit time of pixels of the current block in thehorizontal direction or the vertical direction, the pixels including a pixel being adjacent toan inside of a boundary of the current block, is determined by using values aboutreference pixels included in the first reference block and the second reference block,without using a stored value about a pixel located outside boundaries of the firstreference block and the second reference block.
[00131] According to various embodiments, a computer-readable recording mediummay include a recorded program for executing the methods.MODE OF DISCLOSURE
[00132] Hereinafter, an "image" may indicate a still image of a video or may indicate amoving picture that is the video itself.
[00133] Hereinafter, a "sample" may refer to data that is allocated to a samplinglocation of an image and is a processing target. For example, pixels in an image of aspatial domain may be samples.
[00134] Hereinafter, a "current block" may refer to a block of an image which is to beencoded or decoded
[00135] FIG. 1A illustrates a block diagram of a video decoding apparatus, according tovarious embodiments.
[00136] A video decoding apparatus 100 according to various embodiments includesan obtainer 105, an inter predictor 110, and a re-constructor 125.
[00137] The obtainer 105 receives a bitstream including prediction mode informationabout a current block, motion prediction mode information about the current block, andmotion vector information.
[00138] The obtainer 105 may obtain, from the received bitstream, the prediction modeinformation about the current block, the motion prediction mode information about thecurrent block, and the motion vector information. Also, the obtainer 105 may obtain, fromthe received bitstream, a reference picture index indicating a reference picture fromamong pictures that have been previously decoded.
[00139] When a prediction mode of a current block is an inter prediction mode, the interpredictor 110 performs inter prediction on the current block. That is, the inter predictor110 may generate a predicted pixel value of the current block by using at least one of thepictures that have been decoded prior to a current picture including the current block. Forexample, when a motion prediction mode of the current block is a bi-directional motionprediction mode, the inter predictor 110 may generate the predicted pixel value of thecurrent block by using two pictures that have been decoded prior to the current picture.That is, when the motion prediction mode information received from the bitstreamindicates the bi-directional motion prediction mode, the inter predictor 110 may generatethe predicted pixel value of the current block by using two pictures that have beendecoded prior to the current picture.
[00140] The inter predictor 110 may include a block-unit motion compensator 115 and apixel group unit motion compensator 120.
[00141] The block-unit motion compensator 115 may perform block-unit motioncompensation on the current block.
[00142] The block-unit motion compensator 115 may determine at least one referencepicture from among the pictures that have been previously decoded, by using thereference picture index obtained from the bitstream. In this regard, the reference pictureindex may refer to reference picture indexes with respect to respective predictiondirections including an L0 direction and an L1 direction. Here, the reference picture indexwith respect to the L0 direction refers to an index indicating a reference picture fromamong pictures included in an L0 reference picture list, and the reference picture indexwith respect to the L1 direction refers to an index indicating a reference picture fromamong pictures included in an L1 reference picture list.
[00143] The block-unit motion compensator 115 may determine a reference block ofthe current block, which is located in at least one reference picture, by using the motionvector information received from the bitstream. In this regard, a corresponding block inthe reference picture, which corresponds to the current block in the current picture, maybe the reference block. That is, the block-unit motion compensator 115 may determinethe reference block of the current block by using a motion vector indicating the referenceblock from the current block. In this regard, the motion vector means a vector indicatingvariation between reference coordinates of the current block in the current picture andreference coordinates of the reference block in the reference picture. For example, whenupper-left coordinates of the current block are (1,1), and upper-left coordinates of thereference block in the reference picture are (3,3), a motion vector may be (2,2).
[00144] In this regard, the motion vector information may include a difference value ofthe motion vector, and the block-unit motion compensator 115 may reconstruct themotion vector by using a predictor of the motion vector and the difference value of themotion vector, which is obtained from the bitstream, and may determine, by using thereconstructed motion vector, the reference block of the current block which is located inat least one reference picture. The difference value of the motion vector may indicate thedifference value of the motion vector which is related to respective prediction directionsincluding the L0 direction and the L1 direction. In this regard, the difference value of themotion vector with respect to the L0 direction refers to a difference value of a motionvector indicating a reference block in the reference picture included in the L0 referencepicture list, and the difference value of the motion vector with respect to the L1 directionrefers to a difference value of a motion vector indicating a reference block in thereference picture included in the L1 reference picture list.
[00145] The block-unit motion compensator 115 may perform block-unit motioncompensation on the current block by using a pixel value of the reference block. Theblock-unit motion compensator 115 may perform block-unit motion compensation on thecurrent block by using a value of a reference pixel in the reference block whichcorresponds to a current pixel in the current block. In this regard, the reference pixel maybe a pixel included in the reference block and may correspond to the current pixel in thecurrent block.
[00146] The block-unit motion compensator 115 may perform block-unit motioncompensation on the current block by using a plurality of reference blocks included in aplurality of reference pictures, respectively. For example, when the motion predictionmode of the current block is the bi-directional motion prediction mode, the block-unitmotion compensator 115 may determine two reference pictures from among the picturesthat have been previously encoded, and may determine two reference blocks included inthe two reference pictures.
[00147] The block-unit motion compensator 115 may perform block-unit motioncompensation on the current block by using pixel values of two reference pixels in thetwo reference blocks. The block-unit motion compensator 115 may generate a block-unitmotion-compensated value by performing block-unit motion compensation on the currentblock by using an average value or a weighted sum with respect to the pixel values of thetwo reference pixels.
[00148] A reference location of the reference block may be a location of an integer pixelbut is not limited thereto and thus may be a location of a fractional pixel. In this regard,the integer pixel may refer to a pixel of which location component is an integer and that isat an integer pixel location. The fractional pixel may refer to a pixel in which a locationcomponent is a faction and that is at a fractional pixel location.
[00149] For example, when the upper-left coordinates of the current block are (1,1),and the motion vector is (2.5,2.5), upper-left coordinates of the reference block in thereference picture may be (3.5,3.5). In this regard, a location of the fractional pixel may bedetermined in a 1 / 4 or 1 / 16 pixel element (pel) unit. However, the disclosure is not limitedthereto, and thus the location of the fractional pixel may be determined according tovarious fraction pel units.
[00150] When the reference location of the reference block is the location of thefractional pixel, the block-unit motion compensator 115 may apply an interpolation filteron a first neighboring region including a first pixel from among pixels of a first referenceblock indicated by a first motion vector and a second neighboring region including asecond pixel from among pixels of a second reference block indicated by a secondmotion vector, and thus may generate a pixel value of the first pixel and a pixel value ofthe second pixel.
[00151] That is, a value of the reference pixel in the reference block may be determinedby using pixel values of neighboring pixels in which predetermined directionalcomponents are integers. In this regard, a predetermined direction may be a horizontaldirection or a vertical direction.
[00152] For example, the block-unit motion compensator 115 may perform, by using aninterpolation filter, filtering on pixel values of pixels in which predetermined directionalcomponents are integers, may determine a reference pixel value as a result of thefiltering, and may determine the block-unit motion-compensated value with respect to thecurrent block by using the reference pixel value The block-unit motion-compensatedvalue may be determined by using an average value or a value of a weighted sum withrespect to reference pixels. In this regard, the interpolation filter may be an M-tapinterpolation filter based on discrete cosine transform (DCT). A coefficient of the M-tapinterpolation filter based on DCT may be derived from DCT and inverse discrete cosinetransform (IDCT). In this regard, a coefficient of an interpolation filter may be a filtercoefficient that is scaled to an integer coefficient to decrease real number operationswhile filtering is performed. In this regard, the interpolation filter may be aone-dimensional (1D) interpolation filter in a horizontal direction or a vertical direction.For example, when a location of a pixel is expressed as x and y perpendicular coordinatecomponents, a horizontal direction may refer to a direction that is in parallel with an xaxis. A vertical direction may refer to a direction that is in parallel with a y axis.
[00153] The block-unit motion compensator 115 may first perform filtering on values ofpixels at integer locations by using the 1D vertical direction interpolation filter, and maydetermine the reference pixel value at the location of the fractional pixel by performingfiltering on a value generated via the filtering, by using the 1D horizontal directioninterpolation filter.
[00154] When a scaled filter coefficient is used, the value generated via the filteringmay be greater than a case in which a not-scaled filter coefficient is used. Therefore, theblock-unit motion compensator 115 may perform de-scaling on the value generated viathe filtering.
[00155] The block-unit motion compensator 115 may perform filtering on the values ofthe pixels at integer locations by using the 1D vertical direction interpolation filter, andthen may perform de-scaling. In this regard, the de-scaling may include bit-shifting to theright by the number of de-scaling bits. The number of de-scaling bits may be determinedbased on a bit depth of a sample of an input image. For example, the number ofde-scaling bits may be a value obtained by subtracting 8 from the bit depth of thesample.
[00156] Also, the block-unit motion compensator 115 may perform filtering on thevalues of the pixels at integer locations by using the 1D vertical direction interpolationfilter, may perform filtering on the value generated via the filtering by using the 1Dhorizontal direction interpolation filter, and then may perform the de-scaling. In thisregard, the de-scaling may include bit-shifting to the right by the number of de-scalingbits. The number of de-scaling bits may be determined based on a scaling bit number ofthe 1D vertical direction interpolation filter, a scaling bit number of the 1D horizontaldirection interpolation filter, and the bit depth of the sample. For example, when a scalingbit number p of the 1D vertical direction interpolation filter is 6, a scaling bit number q ofthe 1D horizontal direction interpolation filter is 6, and the bit depth of the sample is b, thenumber of de-scaling bits may be p+q+8-b, i.e., 20-b.
[00157] When the block-unit motion compensator 115 performs filtering on a pixel inwhich a predetermined directional component is an integer by using the ID interpolationfilter, and then performs bit-shifting to the right by the number of de-scaling bits, around-off error may occur. Therefore, the block-unit motion compensator 115 mayperform filtering on the pixel in which a predetermined directional component is aninteger by using the ID interpolation filter, may add an offset value, and then mayperform de-scaling. In this regard, the offset value may be 2^(number of de-scaling bits -1).
[00158] The pixel group unit motion compensator 120 may generate a pixel group unitmotion-compensated value by performing pixel group unit motion compensation on thecurrent block. When the motion prediction mode of the current block is the bi-directionalmotion prediction mode, the pixel group unit motion compensator 120 may generate thepixel group unit motion-compensated value by performing pixel group unit motioncompensation on the current block.
[00159] The pixel group unit motion compensator 120 may generate the pixel groupunit motion-compensated value by performing pixel group unit motion compensation onthe current block, based on an optical flow of pixel groups of a first reference picture anda second reference picture. The optical flow will be described below in descriptions withreference to FIG. 3A.
[00160] The pixel group unit motion compensator 120 may generate the pixel groupunit motion-compensated value by performing pixel group unit motion compensation onpixel groups included in the reference block of the current block. A pixel group mayinclude at least one pixel. For example, the pixel group may be a pixel. Alternatively, thepixel group may be a plurality of pixels including two or more pixels. The pixel group maybe a plurality of pixels included in a block with a size of KxK (where K is an integer).
[00161] The pixel group unit motion compensator 120 may determine the pixel group,and may perform pixel group unit motion compensation on the current block, based onthe determined pixel group.00162] The pixel group unit motion compensator 120 may determine a size of the pixelgroup, based on a size of the current block. For example, the pixel group unit motioncompensator 120 may determine a maximum value to be a height and a width of thepixel group, the maximum value being selected from among 2 and a value obtained bydividing a minimum value from among a height and a width of the current block by 8.
[00163] The pixel group unit motion compensator 120 may perform motioncompensation in a pixel group unit including a plurality of pixels and thus may decreasecomplexity in encoding / decoding, compared to a case in which motion compensation ina pixel unit is performed in a high image resolution. Also, the pixel group unit motioncompensator 120 may perform motion compensation in a pixel group unit including aplurality of pixels and thus may decrease complexity in encoding / decoding, compared toa case in which motion compensation in a pixel unit is performed at a high frame rate.
[00164] The obtainer 105 may obtain pixel group size information included in thebitstream. When a size of the pixel group is KxK, the pixel group size information may beinformation indicating a height or a width K. The pixel group size information may beincluded in a high level syntax carrier.
[00165] The pixel group unit motion compensator 120 may determine at least one pixelgroup partition including pixels having similar pixel values from among the plurality ofpixels included in the pixel group, and may perform motion compensation on pixel grouppartitions. In this regard, there is a high probability that the pixel group partition includingthe pixels having the similar pixel values is a same object and motions are similar toeach other, and thus, the pixel group unit motion compensator 120 may perform furtherdetailed motion compensation in a pixel group unit.
[00166] The pixel group unit motion compensation is performed when the motionprediction mode information indicates the bi-directional motion prediction mode, but evenin this case, the pixel group unit motion compensation may not always performed butmay be selectively performed.
[00167] The pixel group unit motion compensator 120 may determine a reference pixelgroup in the reference block which corresponds to a current pixel group of the currentblock, and may determine a gradient value of the reference pixel group. For example, thepixel group unit motion compensator 120 may determine the gradient value of thereference pixel group by using a gradient value of at least one pixel value included in thereference pixel group.
[00168] The pixel group unit motion compensator 120 may generate the pixel groupunit motion-compensated value by performing pixel group unit motion compensation onthe current block by using the gradient value of the reference pixel group.
[00169] The pixel group unit motion compensator 120 may apply a filter to a firstneighboring region of a first pixel group which includes the first pixel group from amongpixel groups indicated by the first motion vector and a second neighboring region of asecond pixel group which includes the second pixel group from among pixel groupsindicated by the second motion vector, and thus may generate a gradient value of thefirst pixel group and a gradient value of the second pixel group.
[00170] The pixel group unit motion compensator 120 may determine pixel values andgradient values of pixels in a predetermined-size first window including the first pixelgroup with respect to the first pixel group in the first reference picture, and maydetermine pixel values and gradient values of pixels in a predetermined-size secondwindow including the second pixel group with respect to the second pixel group in thesecond reference picture.
[00171] The pixel group unit motion compensator 120 may determine a displacementvector per unit time with respect to the current pixel group by using the pixel values andthe gradient values of the pixels in the first window and the pixel values and the gradientvalues of the pixels in the second window. In this regard, a value of the displacementvector per unit time with respect to the current pixel group may be adjusted by aregularization parameter. The regularization parameter is a parameter used to preventan error from occurring when an ill-posed displacement vector per unit time with respectto the current pixel group is determined to perform motion compensation on a pixel group.The pixel group unit motion compensator 120 may perform pixel group unit motioncompensation on the current block, based on a regularization parameter about adisplacement vector per unit time in a horizontal direction or vertical direction. Theregularization parameter will be described below in descriptions with reference to FIG.8A.
[00172] The pixel group unit motion compensator 120 may perform pixel group unitmotion compensation on the current block by using the displacement vector per unit timewith respect to the current pixel group and the gradient value of the reference pixel.
[00173] The reference location of the reference block may be the location of the integerpixel but is not limited thereto and thus may be the location of the fractional pixel.
[00174] When the reference location of the reference block is the location of thefractional pixel, the gradient value of the reference pixel in the reference block may bedetermined by using pixel values of neighboring pixels in which predetermineddirectional components are integers.
[00175] For example, the pixel group unit motion compensator 120 may performfiltering, by using a gradient filter, on the pixel values of the neighboring pixels in whichpredetermined directional components are integers, and may determine the gradientvalue of the reference pixel as a resultant value thereof. In this regard, a filter coefficientof the gradient filter may be determined by using a coefficient predetermined withrespect to a DCT-based interpolation filter. A coefficient of the gradient filter may be thefilter coefficient that is scaled to an integer coefficient to decrease real numberoperations while filtering is performed.
[00176] In this regard, the gradient filter may be a 1D horizontal direction gradient filteror a 1D vertical direction gradient filter.
[00177] The pixel group unit motion compensator 120 may perform, by using the 1Dhorizontal direction gradient filter or the 1D vertical direction gradient filter, filtering on aneighboring pixel in which a corresponding directional component is an integer so as todetermine a gradient value in a horizontal direction or a vertical direction with respect tothe reference pixel.
[00178] For example, the pixel group unit motion compensator 120 may perform, byusing the 1D horizontal direction gradient filter, filtering on a pixel located in a horizontaldirection from a pixel in which a horizontal direction component is an integer from amongpixels located adjacent to the reference pixel, and thus may determine the gradient valuein a horizontal direction with respect to the reference pixel.
[00179] When a location of the reference pixel is (x+α, y+β) (where, x and y areintegers and α and β are fractions), the pixel group unit motion compensator 120 mayperform, by using the 1D vertical direction interpolation filter, filtering on a pixel at a (x, y)location and a pixel in which a vertical component is an integer and is from among pixelslocated in a vertical direction from the pixel at the (x, y) location, and may determine apixel value of (x, y+β) as a resultant value thereof.
[00180] The pixel group unit motion compensator 120 may perform, by using thehorizontal direction gradient filter, filtering on a value of a pixel at a (x, y+β) location andpixels in which horizontal components are integers and are from among pixels located ina horizontal direction from the (x, y+β) location, and may determine a gradient value at a(x+α, y+β) location as a resultant value thereof.
[00181] An order of using the ID gradient filter and the ID interpolation filter is notlimited. In the above descriptions, first, an interpolated filtering value in a verticaldirection is generated by performing filtering on a pixel at an integer location by using 1Dvertical direction interpolation filter, and then filtering is performed on the interpolatedfiltering value in a vertical direction by using a 1D horizontal direction gradient filter.However, first, an interpolated filtering value in a horizontal direction may be generatedby performing on the pixel at the integer location by using the 1D horizontal directiongradient filter, and then filtering may be performed on the interpolated filtering value in ahorizontal direction by using the 1D vertical direction interpolation filter.
[00182] In the above descriptions, a method, performed by the pixel group unit motioncompensator 120, of determining the gradient value in a horizontal direction at the (x+α,y+β) location has been particularly described. The pixel group unit motion compensator120 determines a gradient value in a vertical direction at the (x+α, y+β) location in asimilar manner to that of determining the gradient value in a horizontal direction, andthus detailed descriptions thereof are omitted.
[00183] In the above descriptions, a method, performed by the pixel group unit motioncompensator 230, of using the 1D gradient filter and the 1D interpolation filter so as todetermine a gradient value at the location of the fractional pixel has been particularlydescribed. However, the disclosure is not limited thereto, and the gradient filter and theinterpolation filter may also be used to determine a gradient value at the location of theinteger pixel. Even when a pixel value of the integer pixel may be determined withoutusing the interpolation filter, for processes uniform with respect to processes in thefractional pixel, the pixel value of the integer pixel may be determined by performing, byusing the interpolation filter, filtering on the integer pixel and a neighboring pixel in whicha predetermined directional component is an integer. For example, the interpolation filtercoefficient in the integer pixel may be {0, 0, 64, 0, 0}, and because an interpolation filtercoefficient related to a neighboring integer pixel is 0, filtering may be performed by usingonly a pixel value of a current integer pixel, and as a result, filtering may be performed onthe current integer pixel and the neighboring integer pixel by using an interpolation filterso as to determine the pixel value of the current integer pixel.
[00184] The pixel group unit motion compensator 120 may perform de-scaling afterperforming filtering on a pixel at an integer location by using the 1D vertical directioninterpolation filter. In this regard, the de-scaling may include bit-shifting to the right by thenumber of de-scaling bits. The number of de-scaling bits may be determined based on abit depth of a sample. Also, the number of de-scaling bits may be determined based onspecific input data in a block.
[00185] For example, the number of de-scaling bits may be a value obtained bysubtracting 8 from the bit depth of the sample.
[00186] The pixel group unit motion compensator 120 may perform de-scaling afterperforming filtering on a value generated via the de-scaling using the horizontal directiongradient filter. Equally, the de-scaling may include bit-shifting to the right by the numberof de-scaling bits. The number of de-scaling bits may be determined based on ascaled-bit number of the 1D vertical direction interpolation filter, a scaled-bit number ofthe 1D horizontal direction gradient filter, and the bit depth of the sample. For example,when a scaled-bit number p of the 1D interpolation filter is 6, a scaled-bit number q of the1D gradient filter is 4, and the bit depth of the sample is b, the number of de-scaling bitsmay be p+q+8-b, i.e., 18 - b.
[00187] When the pixel group unit motion compensator 120 performs only bit-shifting tothe right by the number of de-scaling bits on a value generated via filtering afterperforming the filtering, a round-off error may occur, such that the pixel group unit motioncompensator 120 may add an offset value to the value generated via the filtering andthen may perform the de-scaling. Here, the offset value may be 2^(number of de-scalingbits - 1).
[00188] The inter predictor 110 may generate a predicted pixel value of the currentblock by using the block-unit motion-compensated value and the pixel group unitmotion-compensated value with respect to the current block. For example, the interpredictor 110 may generate a predicted pixel value of the current block by adding theblock-unit motion-compensated value to the pixel group unit motion-compensated valuewith respect to the current block. In this regard, the block-unit motion-compensated valuerefers to a value generated by performing the block-unit motion compensation, and thepixel group unit motion-compensated value refers to a value generated by performingthe pixel group unit motion compensation, wherein the block-unit motion-compensatedvalue may be an average value or a weighted sum with respect to reference pixels, andthe pixel group unit motion-compensated value may be a value determined based on thedisplacement vector per unit time with respect to the current pixel and the gradient valueof the reference pixel.
[00189] The pixel group unit motion compensator 120 may obtain a shift value for thede-scaling after performing interpolation calculation or gradient calculation based on atleast one of the bit depth of the sample, an input range of a filter used in the interpolationcalculation or the gradient calculation, and a coefficient of the filter. The pixel group unitmotion compensator 120 may perform the interpolation calculation or the gradientcalculation with respect to pixels included in the first reference block and the secondreference block by using the shift value for the de-scaling, and then may perform thede-scaling.
[00190] The inter predictor 110 may use a motion vector while performing the block-unitmotion compensation, and may store the motion vector. In this regard, a unit of themotion vector may be a block of a 4x4 size. When the motion vector is stored after theblock-unit motion compensation is performed, a motion vector storage unit may not bethe 4x4 size but may be a block having various sizes (e.g., a block of an RxR size; whereR is an integer). In this regard, the motion vector storage unit may be a block larger thanthe 4x4 size. For example, it may be a block of a 16x16 size.
[00191] When the pixel group unit motion compensation is performed, a size of a targetblock on which the pixel group unit motion compensation is to be performed may beenlarged, based on a size of the current block, a size of a window, and a length of theinterpolation filter. The reason why the size of the target block is enlarged based on thesize of the window, compared to the size of the current block, is because, for a pixellocated at an edge of the current block, the pixel group unit motion compensation isperformed on the current block based on the pixel located at the edge of the currentblock and a neighboring pixel by using the window.
[00192] Therefore, to decrease the number of memory accesses and decreaseperforming of a multiplication operation, while the pixel group unit motion compensationis performed by using the window, the pixel group unit motion compensator 120 mayadjust a location of a pixel deviating from the current block from among pixels in thewindow to a location of a pixel adjacent to the inside of the current block, and thus maydetermine a pixel value and a gradient value at the adjusted location of the pixel, suchthat the number of memory accesses and the number of multiplication operations maybe decreased.
[00193] The pixel group unit motion compensator 120 may not use the pixel values ofthe pixels at the integer locations so as to determine gradient values of the referencepixels which are necessary values for the pixel group unit motion compensation. That is,the pixel group unit motion compensator 120 may calculate a gradient value of a firstcorresponding reference pixel in the first reference block in a horizontal direction or avertical direction or a second corresponding reference pixel in the second referenceblock by applying the horizontal direction gradient filter or the vertical direction gradientfilter to pixel values of pixels at fractional locations. In this regard, a length of the gradientfilter may be 5. A coefficient of the gradient filter may have coefficients that aresymmetrical with respect to a center coefficient of the filter. A pixel at a fractional locationmay be a pixel in which at least one of a horizontal direction component and a verticaldirection component which indicate a location of the pixel has a fractional value.
[00194] Pixel values of pixels at fractional locations in the first reference block or thesecond reference block may have been calculated by applying the horizontal directioninterpolation filter or the vertical direction interpolation filter to pixel values of pixels atinteger locations.
[00195] A displacement vector per unit time in a horizontal direction or a verticaldirection with respect to a current pixel group may be a displacement vector per unit timedetermined by using pixel values of first corresponding reference pixels in a firstcorresponding reference pixel group in the first reference picture, the first correspondingreference pixel group corresponding to the current pixel group, pixel values of secondcorresponding reference pixels in a second corresponding reference pixel group in thesecond reference picture and neighboring pixels of the first corresponding referencepixels and the second corresponding reference pixels, gradient values, a first pictureorder count (POC) difference between the first reference picture and the current picture,and a second POC difference between the second reference picture and the currentpicture.
[00196] When the first corresponding reference pixel or the second correspondingreference pixel is a boundary neighboring pixel adjacent to the inside of a boundary ofthe first reference block or the second reference block, the pixel group unit motioncompensator 120 may derive, by using a pixel value of the boundary neighboring pixel, apixel value of a pixel located outside the boundary of the first reference block or thesecond reference block.
[00197] The pixel group unit motion compensator 120 may determine a displacementvector per unit time of the current block in a horizontal direction or a vertical direction,based on the pixel value of the boundary neighboring pixel and the pixel value of thepixel located outside the boundary of the current block, the pixel value of the pixel beingderived by using the pixel value of the boundary neighboring pixel. That is, the pixellocated outside the boundary may be present from among pixels included in a windowwith respect to the boundary neighboring pixel, and in this regard, the pixel value and agradient value of the pixel located outside the boundary may be a pixel value and agradient value of the pixel which are not included in a memory but are derived from theboundary neighboring pixel.
[00198] The pixel group unit motion compensator 120 may calculate a value about thecurrent pixel by using a pixel value of the first corresponding reference pixel included inthe first reference block, a pixel value of the second corresponding reference pixelincluded in the second reference block, a gradient value of the first correspondingreference pixel, a gradient value of the second corresponding reference pixel, a firstPOC difference between the first reference picture and the current picture, and a secondPOC difference between the second reference picture and the current picture. That is,the value about the current pixel may be a resultant value of a function based on a pixelvalue of a corresponding reference pixel of each reference picture, a gradient value, anda POC difference between each reference picture and the current picture.
[00199] The pixel group unit motion compensator 120 may calculate values aboutneighboring pixels by using pixel values of first corresponding neighboring pixels of thefirst corresponding reference pixel, gradient values of the first corresponding neighboringpixels, pixel values of second corresponding neighboring pixels of the secondcorresponding reference pixel, gradient values of the second corresponding neighboringpixels, the first POC difference between the first reference picture and the current picture,and the second POC difference between the second reference picture and the currentpicture. That is, the value about the neighboring pixel may be a resultant value of afunction based on the pixel value of the corresponding reference pixel of each referencepicture, the gradient value the corresponding reference pixel of each reference picture,and the POC difference between each reference picture and the current picture. That is,the value about the neighboring pixel may be a resultant value of a function based on apixel value of a corresponding neighboring pixel of each reference picture, a gradientvalue, and the POC difference between each reference picture and the current picture.
[00200] The pixel group unit motion compensator 120 may calculate a weightedaverage value for the current pixel which is required to calculate a displacement vectorper unit time in a horizontal direction, by using the value about the current pixel, thevalues about the neighboring pixels, and a weight. In this regard, the weight may bedetermined based on a distance between the current pixel and the neighboring pixel, adistance between a pixel and a boundary of a block, the number of pixels located outsidethe boundary, or whether the pixel is located in the inside of the boundary or outside theboundary.
[00201] The weighted average value for the current pixel may be a value that iscalculated by applying an exponential smoothing technique in vertical and horizontaldirections to values about the pixels included in the first reference block and the secondreference block. The value that has been calculated with respect to the current pixel byapplying the exponential smoothing technique in vertical and horizontal directions to thevalues about the pixels may be a value of which weight with respect to the value of thecurrent pixel is largest and weight with respect to a value of a neighboring pixel thereof isexponentially decreased according to distances from the current pixel.
[00202] The pixel group unit motion compensator 120 may determine the displacementvector per unit time of the current pixel in a horizontal direction or a vertical direction byusing the weighted average value of the current pixel.
[00203] The reconstructor 125 may obtain a residual block of the current block form abitstream, and may reconstruct the current block by using the residual block and aprediction block of the current block. For example, the reconstructor 125 may generate apixel value of a reconstructed block by summing a pixel value of the residual block of thecurrent block from the bitstream and a pixel value of the prediction block of the currentblock.
[00204] The video decoding apparatus 100 may include an image decoder (not shown),and the image decoder (not shown) may include the obtainer 105, the inter predictor 110,and the reconstructor 125. The image decoder will be described with reference to FIG.1E.
[00205] FIG. 1B illustrates a flowchart of a video decoding method according to variousembodiments.
[00206] In operation S105, the video decoding apparatus 100 may obtain, from abitstream, motion prediction mode information about a current block in a current picture.The video decoding apparatus 100 may receive the bitstream including the motionprediction mode information about the current block in the current picture, and mayobtain the motion prediction mode information about the current block from the receivedbitstream. The video decoding apparatus 100 may obtain the motion prediction modeinformation about the current block from the bitstream, and may determine a predictionmode of the current block based on the motion prediction mode information. In thisregard, when the prediction mode of the current block is an inter prediction mode, thevideo decoding apparatus 100 may obtain the motion prediction mode information aboutthe current block.
[00207] For example, the video decoding apparatus 100 may determine the predictionmode of the current block as the inter prediction mode, based on the motion predictionmode information about the current block. When the prediction mode of the current blockis an inter prediction mode, the video decoding apparatus 100 may obtain the motionprediction mode information about the current block from the bitstream.
[00208] In operation S110, when the motion prediction mode information indicates abi-directional motion prediction mode, the video decoding apparatus 100 may obtain,from the bitstream, a first motion vector indicating a first reference block of the currentblock in a first reference picture and a second motion vector indicating a secondreference block of the current block in a second reference picture.
[00209] That is, the video decoding apparatus 100 may receive the bitstream includinginformation about the first motion vector and the second motion vector, and may obtainthe first motion vector and the second motion vector from the received bitstream. Thevideo decoding apparatus 100 may obtain a reference picture index from the bitstream,and may determine, based on the reference picture index, the first reference picture andthe second reference picture from among a plurality of pictures that have beenpreviously decoded.
[00210] In operation S115, the video decoding apparatus 100 may determine adisplacement vector of pixels of the current block in a horizontal direction or a verticaldirection, the pixels including a pixel adjacent to the inside of a boundary of the currentblock, by using values about reference pixels included in the first reference block and thesecond reference block, without using a stored value about a pixel located outsideboundaries of the first reference block and the second reference block. In this regard, thestored value about the pixel located outside boundaries of the first reference block andthe second reference block, and the values about the reference pixels included in thefirst reference block and the second reference block may be a pixel value of a relatedpixel or a gradient value in a horizontal direction or a gradient value in a vertical directionof the related pixel. Alternatively, the stored value about the pixel located outsideboundaries of the first reference block and the second reference block, and the valuesabout the reference pixels included in the first reference block and the second referenceblock may be a value determined by using the pixel value of the related pixel or agradient value of the related pixel.
[00211] In operation S120, the video decoding apparatus 100 may obtain the predictionblock of the current block by performing block-unit motion compensation and pixel groupunit motion compensation on the current block by using a gradient value in a horizontaldirection or a vertical direction of a first corresponding reference pixel in the firstreference block which corresponds to a current pixel included in a current pixel group inthe current block, a gradient value in a horizontal direction or a vertical direction of asecond corresponding reference pixel in the second reference block which correspondsto the current pixel, a pixel value of the first corresponding reference pixel, a pixel valueof the second corresponding reference pixel, and a displacement vector of the currentpixel in a horizontal direction or vertical direction.
[00212] That is, the video decoding apparatus 100 may generate the prediction block ofthe current block by performing the block-unit motion compensation and the pixel groupunit motion compensation on the current block, based on the first motion vector and thesecond motion vector. The video decoding apparatus 100 may perform the block-unitmotion compensation on the current block by using pixel values of the first referenceblock indicated by first motion vector and pixel values of the second reference blockindicated by the second motion vector. Also, the video decoding apparatus 100 mayperform pixel group unit motion compensation on the current pixel group by using agradient value in a horizontal direction or a vertical direction of at least one firstcorresponding reference pixel in the first reference block corresponding to at least onepixel included in the current pixel group in the current block, a gradient value in ahorizontal direction or a vertical direction of at least one second corresponding referencepixel in the second reference block corresponding to the at least one pixel, the pixelvalue of the first corresponding reference pixel, the pixel value of the secondcorresponding reference pixel, and the displacement vector of the current pixel in ahorizontal direction or a vertical direction.
[00213] The video decoding apparatus 100 may obtain the prediction block of thecurrent block by using a block-unit motion-compensated value generated by performingthe block-unit motion compensation on the current block, and a pixel group unitmotion-compensated value generated by performing the pixel group unit motioncompensation on the current pixel group.
[00214] In operation S125, the video decoding apparatus 100 may obtain, from thebitstream, information about a residual block of the current block.
[00215] In operation S130, the video decoding apparatus 100 may reconstruct thecurrent block based on the prediction block and the residual block. That is, the videodecoding apparatus 100 may generate a pixel value of a reconstructed block of thecurrent block by summing a pixel value of the residual block which is indicated by theinformation about the residual block of the current block and a predicted pixel value ofthe prediction block.
[00216] FIG. 1C illustrates a block diagram of a video encoding apparatus, according tovarious embodiments.
[00217] A video encoding apparatus 150 according to various embodiments includesan inter predictor 155 and a bitstream generator 170.
[00218] The inter predictor 155 performs inter prediction on a current block by referringto various blocks based on rate and distortion costs. That is, the inter predictor 155 maygenerate a predicted pixel value of the current block by using at least one of pictures thathave been encoded prior to a current picture including the current block.
[00219] The inter predictor 155 may include a block-unit motion compensator 160 and apixel group unit motion compensator 165.
[00220] The block-unit motion compensator 160 may generate a block-unitmotion-compensated value by performing block-unit motion compensation on the currentblock.
[00221] The block-unit motion compensator 160 may determine at least one referencepicture from among picture that have been previously decoded, and may determine areference block of the current block which is located in the at least one reference picture.
[00222] The block-unit motion compensator 160 may generate the block-unitmotion-compensated value by performing the block-unit motion compensation on thecurrent block by using a pixel value of the reference block. The block-unit motioncompensator 160 may generate the block-unit motion-compensated value by performingthe block-unit motion compensation on the current block by using a value of a referencepixel of the reference block, the reference pixel corresponding to a current pixel of thecurrent block.
[00223] The block-unit motion compensator 160 may generate the block-unitmotion-compensated value by performing the block-unit motion compensation on thecurrent block by using a plurality of reference blocks respectively included in a plurality ofreference pictures. For example, when a motion prediction mode of the current blockindicates a bi-directional prediction mode, the block-unit motion compensator 160 maydetermine two reference pictures from among the previously-encoded pictures, and maydetermine two reference blocks included in the two reference pictures. In this regard,bi-directional prediction may not be limited to inter prediction using a picture of whichdisplay order is prior to the current block and a picture of which display order is after thecurrent block, and may indicate inter prediction using two pictures that have beenencoded prior to the current picture, regardless of a display order.
[00224] The block-unit motion compensator 160 may generate the block-unitmotion-compensated value by performing the block-unit motion compensation on thecurrent block by using values of two reference pixels in the two reference blocks. Theblock-unit motion compensator 160 may generate the block-unit motion-compensatedvalue by performing the block-unit motion compensation on the current block by using anaverage pixel value or a weighted sum of the two reference pixels.
[00225] The block-unit motion compensator 160 may output a reference picture indexindicating a reference picture for motion compensation on the current block, thereference picture being from among the previously-encoded pictures.
[00226] The block-unit motion compensator 160 may determine a motion vector ofwhich start point is the current block and end point is a reference block of the currentblock, and may output the motion vector. The motion vector may mean a vectorindicating displacement between reference coordinates of the current block in thecurrent picture and reference coordinates of the reference block in the reference picture.For example, when upper-left coordinates of the current block are (1,1), and upper-leftcoordinates of the reference block in the reference picture are (3,3), a motion vector maybe (2,2).
[00227] A reference location of the reference block may be a location of an integer pixelbut is not limited thereto and thus may be a location of a fractional pixel. In this regard, alocation of the fractional pixel may be determined in a 1 / 4 pel unit or a 1 / 16 pel unit.However, the disclosure is not limited thereto, and thus the location of the fractional pixelmay be determined according to various fraction pel units.
[00228] For example, when the reference location of the reference block is (1.5, 1.5),and coordinates of an upper-left corner of the current block are (1,1), the motion vectormay be (0.5,0.5). When the motion vector is determined in a 1 / 4 or 1 / 16 pel unit toindicate the reference location of the reference block, which is a location of a fractionalpixel, a motion vector of an integer may be determined by scaling the motion vector, andthe reference location of the reference block may be determined by using the up-scaledmotion vector. When the reference location of the reference block is a location of afractional pixel, a location of the reference pixel of the reference block may also be alocation of a fractional pixel. Accordingly, a pixel value at a fractional pixel location in thereference block may be determined by using pixel values of neighboring pixels in which acomponent in a predetermined direction is an integer.
[00229] For example, the block-unit motion compensator 160 may determine, as thepixel value of the reference pixel at the fractional pixel location, a value obtained byperforming filtering on pixel values of neighboring pixels in which a component in apredetermined direction is an integer, by using an interpolation filter, and may determinethe motion-compensated value in block units with respect to the current block, by usingthe pixel value of the reference pixel. In this regard, the interpolation filter may be aDCT-based M-tap interpolation filter. A coefficient of the DCT-based M-tap interpolationfilter may be derived from DCT and IDCT. In this regard, the coefficient of theinterpolation filter may be a filter coefficient scaled to an integer coefficient so as todecrease real number operations while the filtering is performed.
[00230] In this regard, the interpolation filter may be a 1D interpolation filter in ahorizontal direction or a vertical direction.
[00231] The block-unit motion compensator 160 may determine the pixel value of thereference pixel at the fractional pixel location by first performing filtering on neighboringinteger pixels by using a 1D vertical direction interpolation filter, and then performingfiltering on a value on which the filtering has been performed, by using a 1D horizontaldirection interpolation filter. When a scaled filter coefficient is used, the block-unit motioncompensator 160 may perform de-scaling on a value on which filtering has beenperformed, after performing filtering on a pixel at an integer location by using the 1Dvertical direction interpolation filter. In this regard, the de-scaling may include bit-shiftingto the right by the number of de-scaling bits. The number of de-scaling bits may bedetermined based on a bit depth of a sample. For example, the number of de-scaling bitsmay be a value obtained by subtracting 8 from the bit depth of the sample.
[00232] Also, the block-unit motion compensator 160 may perform filtering on a pixel, inwhich a horizontal direction component is an integer, by using the 1D vertical directioninterpolation filter, and then may perform the bit-shifting to the right by the number ofde-scaling bits. The number of de-scaling bits may be determined based on a scaling bitnumber with respect to a coefficient of the 1D vertical direction interpolation filter, ascaling bit number with respect to a coefficient of the 1D horizontal direction interpolationfilter, and the bit depth of the sample.
[00233] When the block-unit motion compensator 160 only performs bit-shifting to theright by the number of de-scaling bits, a round-off error may occur. Therefore, theblock-unit motion compensator 160 may perform filtering on the pixel in which apredetermined directional component is an integer by using a ID interpolation filter in apredetermined direction, may add an offset value to a value on which the filtering hasbeen performed, and then may perform de-scaling on the value to which the offset valuehas been added. In this regard, the offset value may be 2^(number of de-scaling bits - 1).
[00234] Hereinabove, determining of the number of de-scaling bits based on the bitdepth of the sample after filtering using the 1D vertical direction interpolation filter hasbeen described, but the disclosure is not limited thereto, and thus the number ofde-scaling bits may be determined not only the bit depth of the sample but also based ona number of bits scaled with respect to an interpolation filter coefficient. That is, thenumber of de-scaling bits may be determined based on the bit depth of the sample andthe number of bits scaled with respect to the interpolation filter coefficient, within a rangethat overflow does not occur, while considering a size of a register used during filteringand a size of a buffer storing a value generated during the filtering.
[00235] The pixel group unit motion compensator 165 may generate a pixel group unitmotion-compensated value by performing pixel group unit motion compensation on thecurrent block. In this regard, when a motion prediction mode is a bi-directional motionprediction mode, the pixel group unit motion compensator 165 may generate the pixelgroup unit motion-compensated value by performing the pixel group unit motioncompensation on the current block.
[00236] The pixel group unit motion compensator 165 may generate the pixel groupunit motion-compensated value by performing the pixel group unit motion compensationon the current block by using gradient values of pixels included in a reference block ofthe current block.
[00237] The pixel group unit motion compensator 165 may generate a gradient value ofa first pixel from among pixels of a first reference block in a first reference picture and agradient value of a second pixel from among pixels of a second reference block in asecond reference picture by applying a filter to a first neighboring region of the first pixeland a second neighboring region of the second pixel.
[00238] The pixel group unit motion compensator 165 may determine pixel values andgradient values of pixels in a first window having a predetermined size and including thefirst reference pixel with respect to the first reference pixel in the first reference picture,and may determine pixel values and gradient values of pixels in a second window havinga predetermined size and including the second reference pixel with respect to thesecond reference pixel in the second reference picture. The pixel group unit motioncompensator 165 may determine a displacement vector per unit time with respect to thecurrent pixel by using the pixel values and gradient values of the pixels in the firstwindow and the pixel values and gradient values of the pixels in the second window.
[00239] The pixel group unit motion compensator 165 may generate the pixel groupunit motion-compensated value by performing the pixel group unit motion compensationon the current block by using the displacement vector per unit time and the gradientvalue of the reference pixel.
[00240] A reference location of the reference pixel may be a location of an integer pixelbut is not limited thereto and thus may be a location of a fractional pixel.
[00241] When the reference location of the reference block is the location of thefractional pixel, the gradient value of the reference pixel in the reference block may bedetermined by using pixel values of neighboring pixels in which predetermineddirectional components are integers.
[00242] For example, the pixel group unit motion compensator 165 may determine, asthe gradient value of the reference pixel, a resultant value obtained by performingfiltering on the pixel values of the neighboring pixels, in which a predetermineddirectional component is an integer, by using a gradient filter. In this regard, a filtercoefficient of the gradient filter may be determined by using a coefficient predeterminedwith respect to a DCT-based interpolation filter.
[00243] A coefficient of the gradient filter may be the filter coefficient that is scaled to aninteger coefficient to decrease real number operations while filtering is performed. In thisregard, the gradient filter may be a 1D horizontal direction gradient filter or a 1D verticaldirection gradient filter.
[00244] The pixel group unit motion compensator 165 may perform, by using the 1Dhorizontal direction gradient filter or the 1D vertical direction gradient filter, filtering on aneighboring pixel in which a corresponding directional component is an integer so as todetermine a gradient value in a horizontal direction or a vertical direction with respect tothe reference pixel.
[00245] For example, the pixel group unit motion compensator 165 may determine apixel value of a pixel, in which a vertical component is a fraction, by performing filteringon pixels, in which a vertical component is an integer, from among pixels in a verticaldirection from an integer pixel adjacent to a reference pixel, by using the 1D verticaldirection interpolation filter.
[00246] With respect to a pixel located in another column adjacent to the integer pixeladjacent to the reference pixel, the pixel group unit motion compensator 165 maydetermine a pixel value of a fractional pixel location located in the other column byperforming filtering on a neighboring integer pixel in the vertical direction, by using the1D vertical direction interpolation filter. In this regard, a location of the pixel located in theother column may be a location of a fractional pixel in the vertical direction and a locationof an integer pixel in the horizontal direction.
[00247] That is, when the location of the reference pixel is (x+α, y+β) (where, x and yare integers and α and β are fractions), the pixel group unit motion compensator 165may determine a pixel value at a (x, y+β) location by performing filtering on aneighboring integer pixel in the vertical direction from a (x, y) location by using a verticaldirection interpolation filter.
[00248] The pixel group unit motion compensator 165 may determine a gradient valueat a (x+α, y+β) location in the horizontal direction by performing filtering on the pixelvalue at the (x, y+β) location and a pixel value of a pixel, in which a horizontalcomponent is an integer, from among pixels located in the horizontal direction from thepixel value at the (x, y+β) location, by using a horizontal direction gradient filter.
[00249] An order of using the 1D gradient filter and the 1D interpolation filter is notlimited. As described above, an interpolation filtered value in a vertical direction may befirst generated by performing filtering on a pixel at an integer location by using the 1Dvertical direction interpolation filter, and then filtering may be performed on theinterpolation filtered value in the vertical direction by using the 1D horizontal directiongradient filter, but the disclosure is not limited thereto, and thus a gradient filtered valuein the horizontal direction may be generated first by performing filtering on the pixel atthe integer location by using the 1D horizontal direction gradient filter, and then filteringmay be performed on the gradient filtered value in the horizontal direction by using the1D vertical direction interpolation filter.
[00250] Hereinabove, the pixel group unit motion compensator 165 determining thegradient value in the horizontal direction at the (x+α, y+β) location has been described indetail.
[00251] The pixel group unit motion compensator 165 may determine a gradient valuein a vertical direction at a (x+α, y+β) location in the similar manner as determining of thegradient value in the horizontal direction.
[00252] The pixel group unit motion compensator 165 may determine a gradient valueof a reference pixel in a vertical direction by performing filtering on a neighboring integerpixel in the vertical direction from integer pixels adjacent to the reference pixel, by usingthe 1D vertical direction gradient filter. With respect to a pixel adjacent to the referencepixel and located in another column, the pixel group unit motion compensator 165 maydetermine a gradient value in the vertical direction with respect to the pixel adjacent tothe reference pixel and located in the other column by performing filtering on aneighboring integer pixel in the vertical direction, by using the 1D vertical directiongradient filter. In this regard, a location of the pixel may be a location of a fractional pixelin the vertical direction and a location of an integer pixel in a horizontal direction.
[00253] That is, when a location of a reference pixel is (x+α, y+β) (where, x and y areintegers and α and β are fractions), the pixel group unit motion compensator 165 maydetermine a gradient value in a vertical direction at a (x, y+ β) location by performingfiltering on a neighboring integer pixel in the vertical direction from a (x, y) location, byusing the vertical direction gradient filter.
[00254] The pixel group unit motion compensator 165 may determine a gradient valuein a vertical direction at a (x+α, y+β) location by performing filtering on a gradient value ata (x, y+β) location and a gradient value of a neighboring integer pixel located in ahorizontal direction from the (x, y+β) location, by using the horizontal directioninterpolation filter.
[00255] An order of using the 1D gradient filter and the 1D interpolation filter is notlimited. As described above, a gradient filtered value in a vertical direction may be firstgenerated by performing filtering on pixels at an integer location by using the 1D verticaldirection gradient filter, and then filtering may be performed on the gradient filtered valuein the vertical direction by using the 1D horizontal direction interpolation filter, but thedisclosure is not limited thereto and thus, an interpolation filtered value in the horizontaldirection may be generated first by performing filtering on the pixel at the integer locationby using the 1D horizontal direction interpolation filter, and then filtering may beperformed on the interpolation filtered value in the horizontal direction by using the 1Dvertical direction gradient filter.
[00256] Hereinabove, the pixel group unit motion compensator 165 using a gradientfilter and an interpolation filter so as to determine a gradient value at a fractional pixellocation has been described in detail. However, the disclosure is not limited thereto andthus, a gradient filter and an interpolation filter may be used to determine a gradientvalue at an integer pixel location.
[00257] In a case of an integer pixel, a pixel value thereof may be determined withoutusing an interpolation filter, but filtering may be performed on the integer pixel and aneighboring integer pixel by using the interpolation filter for processes uniform withrespect to processes in a fractional pixel. For example, the interpolation filter coefficientin the integer pixel may be {0, 0, 64, 0, 0}, and because an interpolation filter coefficientrelated to a neighboring integer pixel is 0, filtering may be performed by using only apixel value of a current integer pixel, and as a result, filtering may be performed on thecurrent integer pixel and the neighboring integer pixel by using an interpolation filter soas to determine the pixel value of the current integer pixel.
[00258] When a scaled filter coefficient is used, the pixel group unit motioncompensator 165 may perform filtering on a pixel at an integer location by using the 1Dhorizontal direction gradient filter, and then may perform de-scaling on a value on whichthe filtering has been performed. In this regard, the de-scaling may include bit-shifting tothe right by the number of de-scaling bits. The number of de-scaling bits may bedetermined based on a bit depth of a sample. For example, the number of de-scaling bitsmay be a value obtained by subtracting 8 from the bit depth of the sample.
[00259] The pixel group unit motion compensator 165 may perform filtering on a pixel,in which a vertical directional component is an integer, by using the vertical directioninterpolation filter, and then may perform de-scaling. In this regard, the de-scaling mayinclude bit-shifting to the right by the number of de-scaling bits. The number of de-scalingbits may be determined based on a scaling bit number of the 1D verticaldirection interpolation filter, a scaling bit number of the 1D horizontal direction gradientfilter, and the bit depth of the sample.
[00260] When the pixel group unit motion compensator 165 only performs bit-shifting tothe right by the number of de-scaling bits, a round-off error may occur. Therefore, afterfiltering is performed by using the 1D interpolation filter, an offset value may be added toa value on which the filtering has been performed, and de-scaling may be performed onthe value to which the offset value has been added. In this regard, the offset value maybe 2^(bit shifting number - 1)
[00261] The inter predictor 110 may use a motion vector while performing block-unitmotion compensation, and may store the motion vector. In this regard, a unit of themotion vector may be a block of a 4x4 size. When the motion vector is stored after theblock-unit motion compensation is performed, a motion vector storage unit may not bethe 4x4 size but may be a block having various sizes (e.g., a block of an RxR size; whereR is an integer). In this regard, the motion vector storage unit may be a block larger thanthe 4x4 size. For example, it may be a block of a 16x16 size.
[00262] When the pixel group unit motion compensation is performed, a size of a targetlock on which the pixel group unit motion compensation is to be performed may beenlarged, based on a size of the current block, a size of a window, and a length of theinterpolation filter. The reason why the size of the target block is enlarged based on thesize of the window, compared to the size of the current block, is because, for a pixellocated at an edge of the current block, the pixel group unit motion compensation isperformed on the current block based on the pixel located at the edge of the currentblock and a neighboring pixel by using the window.
[00263] Therefore, to decrease the number of memory accesses and decreaseperforming of a multiplication operation, while the pixel group unit motion compensationis performed by using the window, the pixel group unit motion compensator 120 adjust alocation of a pixel deviating from the current block from among pixels in the window to alocation of a pixel adjacent to the inside of the current block, and thus may determine apixel value and a gradient value at the adjusted location of the pixel, such that thenumber of memory accesses and the number of multiplication operations may bedecreased.
[00264] The pixel group unit motion compensator 120 may not use the pixel values ofthe pixels at the integer locations so as to determine gradient values of the referencepixels which are necessary values for the pixel group unit motion compensation. That is,the pixel group unit motion compensator 120 may calculate a gradient value of a firstcorresponding reference pixel in the first reference block in a horizontal direction or avertical direction or a second corresponding reference pixel in the second referenceblock by applying the horizontal direction gradient filter or the vertical direction gradientfilter to pixel values of pixels at fractional locations. In this regard, a length of the gradientfilter may be 5. A coefficient of the gradient filter may have coefficients that aresymmetrical with respect to a center coefficient of the filter. A pixel at a fractional locationmay be a pixel in which at least one of a horizontal direction component and a verticaldirection component which indicate a location of the pixel has a fractional value.
[00265] Pixel values of pixels at fractional locations in the first reference block or thesecond reference block may have been calculated by applying the horizontal directioninterpolation filter or the vertical direction interpolation filter to pixel values of pixels atinteger locations.
[00266] A displacement vector per unit time in a horizontal direction or a verticaldirection with respect to a current pixel group may be a displacement vector per unit timedetermined by using first corresponding reference pixels in a first correspondingreference pixel group in the first reference picture and second corresponding referencepixels in a second corresponding reference pixel group in the second reference picturewhich correspond to the current pixel group, pixel values of neighboring pixels of the firstand second corresponding reference pixels thereof, gradient values of the first andsecond corresponding reference pixels and the neighboring pixels of the first and secondcorresponding reference pixels, a first POC difference between the first reference pictureand the current picture, and a second POC difference between the second referencepicture and the current picture.
[00267] When the first corresponding reference pixel or the second correspondingreference pixel is a boundary neighboring pixel adjacent to the inside of a boundary ofthe first reference block or the second reference block, the pixel group unit motioncompensator 120 may derive, by using a pixel value of the boundary neighboring pixel, apixel value of a pixel located outside the boundary of the first reference block or thesecond reference block.
[00268] The pixel group unit motion compensator 120 may determine a displacementvector per unit time of the current block in a horizontal direction or a vertical direction,based on the pixel value of the boundary neighboring pixel and the pixel value of thepixel located outside the boundary of the current block, the pixel value of the pixel beingderived by using the pixel value of the boundary neighboring pixel. That is, the pixellocated outside the boundary may be present from among pixels included in a windowwith respect to the boundary neighboring pixel, and in this regard, the pixel value and agradient value of the pixel located outside the boundary may be a pixel value and agradient value of the pixel which are not included in a memory but are derived from theboundary neighboring pixel.
[00269] The pixel group unit motion compensator 120 may calculate a value about thecurrent pixel by using a pixel value of the first corresponding reference pixel included inthe first reference block, a pixel value of the second corresponding reference pixelincluded in the second reference block, a gradient value of the first correspondingreference pixel, a gradient value of the second corresponding reference pixel, a firstPOC difference between the first reference picture and the current picture, and a secondPOC difference between the second reference picture and the current picture. That is,the value about the current pixel may be a resultant value of a function based on a pixelvalue of a corresponding reference pixel of each reference picture, a gradient value, anda POC difference between each reference picture and the current picture.
[00270] The pixel group unit motion compensator 120 may calculate values aboutneighboring pixels by using pixel values of first corresponding neighboring pixels of thefirst corresponding reference pixel, gradient values of the first corresponding neighboringpixels, pixel values of second corresponding neighboring pixels of the secondcorresponding reference pixel, gradient values of the second corresponding neighboringpixels, the first POC difference between the first reference picture and the current picture,and the second POC difference between the second reference picture and the currentpicture. That is, the value about the neighboring pixel may be a resultant value of afunction based on the pixel value of the corresponding reference pixel of each referencepicture, the gradient value, and the POC difference between each reference picture andthe current picture. That is, the value about the corresponding neighboring pixel may bea resultant value of a function based on a pixel value of a corresponding neighboringpixel of each reference picture, a gradient, and the POC difference between eachreference picture and the current picture.
[00271] The pixel group unit motion compensator 120 may calculate a weightedaverage value for the current pixel which is required to calculate a displacement vectorper unit time in a horizontal direction, by using the value about the current pixel, thevalues about the corresponding neighboring pixels, and a weight. In this regard, theweight may be determined based on a distance between the current pixel and theneighboring pixel, a distance between a pixel and a boundary of a block, the number ofpixels located outside the boundary, or whether the pixel is located in the inside of theboundary or outside the boundary.
[00272] The weighted average value for the current pixel may be a value that has beencalculated by applying an exponential smoothing technique in vertical and horizontaldirections to values about the pixels included in the first reference block and the secondreference block. The value that has been calculated with respect to the current pixel byapplying the exponential smoothing technique in vertical and horizontal directions to thevalues about the pixels may be a value of which weight with respect to the value aboutthe current pixel is largest and weight with respect to a value about a neighboring pixelthereof is exponentially decreased according to distances from the current pixel.
[00273] The pixel group unit motion compensator 120 may determine the displacementvector per unit time of the current pixel in a horizontal direction or a vertical direction byusing the weighted average value of the current pixel.
[00274] The inter predictor 155 may generate the predicted pixel value of the currentblock by using the block-unit motion-compensated value with respect to the current blockand the pixel group unit motion-compensated value with respect to the current block. Forexample, the inter predictor 155 may generate the predicted pixel value of the currentblock by adding the block-unit motion-compensated value and the pixel group unitmotion-compensated value with respect to the current block. In particular, when themotion prediction mode of the current block is a bi-directional motion prediction mode,the inter predictor 155 may generate the predicted pixel value of the current block byusing the block-unit motion-compensated value and the pixel group unitmotion-compensated value with respect to the current block.
[00275] When the motion prediction mode of the current block is a unidirectionalmotion prediction mode, the inter predictor 155 may generate the predicted pixel value ofthe current block by using the block-unit motion-compensated value and the pixel groupunit motion-compensated value with respect to the current block. In this regard, auni-direction indicates using of one reference picture from among pictures that werepreviously encoded. The one reference picture is not limited to a picture prior to thecurrent picture in a display order but may be a picture displayed after the current picture.
[00276] The inter predictor 155 may determine the motion prediction mode of thecurrent block, and may output information indicating the motion prediction mode of thecurrent block. For example, the inter predictor 155 may determine the motion predictionmode of the current block to be a bi-directional motion prediction mode, and may outputinformation indicating the bi-directional motion prediction mode. In this regard, thebi-directional motion prediction mode refers to a mode in which motion is predicted byusing reference blocks in two decoded reference pictures.
[00277] The bitstream generator 170 may generate a bitstream including a motionvector indicating the reference block. The bitstream generator 170 may encode themotion vector indicating the reference block, and may generate a bitstream including theencoded motion vector. The bitstream generator 170 may encode a difference value ofthe motion vector indicating the reference block, and may generate a bitstream includingthe encoded difference value of the motion vector. In this regard, the difference value ofthe motion vector may indicate a difference between the motion vector and a predictor ofthe motion vector. In this regard, the difference value of the motion vector may indicate adifference value of motion vectors with respect to reference pictures respectively relatedto prediction directions including an L0 direction and an L1 direction. In this regard, thedifference value of the motion vector with respect to the L0 direction may indicate adifference value of a motion vector indicating a reference block in a reference pictureincluded in an L0 reference picture list, and the difference value of the motion vector withrespect to the L1 direction may indicate a difference value of a motion vector indicating areference block in a reference picture included in an L1 reference picture list.
[00278] Also, the bitstream generator 170 may generate the bitstream further includinginformation indicating the motion prediction mode of the current block. The bitstreamgenerator 170 may encode a reference picture index indicating the reference picture ofthe current block from among the previously encoded pictures, and may generate abitstream including the encoded reference picture index. In this regard, the referencepicture index may indicate a reference picture index with respect to each of theprediction directions including the L0 direction and the L1 direction. In this regard, thereference picture index with respect to the L0 direction may indicate an index indicating areference picture among pictures included in the L0 reference picture list, and thereference picture index with respect to the L1 direction may indicate an index indicating areference picture among pictures included in the L1 reference picture list.
[00279] The video encoding apparatus 150 may include an image encoder (not shown),and the image encoder may include the inter predictor 155 and the bitstream generator170. The image encoder will be described below with reference to FIG. 1F.
[00280] FIG. 1D illustrates a flowchart of a video encoding method according to variousembodiments.
[00281] Referring to FIG. 1D, in operation S150, the video encoding apparatus 150may obtain a prediction block, a first motion vector, and a second motion vector of acurrent block, and a parameter with respect to pixel group unit motion compensation byperforming motion compensation and the pixel group unit motion compensation on thecurrent block.
[00282] In operation S155, the video encoding apparatus 150 may generate abitstream including information about the first motion vector and the second motionvector, and motion prediction mode information indicating that a motion prediction modeof the current block is a bi-directional motion prediction mode. In this regard, the firstmotion vector may be a motion vector indicating, from the current block, a first referenceblock of a first reference picture which corresponds to the current block in a currentpicture, and the second motion vector may be a motion vector indicating, from thecurrent block, a second reference block of a second reference picture which correspondsto the current block in the current picture.
[00283] The video encoding apparatus 150 may encode a residual block of the currentblock which indicates a difference between a pixel of the prediction block of the currentblock and an original block of the current block, and may generate the bitstream furtherincluding the encoded residual signal. The video encoding apparatus 150 may encodeinformation about the prediction mode of the current block and a reference picture index,and may generate the bitstream further including the encoded information about theprediction mode of the current block and the encoded reference picture index. Forexample, the video encoding apparatus 150 may encode information indicating that theprediction mode of the current block is an inter prediction mode and the reference pictureindex indicating at least one picture from among previously decoded pictures, and maygenerate the bitstream further including the encoded information about the predictionmode of the current block and the encoded reference picture index.
[00284] The video encoding apparatus 150 may perform block-unit motioncompensation and pixel group unit motion compensation on the current block by using agradient value in a horizontal direction or a vertical direction of a first correspondingreference pixel in the first reference block which corresponds to a current pixel includedin a current pixel group in the current block, a gradient value in a horizontal direction or avertical direction of a second corresponding reference pixel in the second referenceblock which corresponds to the current pixel, a pixel value of the first correspondingreference pixel, a pixel value of the second corresponding reference pixel, and adisplacement vector per unit time of the current pixel in a horizontal direction or verticaldirection. The video encoding apparatus 150 may obtain the prediction block of thecurrent block by performing the block-unit motion compensation and the pixel group unitmotion compensation on the current block.
[00285] In this regard, the displacement vector per unit time in a horizontal direction orvertical direction of pixels of the current block including a pixel adjacent to the inside of aboundary of the current block may be determined by using values about reference pixelsincluded in the first reference block and the second reference block, without using astored value about a pixel located outside boundaries of the first reference block and thesecond reference block. The values about the reference pixels included in the firstreference block and the second reference block may be pixel values or gradient valuesof the reference pixels.
[00286] FIG. 1E illustrates a block diagram of an image decoder 600 according tovarious embodiments.
[00287] The image decoder 600 according to various embodiments performsoperations performed by the image decoder (not shown) of the video decodingapparatus 100 to decode image data.
[00288] Referring to FIG. 1E, an entropy decoder 615 parses, from a bitstream 605,encoded image data to be decoded and encoding information required for decoding. Theencoded image data is a quantized transform coefficient, and an inverse quantizer 620and an inverse transformer 625 reconstruct residue data from the quantized transformcoefficient.
[00289] An intra predictor 640 performs intra prediction per block. An inter predictor 635performs inter prediction by using a reference image obtained from a reconstructedpicture buffer 630, per block. The inter predictor 635 of FIG. 1E may correspond to theinter predictor 110 of FIG. 1A.
[00290] Data of a spatial domain with respect to a block of a current image 605 may bereconstructed by adding prediction data and the residue data of each block generated bythe intra predictor 640 or the inter predictor 635, and a deblocking unit 645 and asample-adaptive offset (SAO) performer 650 may output a filtered reconstructed image660 by performing loop filtering on the reconstructed data of the spatial domain. Also,reconstructed images stored in the reconstructed picture buffer 630 may be output as areference image.
[00291] In order for a decoder (not shown) of the video decoding apparatus 100 todecode image data, stepwise operations of the image decoder 600 according to variousembodiments may be performed per block.
[00292] FIG. 1F illustrates a block diagram of an image encoder according to variousembodiments.
[00293] An image encoder 700 according to various embodiments performs operationsperformed by the image encoder (not shown) of the video encoding apparatus 150 toencode image data.
[00294] That is, an intra predictor 720 performs intra prediction per block on a currentimage 705, and an inter predictor 715 performs inter prediction by using the currentimage 705 per block and a reference image obtained from a reconstructed picture buffer710. In this regard, the inter predictor 715 of FIG. 1E may correspond to the interpredictor 155 of FIG. 1C.
[00295] Residue data may be generated by subtracting prediction data regarding eachblock output from the intra predictor 720 or the inter predictor 715 from data regarding anencoded block of the current image 705, and a transformer 725 and a quantizer 730 mayoutput a transform coefficient quantized per block by performing transform andquantization on the residue data. An inverse quantizer 745 and an inverse transformer750 may reconstruct residue data of a spatial domain by performing inverse quantizationand inverse transform on the quantized transform coefficient. The reconstructed residuedata of the spatial domain may be added to the prediction data regarding each blockoutput from the intra predictor 720 or the inter predictor 715 to be reconstructed as datain spatial domain regarding a block of the current image 705. A deblocking unit 755 andan SAO performer 760 generate a filtered reconstructed image by performing in-loopfiltering on the reconstructed data of the spatial domain. The generated reconstructedimage is stored in the reconstructed picture buffer 710. Reconstructed images stored inthe reconstructed picture buffer 710 may be used as reference images for interprediction of another image. An entropy encoder 735 may entropy-encode the quantizedtransform coefficient, and the entropy-encoded coefficient may be output as a bitstream740.
[00296] In order for the image encoder 700 according to various embodiments to beapplied to the video encoding apparatus 150, stepwise operations of the image encoder700 according to various embodiments may be performed per block.
[00297] FIG. 2 is a reference diagram for describing block-based bi-directional motionprediction and compensation processes, according to an embodiment.
[00298] Referring to FIG. 2, the video encoding apparatus 150 performs bi-directionalmotion prediction in which a region most similar to a current block 201 of a currentpicture 200 to be encoded is searched for in a first reference picture 210 and a secondreference picture 220. In this regard, it is assumed that the first reference picture 210 is apicture prior to the current picture 200, and the second reference picture 220 is a pictureafter the current picture 200. As a result of the bi-directional motion prediction, the videoencoding apparatus 150 determines a first corresponding region 212 most similar to thecurrent block 201 from the first reference picture 210, and a second correspondingregion 222 most similar to the current block 201 from the second reference picture 220.In this regard, the first corresponding region 212 and the second corresponding region222 may be reference regions of the current block 201.
[00299] The video encoding apparatus 150 may determine a first motion vector MV1based on a location difference between the first corresponding region 212 and a block211 of the first reference picture 210 at the same location as the current block 201, andmay determine a second motion vector MV2 based on a location difference between thesecond corresponding region 222 and a block 221 of the second reference picture 220 atthe same location as the current block 201.
[00300] The video encoding apparatus 150 performs block-unit bi-directional motioncompensation on the current block 201 by using the first motion vector MV1 and thesecond motion vector MV2.
[00301] For example, when a pixel value located at (i, j) of the first reference picture210 is P0(i,j), a pixel value located at (i, j) of the second reference picture 220 is P1(i,j),MV1=(MVx1, MVy1), and MV2=(MVx2, MVy2), wherein i and j are integers, a block-unitbi-directional motion compensation value P_BiPredBlock(i,j) of a pixel at a (i, j) locationof the current block 201 may be calculated according to an equation:P_BiPredBlock(i,j)={P0(i+MVx1, j+MVy1)+P1(i+MVx2, j+MVy2)} / 2. In this manner, thevideo encoding apparatus 150 may generate a block-unit motion-compensated value byperforming block-unit motion compensation on the current block 201 by using anaverage value or weighted sum of pixels in the first and second corresponding regions212 and 222 indicated by the first and second motion vectors MV1 and MV2.
[00302] FIGS. 3A to 3C are reference diagrams for describing processes of performingpixel group unit motion compensation, according to an embodiment.
[00303] In FIG. 3A, it is assumed that a first corresponding region 310 and a secondcorresponding region 320 respectively correspond to the first corresponding region 212and the second corresponding region 222 of FIG. 2, and have shifted to overlap acurrent block 300 by using bi-directional motion vectors MV1 and MV2.
[00304] Also, P(i,j) indicates a pixel of the current block 300 at a (i, j) location (where, iand j are integers) that is bi-directionally predicted, P0(i,j) indicates a first reference pixelvalue of a first reference picture corresponding to the pixel P(i,j) of the current block 300that is bi-directionally predicted, and P1(i,j) indicates a second reference pixel value of a30 second reference picture corresponding to the pixel P(i,j) of the current block 300 that isbi-directionally predicted.
[00305] In other words, the first reference pixel value P0(i,j) is a pixel value of a pixelcorresponding to the pixel P(i,j) of the current block 300 determined by the bi-directionalmotion vector MV1 indicating the first reference picture, and the second reference pixelvalue P1(i,j) is a pixel value of a pixel corresponding to the pixel P(i,j) of the current block300 determined by the bi-directional motion vector MV2 indicating the second referencepicture.
[00306] Also, it is defined that(Equation) indicates a gradient value of a firstreference pixel in a horizontal direction,(Equation) indicates a gradient value of thefirst reference pixel in a vertical direction,(Equation) indicates a gradient value of asecond reference pixel in the horizontal direction, and(Equation) indicates a gradientvalue of the second reference pixel in the vertical direction. Also, it is defined that τ0indicates a temporal distance between a current picture to which the current block 300belongs and the first reference picture to which the first corresponding region 310belongs, and τ1 indicates a temporal distance between the current picture and thesecond reference picture to which the second corresponding region 320 belongs. In thisregard, a temporal distance between pictures may indicate a difference of POCs of thepictures.
[00307] When there is uniform small motion in a video sequence, a pixel in the firstcorresponding region 310 of the first reference picture, which is most similar to the pixelP(i,j) on which bi-directional pixel group unit motion compensation is performed, is notthe first reference pixel P0(i,j), but is a first displacement reference pixel PA, in which thefirst reference pixel P0(i,j) is moved by a predetermined displacement vector. Because itis assumed that, as described above, there is uniform motion in the video sequence, apixel in the second corresponding region 320 of the second reference picture, which ismost similar to the pixel P(i,j), may be a second displacement reference pixel PB, inwhich the second reference pixel P1(i,j) is moved by a predetermined displacementvector.
[00308] A displacement vector may include a displacement vector Vx in an x-axisdirection and a displacement vector Vy in a y-axis direction. Accordingly, the pixel groupunit motion compensator 165 calculates the displacement vector Vx in the x-axisdirection and the displacement vector Vy in the y-axis direction included in thedisplacement vector, and may perform pixel group unit motion compensation by usingthe displacement vector.
[00309] An optical flow indicates a pattern of apparent motion on an object or surface,which is caused by relative motion between a scene and an observer (eyes or a videoimage obtaining apparatus such as a camera). In a video sequence, an optical flow maybe expressed by calculating motion between frames obtained at arbitrary times t andt+Δt. A pixel value located at (x, y) in the frame of the time t may be defined as I(x,y,t).That is, I(x,y,t) may be a value that changes temporally and spatially. I(x,y,t) may bedifferentiated according to Equation1 with respect to the time t.
[00310] [Equation 1]
[00311]
[00312] When a pixel value changes according to motion but does not changeaccording to time with respect to small motion in a block, dI / dt is 0. Also, when it isassumed that motion of a change in a pixel value according to time is constant, dx / dtmay indicate the displacement vector Vx of the pixel value I(x,y,t) in the x-axis directionand dy / dt may indicate the displacement vector Vy of the pixel value I(x,y,t) in the y-axisdirection, and accordingly, Equation 1 may be expressed as Equation 2.
[00313] [Equation 2]
[00314]
[00315] In this regard, sizes of the displacement vector Vx in the x-axis direction andthe displacement vector Vy in the y-axis direction may have a value smaller than pixelaccuracy used in bi-directional motion prediction. For example, when the pixel accuracyis 1 / 4 or 1 / 16 during the bi-directional motion prediction, the sizes of the displacementvectors Vx and Vy may have a value smaller than 1 / 4 or 1 / 16.
[00316] The pixel group unit motion compensator 165 calculates the displacementvector Vx in the x-axis direction and the displacement vector Vy in the y-axis directionaccording to Equation 2, and performs pixel group unit motion compensation by usingthe displacement vectors Vx and Vy. In Equation 2, because the pixel value I(x,y,t) is avalue of an original signal, high overheads may be caused during encoding when thevalue of the original signal is changelessly used. Accordingly, the pixel group unit motioncompensator 165 may calculate the displacement vectors Vx and Vy according toEquation 2 by using pixels of the first reference picture and the second reference picture,which are determined as results of performing bi-directional block-unit motioncompensation. That is, the pixel group unit motion compensator 165 determines thedisplacement vector Vx in the x-axis direction and the displacement vector Vy in they-axis direction, in which Δ is minimum in a window Ωij having a predetermined size andincluding neighboring pixels around the pixel P(i,j) on which bi-directional motioncompensation is performed. Δ may be 0, but the displacement vector Vx in the x-axisdirection and the displacement vector Vy in the y-axis direction, which satisfy Δ=0 withrespect to all pixels in the window Ωij, may not exist, and thus the displacement vectorVx in the x-axis direction and the displacement vector Vy in the y-axis direction, in whichΔ is minimum, are determined. Processes of obtaining the displacement vectors Vx andVy will be described in detail with reference to FIG. 8A.
[00317] To determine a predicted pixel value of a current pixel, a function P(t) withrespect to t may be determined according to Equation 3.
[00318] [Equation 3]
[00319]
[00320] In this regard, it is assumed that a picture when t=0 is a current pictureincluding a current block. Accordingly, the predicted pixel value of the current pixelincluded in the current block may be defined as a value of P(t) when t is 0.
[00321] When the temporal distance between the current picture and the first referencepicture (the first reference picture is temporally prior to the current picture) is τ0 and thetemporal distance between the current picture and the second reference picture (thesecond reference picture is temporally after the current picture) is τ1, a reference pixelvalue in the first reference picture is equal to P(-τ0), and a reference pixel value in thesecond reference picture is equal to P(τ1). Hereinafter, for convenience of calculation, itis assumed that τ0 and τ1 are both equal to τ.
[00322] Coefficients of each degree of P(t) may be determined according to Equation 4.Here, P0(i,j) may indicate a pixel value at a (i,j) location of the first reference picture, andP1(i,j) may indicate a pixel value at a (i,j) location of the second reference picture.
[00323] [Equation 4]
[00324]
[00325] Accordingly, a predicted pixel value P(0) of the current pixel in the current blockmay be determined according to Equation 5.
[00326] [Equation 5]
[00327]
[00328] Equation 5 may be expressed as Equation 6, in consideration of Equation 2.
[00329] [Equation 6]
[00330]
[00331] Accordingly, the predicted pixel value of the current pixel may be determinedby using the displacement vector Vx, the displacement vector Vy, gradient values of thefirst reference pixel in the horizontal and vertical directions, and gradient values of thesecond reference pixel in the horizontal and vertical directions. In this regard, a portion(P0(i,j)+P1(i,j)) / 2) that is not related to the displacement vectors Vx and Vy may be ablock-unit motion-compensated value, and a portion that is related to the displacementvectors Vx and Vy may be a pixel group unit motion-compensated value. As a result, thepredicted pixel value of the current pixel may be determined by adding the block-unitmotion-compensated value and the pixel group unit motion-compensated value.
[00332] Hereinabove, processes of determining the predicted pixel value of the currentpixel when the temporal distance between the first reference picture and the currentpicture and the temporal distance between the second reference picture and the currentpicture are both τ and thus are the same are described for convenience of descriptions,but the temporal distance between the first reference picture and the current picture maybe τ0 and the temporal distance between the second reference picture and the currentpicture may be τ1. Here, the predicted pixel value P(0) of the current pixel may bedetermined according to Equation 7.
[00333] [Equation 7]
[00334]
[00335] Considering Equation 2, Equation 7 may be expressed as Equation 8.
[00336] [Equation 8]
[00337]
[00338] According to the aforementioned descriptions, the first reference picture islocated temporally after the current picture and the second reference picture is locatedtemporally prior to the current picture, but both the first reference picture and the secondreference picture may be located temporally prior to the current picture or after thecurrent picture.
[00339] For example, as illustrated in FIG. 3B, the first reference picture including thefirst corresponding region 310 and the second reference picture including the secondcorresponding region 320 may both be located temporally prior to the current pictureincluding the current block 300.
[00340] In this case, the predicted pixel value P(0) of the current pixel may bedetermined according to Equation 9 in which τ1 indicating the temporal distance betweenthe second reference picture and the current picture in Equation 8 indicated withreference to FIG. 3A is replaced by -τ1.
[00341] [Equation 9]
[00342]
[00343] For example, as illustrated in FIG. 3C, the first reference picture including thefirst corresponding region 310 and the second reference picture including the secondcorresponding region 320 may both be located temporally after the current pictureincluding the current block 300.
[00344] In this case, the predicted pixel value P(0) of the current pixel may bedetermined according to Equation 10 in which τ0 indicating the temporal distancebetween the first reference picture and the current picture in Equation 8 indicated withreference to FIG. 3A is replaced by -τ0.
[00345] [Equation 10]
[00346]
[00347] However, when both the first reference picture and the second referencepicture are located temporally prior to the current picture or after the current picture asillustrated in FIGS. 3B and 3C, pixel group unit motion compensation may be performedwhen the first reference picture and the second reference picture are not the samereference picture. Also, in this case, pixel group unit motion compensation may beperformed only when the bi-directional motion vectors MV1 and MV2 each have anon-zero component. Also, in this case, the pixel group unit motion compensation maybe performed only when a ratio of the motion vectors MV1 and MV2 is the same as aratio of the temporal distance between the first reference picture and the current pictureand the temporal distance between the second reference picture and the current picture.For example, the pixel group unit motion compensation may be performed when a ratioof an x component of the motion vector MV1 and an x component of the motion vectorMV2 is the same as a ratio of a y component of the motion vector MV1 and a ycomponent of the motion vector MV2, and is the same as a ratio of the temporal distanceτ0 between the first reference picture and the current picture and the temporal distance τ1between the second reference picture and the current picture.
[00348] FIG. 4 is a reference diagram for describing processes of calculating gradientvalues in horizontal and vertical directions, according to an embodiment.
[00349] Referring to FIG. 4, a gradient value (Equation)of a first referencepixel P0(i,j) 410 of a first reference picture in a horizontal direction and a gradient value(Equation)of the first reference pixel P0(i,j) 410 in a vertical direction may becalculated by obtaining a variation of a pixel value at a neighboring fractional pixellocation adjacent to the first reference pixel P0(i,j) 410 in the horizontal direction and avariation of a pixel value at a neighboring fractional pixel location adjacent to the firstreference pixel P0(i,j) 410 in the vertical direction. That is, according to Equation 11, thegradient value (Equation)in the horizontal direction may be calculated bycalculating a variation of pixel values of a fractional pixel P0(i-h,j) 460 and a fractionalpixel P0(i+h,j) 470 distant from P0(i,j) by h in the horizontal direction, wherein h is afractional value smaller than 1, and the gradient value(Equation) in the verticaldirection may be calculated by calculating a variation of pixel values of a fractional pixelP0(i,j-h) 480 and a fractional pixel P0(i,j+h) 490 away from P0(i,j) by h in the verticaldirection.
[00350] [Equation 11]
[00351]
[00352] Pixel values of the fractional pixels P0(i-h,j) 460, P0(i+h,j) 470, P0(i,j-h) 480,and P0(i,j+h) 490 may be calculated by using general interpolation. In addition, gradientvalues of a second reference pixel of another second reference picture in horizontal andvertical directions may also be calculated similarly to Equation 11.
[00353] According to an embodiment, instead of calculating a gradient value bycalculating a variation of pixel values at fractional pixel locations as in Equation 11, agradient value in a reference pixel may be calculated by using a predetermined filter. Afilter coefficient of the predetermined filter may be determined based on a coefficient ofan interpolation filter used to obtain a pixel value at a fractional pixel location, based onlinearity of a filter.
[00354] FIG. 5 is a reference diagram for describing processes of calculating gradientvalues in horizontal and vertical directions, according to another embodiment.
[00355] According to another embodiment, a gradient value may be determined byapplying a predetermined filter to pixels of a reference picture. Referring to FIG. 5, thevideo decoding apparatus 100 may calculate a gradient value of a reference pixel P0500 in a horizontal direction by applying a predetermined filter to MMax left pixels 520 and|MMin| right pixels 510 based on the reference pixel P0 500 of which a current horizontalgradient value is to be obtained. A filter coefficient used herein may be determinedaccording to a value α indicating an interpolation location (fractional pel location)between MMax and MMin integer pixels used to determine a window size, as illustrated inFIGS. 7A to 7D. For example, referring to FIG. 7A, when MMin and MMax for determining awindow size are respectively -2 and 3, and are distant from the reference pixel P0 500 by1 / 4, i.e., α=1 / 4, coefficient filters {4, -17. -36. 60, -15, 4} in a second row of FIG. 7A areapplied to neighboring pixels P-2, P-1, P0, P1, P2, and P3. In this case, a gradient value(Equation) of the reference pixel P0 500 in the horizontal direction may becalculated according to a weighted sum using a filter coefficient and a neighboring pixel,such as an equation;(Equation) 4*P-2 - 17*P-1 + -36*P0 +60*P1-15*P2+4*P3 +32>>6. Similarly, a gradient value in a vertical direction may also be calculated byapplying the filter coefficients shown in FIGS. 7A to 7E to neighboring pixels according toan interpolation location, and MMin and MMax for determining a window size.
[00356] FIGS. 6A to 6B are diagrams for describing processes of determining gradientvalues in horizontal and vertical directions by using 1D filters, according to anembodiment.
[00357] Referring to FIG. 6A, filtering may be performed by using a plurality of 1D filterswith respect to an integer pixel so as to determine a gradient value of a reference pixel ina horizontal direction in a reference picture. Pixel group unit motion compensation isadditional motion compensation performed after block-unit motion compensation isperformed. Accordingly, a reference location of reference blocks of a current blockindicated by a motion vector during the block-unit motion compensation may be afractional pixel location, and the pixel group unit motion compensation may be performedon reference pixels in a reference block at a fractional pixel location. Therefore, filteringmay be performed, in consideration that a gradient value of a pixel at a fractional pixellocation is determined.
[00358] Referring to FIG. 6A, first, the video decoding apparatus 100 may performfiltering on pixels located in a horizontal or vertical direction from a neighboring integerpixel of a reference pixel in a reference picture, by using a first 1D filter. Equally, thevideo decoding apparatus 100 may perform filtering on neighboring integer pixels in arow or column which are different from the reference pixel, by using the first 1D filter. Thevideo decoding apparatus 100 may generate a gradient value of the reference pixel inthe horizontal direction by performing filtering on values generated via the filtering, byusing a second 1D filter.
[00359] For example, when a location of a reference pixel is a location of a fractionalpixel at (x+α, y+β) (where, x and y are integers and α and β are fractions), filtering maybe performed according to Equation 12 by using a 1D vertical interpolation filter oninteger pixels (x,y), (x-1,y), (x+1, y), through (x+MMin,y) and (x+MMax,y) (where, MMin andMMmax are integers) in a horizontal direction.
[00360] [Equation 12]
[00361]
[00362] Here, fracFilterβ may indicate an interpolation filter for determining a pixel valueat a fractional pixel location β in a vertical direction, and fracFilterβ[j'] may indicate acoefficient of an interpolation filter to be applied to a pixel at a (i,j') location. I[i,j'] mayindicate a pixel value at the (i,j') location.
[00363] That is, the first 1D filter may be an interpolation filter for determining afractional pixel value in a vertical direction. offset1 may indicate an offset for preventing around-off error, and shift1 may indicate the number of de-scaling bits. Temp[i,j+β] mayindicate pixel value at a fractional pixel location (i,j+β). Temp[i',j+β] (where, i' is aninteger from i+Mmin to i+Mmax excluding i) may also be determined according to Equationby replacing i by i'.
[00364] Afterward, the video decoding apparatus 100 may perform filtering on a pixelvalue at a fractional pixel location (i,j+β) and a pixel value at a fractional pixel location(i',j+β) by using a second 1D filter.
[00365] [Equation 13]
[00366]
[00367] In this regard, gradFilterα may be a gradient filter for determining a gradientvalue at a fractional pixel location α in a horizontal direction. gradFilterα[i'] may indicate acoefficient of an interpolation filter to be applied to a pixel at a (i',j+β) location. That is, thesecond 1D filter may be a gradient filter for determining a gradient value in a horizontaldirection. offset2 may indicate an offset for preventing a round-off error, and shift2 mayindicate the number of de-scaling bits.
[00368] That is, according to Equation 13, the video decoding apparatus 100 maydetermine a gradient value (Equation)in a horizontal direction at (i+α,j+β) byperforming filtering on a pixel value (Temp[i,j+β]) at a pixel location (i, j+β) and a pixelvalue (Temp[i',j+β]) located in a vertical direction from the pixel location (i, j+β), by usingthe gradient filter gradFilterα.
[00369] Hereinabove, a gradient value in a horizontal direction is determined by firstapplying an interpolation filter and then applying a gradient filter, but the disclosure is notlimited thereto and thus, the gradient value in the horizontal direction may be determinedby first applying the gradient filter and then applying the interpolation filter. Hereinafter,an embodiment in which a gradient value in a horizontal direction is determined byapplying a gradient filter and then an interpolation filter will now be described.
[00370] For example, when a location of a reference pixel is a location of a fractionalpixel at (x+α, y+β) (wherein x and y are integers and α and β are fractions), filtering maybe performed according to Equation 14 by using the first 1D filter, with respect to integerpixels (x,y), (x-1,y), (x+1, y), through (x+MMin,y) and (x+MMax,y) (where, MMin and MMmaxare integers) in a horizontal direction.
[00371] [Equation 14]
[00372]
[00373] In this regard, gradFilterα may indicate a gradient filter for determining agradient value at a fractional pixel location α in a horizontal direction, and gradFilterα[i']may indicate a coefficient of a gradient filter to be applied to a pixel at a (i',j) location. I[i',j]may indicate a pixel value at the (i',j) location.
[00374] That is, the first 1D filter may be an interpolation filter for determining a gradientvalue of a pixel in a horizontal direction, wherein a horizontal component of a pixellocation is a fractional location. offset3 may indicate an offset for preventing a round-offerror, and shift3 may indicate the number of de-scaling bits. Temp[i+α,j] may indicate agradient value at a pixel location (i+α,j) in the horizontal direction. Temp[i+α,j'] (where, j'is an integer from j+Mmin to j+Mmax excluding j) may also be determined according toEquation 14 by replacing j by j'.
[00375] Then, the video decoding apparatus 100 may perform filtering on a gradientvalue at a pixel location (i+α,j) in the horizontal direction and a gradient value at a pixellocation (i+α,j') in the horizontal direction by using the second 1D filter, according toEquation 15.
[00376] [Equation 15]
[00377]
[00378] In this regard, fracFilterβ may be an interpolation filter for determining a pixelvalue at a fractional pixel location β in a vertical direction. fracFilterβ[j'] may indicate acoefficient of an interpolation filter to be applied to a pixel at a (i+β, j') location. That is,the second 1D filter may be an interpolation filter for determining a pixel value at afractional pixel location β in a vertical direction. offset4 may indicate an offset forpreventing a round-off error, and shift4 may indicate the number of de-scaling bits.
[00379] That is, according to Equation 15, the video decoding apparatus 100 maydetermine a gradient value (Equation)in a horizontal direction at (i+α,j+β) byperforming filtering on a gradient value (Temp[i+α,j]) at a pixel location (i+α, j) in ahorizontal direction and a gradient value (Temp[i+α,j']) of pixels in a horizontal directionlocated in a vertical direction from the pixel location (i+α, j), by using the gradient filterfracFilterβ.
[00380] Referring to FIG. 6B, filtering may be performed by using a plurality of 1D filterswith respect to an integer pixel so as to determine a gradient value of a reference pixel ina vertical direction in a reference picture. Pixel group unit motion compensation isadditional motion compensation performed after block-unit motion compensation isperformed. Therefore, a reference location of reference blocks of a current blockindicated by a motion vector during the block-unit motion compensation may be afractional pixel location, and the pixel group unit motion compensation may be performedon reference pixels in a reference block at a fractional pixel location. Accordingly, filteringmay be performed, in consideration that a gradient value of a pixel at a fractional pixellocation is determined.
[00381] Referring to FIG. 6B, first, the video decoding apparatus 100 may performfiltering on pixels located in a horizontal or vertical direction from a neighboring integerpixel of a reference pixel in a reference picture, by using a first 1D filter. Equally, thevideo decoding apparatus 100 may perform filtering on neighboring pixels in a row orcolumn which are different from the reference pixel, by using the first 1D filter. The videodecoding apparatus 100 may generate a gradient value of the reference pixel in thevertical direction by performing filtering on values generated via the filtering, by using asecond 1D filter.
[00382] For example, when a location of a reference pixel is a location of a fractionalpixel at (x+α, y+β) (where, x and y are integers and α and β are fractions), filtering maybe performed according to Equation 16 by using the first 1D filter with respect to integerpixels (x,y), (x-1,y-1), (x+1, y+1) through (x+MMin,y+MMin) and(x+MMax,y+Mmax) (where,MMin and MMmax are integers) in a horizontal direction.
[00383] [Equation 16]
[00384]
[00385] In this regard, fracFilterα may indicate an interpolation filter for determining apixel value at a fractional pixel location α in a horizontal direction, and fracFilterα[i'] mayindicate a coefficient of an interpolation filter to be applied to a pixel at a (i',j) location. I[i',j]may indicate a pixel value at the (i',j) location.
[00386] That is, the first 1D filter may be an interpolation filter for determining a pixelvalue at a fractional pixel location α in a horizontal direction. offset5 may indicate anoffset for preventing a round-off error, and shift5 may indicate the number of de-scalingbits.
[00387] Temp[i+α,j] may indicate a pixel value at a fractional pixel location (i+α,j).Temp[i+α,j'] (where, j' is an integer from j+Mmin to j+Mmax excluding j) may also bedetermined according to Equation 16 by replacing j by j'.
[00388] Afterward, the video decoding apparatus 100 may perform filtering on a pixelvalue at a pixel location (i+α,j) and a pixel value at a pixel location (i+α,j') according toEquation 17, by using a second 1D filter.
[00389] [Equation 17]
[00390]
[00391] In this regard, gradFilterβ may be a gradient filter for determining a gradientvalue at a fractional pixel location β in a vertical direction. gradFilterβ[j'] may indicate acoefficient of an interpolation filter to be applied to a pixel at a (i+α,j') location. That is, thesecond 1D filter may be a gradient filter for determining a gradient value in a verticaldirection at a fractional pixel location β. offset6 may indicate an offset for preventing around-off error, and shift6 may indicate the number of de-scaling bits.
[00392] That is, according to Equation 17, the video decoding apparatus 100 maydetermine a gradient value (Equation)in a vertical direction at (i+α,j+β) byperforming filtering on a pixel value (Temp[i+α,j]) at a pixel location (i+α,j) and a pixelvalue (Temp[i+α,j']) located in a vertical direction from the pixel location (i+α,j), by usingthe gradient filter gradFilterβ.
[00393] According to the aforementioned descriptions, a gradient value in a verticaldirection is determined by first applying an interpolation filter and then applying agradient filter, but the disclosure is not limited thereto and thus, the gradient value in thevertical direction may be determined by first applying the gradient filter and then applyingthe interpolation filter. Hereinafter, an embodiment in which a gradient value in a verticaldirection is determined by applying a gradient filter and then an interpolation filter willnow be described.
[00394] For example, when a location of a reference pixel is a location of a fractionalpixel at (x+α, y+β) (where, x and y are integers and α and β are fractions), filtering maybe performed according to Equation 18 by using the first 1D filter, with respect to integerpixels (x,y), (x,y-1), (x, y+1) through (x,y+MMin) and (x,y+Mmax) (where, MMin and MMmaxare integers) in a vertical direction.
[00395] [Equation 18]
[00396]
[00397] In this regard, gradFilterβ may indicate a gradient filter for determining agradient value at a fractional pixel location β in a vertical direction, and gradFilterβ[j'] mayindicate a coefficient of a gradient filter to be applied to a pixel at a (i,j') position. I[i,j'] mayindicate a pixel value at the (i,j') location.
[00398] That is, the first 1D filter may be an interpolation filter for determining a gradientvalue of a pixel in a vertical direction, wherein a vertical component of a pixel location isa fractional location. offset7 may indicate an offset for preventing a round-off error, andshift7 may indicate the number of de-scaling bits.
[00399] Temp[i,j+β] may indicate a gradient value at a pixel location (i,j+β) in thevertical direction. Temp[i',j+β] (where, i' is an integer from i+Mmin to i+Mmax excluding i)may also be determined according to Equation 18 by replacing i by i'.
[00400] Afterward, the video decoding apparatus 100 may perform filtering on agradient value at a pixel location (i, j+β) in the vertical direction and a gradient value at apixel location (i',j+β) in the vertical direction by using the second 1D filter, according toEquation 19.
[00401] [Equation 19]
[00402]
[00403] In this regard, fracFilterα may be an interpolation filter for determining a pixelvalue at a fractional pixel location α in a horizontal direction. fracFilterα[i'] may indicate acoefficient of an interpolation filter to be applied to a pixel at a (i',j+β) location. That is, thesecond 1D filter may be an interpolation filter for determining a pixel value at a fractionalpixel location α in a horizontal direction. offset8 may indicate an offset for preventing around-off error, and shift8 may indicate the number of de-scaling bits.
[00404] That is, according to Equation 19, the video decoding apparatus 100 maydetermine a gradient value (Equation)in a vertical direction at (i+α,j+β) byperforming filtering on a gradient value (Temp[i,j+β]) at a pixel location (i, j+β) in avertical direction and a gradient value (Temp[i', j+β)]) of pixels in a vertical directionlocated in a horizontal direction from the pixel location (i, j+β), by using the gradient filterfracFilterα.
[00405] According to an embodiment, in the video decoding apparatus 100, gradientvalues in horizontal and vertical directions at (i+α, j+β) may be determined according tocombinations of various filters described above. For example, in order to determine agradient value in a horizontal direction, an interpolation filter for determining a pixel valuein a vertical direction may be used as a first 1D filter and a gradient filter for determininga gradient value in a horizontal direction may be used as a second 1D filter. A gradientfilter for determining a gradient value in a vertical direction may be used as a first 1D filter,and an interpolation filter for determining a pixel value in a horizontal direction may beused as a second 1D filter.
[00406] FIGS. 7A to 7E are Tables showing filter coefficients of filters used todetermine a pixel value at a fractional pixel location of a fractional pixel unit, and gradientvalues in horizontal and vertical directions, according to an embodiment.
[00407] FIGS. 7A to 7B are Tables showing filter coefficients of filters for determining agradient value at a fractional pixel location of a 1 / 4 pel unit, in a horizontal or verticaldirection.
[00408] As described above, a 1D gradient filter and a 1D interpolation filter may beused to determine a gradient value in a horizontal or vertical direction. Referring to FIG.7A, filter coefficients of the 1D gradient filter are illustrated. In this regard, a 6-tap filtermay be used as the 1D gradient filter. The filter coefficients of the 1D gradient filter maybe coefficients scaled by 2^4. Mmin indicates a difference between a location of a centerinteger pixel and a location of a farthest pixel from among integer pixels in a negativedirection applied to a filter based on the center integer pixel, and Mmax indicates adifference between the location of the center integer pixel and a location of a farthestpixel from among integer pixels in a positive direction applied to the filter based on thecenter integer pixel. For example, gradient filter coefficients for obtaining a gradientvalue of a pixel in a horizontal direction, in which a fractional pixel location α is 1 / 4 in thehorizontal direction, may be {4, -17, -36, 60, -15, -4}. Gradient filter coefficients forobtaining a gradient value of a pixel in the horizontal direction, in which a fractional pixellocation α is 0, 1 / 2, or 3 / 4 in the horizontal direction, may also be determined by referringto FIG. 7A.
[00409] Referring to FIG. 7B, filter coefficients of a 1D interpolation filter are illustrated.In this regard, a 6-tap filter may be used as the 1D interpolation filter. The filtercoefficients of the 1D interpolation filter may be coefficients scaled by 2^6. Mminindicates a difference between a location of a center integer pixel and a location of afarthest pixel from among integer pixels in a negative direction applied to a filter basedon the center integer pixel, and Mmax indicates a difference between the location of thecenter integer pixel and a location of a farthest pixel from among integer pixels in apositive direction applied to the filter based on the center integer pixel.
[00410] FIG. 7C is a table showing filter coefficients of a 1D interpolation filter used todetermine a pixel value at a fractional pixel location of a 1 / 4 pel unit.
[00411] As described above, two same 1D interpolation filters may be used inhorizontal and vertical directions so as to determine a pixel value at a fractional pixellocation.
[00412] Referring to FIG. 7C, filter coefficients of a 1D interpolation filter are illustrated.In this regard, a 6-tap filter may be used as the 1D interpolation filter. The filtercoefficients of the 1D interpolation filter may be coefficients scaled by 2^6. Mminindicates a difference between a location of a center integer pixel and a location of afarthest pixel from among integer pixels in a negative direction applied to a filter basedon the center integer pixel, and Mmax indicates a difference between the location of thecenter integer pixel and a location of a farthest pixel from among integer pixels in apositive direction applied to the filter based on the center integer pixel.
[00413] FIG. 7D is a table showing filter coefficients of filters used to determine agradient value in a horizontal or vertical direction at a fractional pixel location of 1 / 16 pelunit.
[00414] As described above, a 1D gradient filter and a 1D interpolation filter may beused to determine a gradient value in a horizontal or vertical direction. Referring to FIG.7D, filter coefficients of the 1D gradient filter are illustrated. In this regard, a 6-tap filtermay be used as the 1D gradient filter. The filter coefficients of the 1D gradient filter maybe coefficients scaled by 2^4. For example, gradient filter coefficients for obtaining agradient value of a pixel in a horizontal direction, in which a fractional pixel location α is1 / 16 in the horizontal direction, may be {8, -32, -13, 50, -18, 5}. Gradient filter coefficientsfor obtaining a gradient value of a pixel in the horizontal direction, in which a fractionalpixel position α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8, 7 / 16, or 1 / 2 in the horizontal direction, mayalso be determined by referring to FIG. 7D. Gradient filter coefficients for obtaining agradient value of a pixel in the horizontal direction, in which a fractional pixel location α is9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16, 7 / 8, or 15 / 16 in the horizontal direction, may be determinedby using symmetry of filter coefficients based on α=1 / 2. That is, filter coefficients at rightfractional pixel locations based on α=1 / 2 may be determined by using filter coefficients atleft fractional pixel locations based on α=1 / 2 shown in FIG. 7D. For example, filtercoefficients at α=15 / 16 may be determined by using filter coefficients {8, -32, -13, 50, -18,5} at α=1 / 16, which is a symmetric location based on α=1 / 2. That is, filter coefficients atα=15 / 16 may be determined to be {5, -18, 50, -13, -32, 8} by arranging {8, -32, -13, 50,-18, 5} in an inverse order.
[00415] Referring to FIG. 7E, filter coefficients of a 1D interpolation filter are illustrated.In this regard, a 6-tap filter may be used as the 1D interpolation filter. The filtercoefficients of the 1D interpolation filter may be coefficients scaled by 2^6. For example,1D interpolation filter coefficients for obtaining a pixel value of a pixel in a horizontaldirection, in which a fractional pixel location α is 1 / 16 in the horizontal direction, may be{1, -3, 64, 4, -2, 0}. Interpolation filter coefficients for obtaining a pixel value of a pixel inthe horizontal direction, in which a fractional pixel location α is 0, 1 / 8, 3 / 16, 1 / 4, 5 / 16, 3 / 8,7 / 16, or 1 / 2 in the horizontal direction, may also be determined by referring to FIG. 7E.Meanwhile, interpolation filter coefficients for obtaining a pixel value of a pixel in thehorizontal direction, in which a fractional pixel position α is 9 / 16, 5 / 8, 11 / 16, 3 / 4, 13 / 16,7 / 8, or 15 / 16 in the horizontal direction, may be determined by using symmetry of filtercoefficients based on α=1 / 2. That is, filter coefficients at right fractional pixel locationsbased on α=1 / 2 may be determined by using filter coefficients at left fractional pixellocations based on α=1 / 2 shown in FIG. 7E. For example, filter coefficients at α=15 / 16may be determined by using filter coefficients {1, -3, 64, 4, -2, 0} at α=1 / 16, which is asymmetric location based on α=1 / 2. That is, filter coefficients at α=15 / 16 may bedetermined to be {0, -2, 4, 64, -3, 1} by arranging {1, -3, 64, 4, -2, 0} in an inverse order.
[00416] FIG. 8A is a reference diagram for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to a pixel, according to an embodiment.
[00417] Referring to FIG. 8A, a window Ωij 800 having a predetermined size has a sizeof (2M+1)*(2N+1) (where, M and N are integers) with respect to a pixel P(i,j) of a currentblock, the pixel being bi-directionally predicted.
[00418] When P(i',j') indicates a pixel of a current block that is bi-directionally predictedin the window Ωij 800, wherein, when i-M≤i'≤i+M and j-N≤j'≤j+N, (i',j')¡oΩij, P0(i',j')indicates a pixel value of a first reference pixel of a first reference picture 810corresponding to the pixel P(i',j') of the current block bi-directionally predicted, P1(i',j')indicates a pixel value of a second reference pixel of a second reference picture 820corresponding to the pixel P(i',j') of the current block bi-directionally predicted,(Equation) indicates a gradient value of the first reference pixel in a horizontaldirection, (Equation) indicates a gradient value of the first reference pixel in avertical direction,(Equation) indicates a gradient value of the second referencepixel in the horizontal direction, and (Equation) indicates a gradient value of thesecond reference pixel in the vertical direction, a first displacement corresponding pixelPA' and a second displacement corresponding pixel PB' may be determined according toEquation 20. In this regard, the first displacement corresponding pixel PA' and thesecond displacement corresponding pixel PB' may be determined by using a first linearterm of a local Taylor expansion.
[00419] [Equation 20]
[00420]
[00421] In Equation 20, a displacement vector Vx in an x-axis direction and adisplacement vector Vy in a y-axis direction may be changed according to a location ofthe pixel P(i,j), i.e., are dependent on (i,j), and thus, the displacement vectors Vx and Vymay be expressed as Vx(i,j) and Vy(i,j).
[00422] A difference value Δi'j' between the first displacement corresponding pixel PA'and the second displacement corresponding pixel PB' may be determined according toEquation 21.
[00423] [Equation 21]
[00424]
[00425] The displacement vector Vx in the x-axis direction and the displacement vectorVy in the y-axis direction, which minimize the difference value Δi'j' between the firstdisplacement corresponding pixel PA' and the second displacement corresponding pixelPB', may be determined by using the sum of squares Φ(Vx,Vy) of the difference valueΔi'j' as in Equation 22.
[00426] [Equation 22]
[00427]
[00428] That is, the displacement vectors Vx and Vy may be determined by using amaximum value or a minimum value of Φ(Vx,Vy). Φ(Vx,Vy) refers to a function where thedisplacement vectors Vx and Vy are parameters, and the maximum value or theminimum value may be determined by calculating a value that becomes 0 by partiallydifferentiating Φ(Vx,Vy) with respect to τVx and τVy according to Equation 23, Φ(Vx,Vy)being arranged with respect to τVx and τVy. Hereinafter, for convenience of calculation,it is assumed that τ0 and τ1 are equal as τ.
[00429] [Equation 23]
[00430]
[00431] Two linear equations where Vx(i,j) and Vy(i,j) are variables as in Equation 24may be obtained by using an equation: (Equation) and an equation:(Equation).
[00432] [Equation 24]
[00433]
[00434] In Equation 24, s1 to s6 are calculated according to Equation 25.
[00435] [Equation 25]
[00436]
[00437] By solving simultaneous equations of Equation 24, values of Vx(i,j) and Vy(i,j)may be calculated according to τ*Vx(i,j)=-det1 / det and τ*Vy(i,j)=-det2 / det based onKramer's formulas. In this regard, det1=s3*s5-s2*s6, det2=s1*s6-s3*s4, anddet=s1*s5-s2*s2.
[00438] Simplified solutions of the above equations may be determined by performingminimization first in a horizontal direction and then in a vertical direction. That is, forexample, when it is assumed that only a displacement vector in a horizontal direction ischanged, Vy may be 0 in the first equation of Equation 24, and thus an equation:τVx=s3 / s1 may be determined.
[00439] Then, an equation: τVy=(s6-τVx*S2) / s5 may be determined when the secondequation of Equation 24 is arranged by using an equation: τVx=s3 / s1.
[00440] In this regard, gradient values (Equation) ,(Equation) ,(Equation), and(Equation) and may be scaled without changingresultant values Vx(i,j) and Vy(i,j). However, it is premised that an overflow does notoccur and a round-off error does not occur.
[00441] Adjustment parameters r and m may be introduced to prevent a divisionoperation from being performed by 0 or a very small value when Vx(i,j) and Vy(i,j) arecalculated.
[00442] For convenience, it is assumed that Vx(i,j) and Vy(i,j) are opposite to directionsillustrated in FIG. 3A. For example, Vx(i,j) and Vy(i,j) derived by Equation 24 based ondirections of Vx(i,j) and Vy(i,j) illustrated in FIG. 3A may have a same value as Vx(i,j) andVy(i,j) determined to be opposite to the directions of FIG. 3A, except for a sign.
[00443] The first displacement corresponding pixel PA' and the second displacementcorresponding pixel PB' may be determined according to Equation 26. In this regard, thefirst displacement corresponding pixel PA' and the second displacement correspondingpixel PB' may be determined by using a first linear term of a local Taylor expansion.
[00444] [Equation 26]
[00445]
[00446] A difference value Δi'j' between the first displacement corresponding pixel PA'and the second displacement corresponding pixel PB' may be determined according toEquation 27.
[00447] [Equation 27]
[00448]
[00449] The displacement vector Vx in the x-axis direction and the displacement vectorVy in the y-axis direction, which minimize the difference value Δi'j' between the firstdisplacement corresponding pixel PA' and the second displacement corresponding pixelPB', may be determined by using the sum of squares Φ(Vx,Vy) of a difference value Δas in Equation 28. That is, the displacement vectors Vx and Vy when Φ(Vx,Vy) isminimum as in Equation 29 may be determined, and may be determined by using amaximum value or a minimum value of Φ(Vx,Vy).
[00450] [Equation 28]
[00451]
[00452] [Equation 29]
[00453]
[00454] Φ(Vx,Vy) is a function where the displacement vectors Vx and Vy areparameters, and the maximum value or the minimum value may be determined bycalculating a value that becomes 0 by partially differentiating Φ(Vx,Vy) with respect tothe displacement vectors Vx and Vy as in Equation 30.
[00455] [Equation 30]
[00456]
[00457] That is, the displacement vectors Vx and Vy that minimize a value of Φ(Vx,Vy)may be determined. To solve the optimization issues, minimization may be firstperformed in a vertical direction and then may be first performed in a horizontal direction.According to the minimization, the displacement vector Vx may be determined accordingto Equation 31.
[00458] [Equation 31]
[00459]
[00460] In this regard, a function clip3(x, y, z) is a function that outputs x when z<x,outputs y when z>y, and outputs z when x<z<y. According to Equation 31, when s1+r>m,the displacement vector Vx may be clip3(-thBIO,thBIO,-s3 / (s1+r)), and when not s1+r>m,the displacement vector Vx may be 0.
[00461] According to the minimization, the displacement vector Vy may be determinedaccording to Equation 32.
[00462] [Equation 32]
[00463]
[00464] In this regard, a function clip3(x, y, z) is a function that outputs x when z<x,outputs y when z>y, and outputs z when x<z<y. According to Equation 32, when s5+r>m,the displacement vector Vy may be clip3(-thBIO,thBIO,-(s6-Vx*s2) / 2 / (s5+r), and whennot s5+r>m, the displacement vector Vy may be 0.
[00465] In this regard, s1, s2, s3, and s5 may be determined according to Equation 33.s4 may have a same value as that of s2.
[00466] [Equation 33]
[00467]
[00468] As described above, r and m may be adjustment parameters introduced toavoid a resultant value of a division operation being 0 or less than 0, and may bedetermined according to Equation 34 based on an internal bit depth d of an input video.That is, the adjustment parameter m may be a minimum allowed denominator, and theadjustment parameter r may be an adjustment parameter introduced to prevent thatdivision is performed by using 0 as a denominator when a gradient value is 0.
[00469] [Equation 34]
[00470]
[00471] The displacement vectors Vx and Vy may have an upper limit and a lower limitof ±thBIO. The displacement vectors Vx and Vy may be clipped by a particular thresholdvalue thBIO because there may be a case in which pixel group unit motion compensationis not dependable due to noise or irregular motion. The adjustment parameter thBIO maybe determined based on whether directions of all reference pictures are equal. Forexample, when the directions of all reference pictures are equal, the adjustmentparameter thBIO may be determined to be 12^(d-8-1) or 12*2^(14-d). When thedirections of all reference pictures are different, the adjustment parameter thBIO may bedetermined to be 12^(d-8-1) / 2 or 12*2^(13-d).
[00472] However, the disclosure is not limited thereto, and values of the adjustmentparameters r,m, and thBIO may be determined based on information about adjustmentparameters, the information being obtained from a bitstream. In this regard, theinformation about adjustment parameters may be included in a slice header, a pictureparameter set, a sequence parameter set, or various high level syntax carriers.
[00473] Also, the adjustment parameter may be determined according to whethertemporally-different bi-directional prediction is usable. For example, adjustmentparameter thBIOdiff of a case where the temporally-different bi-directional prediction isusable is larger than adjustment parameter thBIOsame of a case where temporally-samebi-directional prediction is usable, and a value of the adjustment parameter thBIOdiff maybe twice as large as that of the adjustment parameter thBIOsame.
[00474] FIG. 8B is a reference diagram for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to a pixel group, according to an embodiment.
[00475] Referring to FIG. 8B, a window Ωij 810 having a predetermined size has a sizeof (2M+K+1)*(2N+K+1) (where, M and N are integers) with respect to a pixel group 820having a KxK size and including a plurality of pixels in a current block, not with respect toa pixel that is bi-directionally predicted.
[00476] In this regard, what is different from FIG. 8A is that a size of a window isincreased, and except the difference, the horizontal direction displacement vector andthe vertical direction displacement vector with respect to the pixel group may bedetermined in a same manner thereto.
[00477] FIG. 8C is a reference diagram for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to a pixel, according to an embodiment.
[00478] Referring to FIG. 8C, the video decoding apparatus 100 may determine ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to each of pixels 830. Therefore, a displacement vector per unit time 835 withrespect to each of the pixels 830 may be determined. In this regard, a horizontaldirection displacement vector Vx[i,j] and a vertical direction displacement vector Vy[i,j] ofthe displacement vector per unit time 835 with respect to each of the pixels 830 may bedetermined according to Equation 35. In this regard, i and j may indicate an x-componentand a y-component of coordinates of a pixel. In addition, σ1[i,j], σ2[i,j] σ3[i,j], σ5 [i,j], andσ6 [i,j] may be respectively s1, s2, s3, s5, and s6 of Equation 33.
[00479] [Equation 35]
[00480]
[00481] FIG. 8D is a reference diagram for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to a pixel group, according to an embodiment.
[00482] Referring to FIG. 8D, the video decoding apparatus 100 may determine σ1[i,j],σ2[i,j] σ3[i,j], σ5 [i,j], and σ6 [i,j] with respect to each of pixels included in each of pixelgroups 840, according to Equation 35 described above.
[00483] The video decoding apparatus 100 may determine a horizontal directiondisplacement vector Vx[i,j] with respect to each of the pixel groups 840 by using σ1[i,j]and σ3[i,j] of pixels, according to Equation 36. In this regard, i and j may indicate anx-component and a y-component of upper-left coordinates of a pixel group.
[00484] [Equation 36]
[00485]
[00486] The video decoding apparatus 100 may calculate a vertical directiondisplacement vector of the pixel group by using a similar method of calculating ahorizontal direction displacement vector. That is, the vertical direction displacementvector of the pixel group may be calculated by adding variables to a denominator, thevariables being used by the denominator with respect to each of pixels so as to calculatethe vertical direction displacement vector of the pixel group, and by adding variables to anumerator, the variables being used by the numerator with respect to each of the pixelsso as to calculate the vertical direction displacement vector of the pixel group.
[00487] The video decoding apparatus 100 may determine a vertical directiondisplacement vector Vy[i,j] with respect to each of the pixel groups 840 by using σ2[i,j],σ5 [i,j], σ6 [i,j], and Vx[i,j] of the pixels, according to Equation 37. In this regard, thehorizontal direction displacement vector Vx[i,j] may have been determined according toEquation 36.
[00488] [Equation 37]
[00489]
[00490] As illustrated in FIG. 8C, when the horizontal direction displacement vector andthe vertical direction displacement vector are determined with respect to each pixel, adivision operation has to be performed twice on each pixel. However, as illustrated inFIG. 8D, in a case where the horizontal direction displacement vector and the verticaldirection displacement vector are determined with respect to each group, when a size ofa pixel group is LxL (where, L is an integer), 2 / (L^2) division operations have to beperformed on each pixel. Therefore, when the horizontal direction displacement vectorand the vertical direction displacement vector are determined with respect to each group,a number of required division operations is decreased such that hardware complexitymay be decreased and calculation time may be decreased.
[00491] Hereinabove, it is described in detail with reference to FIG. 8D that a pixelgroup has a 2x2 size, but the disclosure is not limited thereto, and thus the pixel groupmay have an LxL size (where, L is an integer).
[00492] In this regard, a size L of the pixel group may be determined according toEquation 38. W and H may respectively refer to a width and a height of a current block.
[00493] [Equation 38]
[00494]
[00495] When the video decoding apparatus 100 performs pixel group unit motioncompensation along with block enlargement by a window size, and performs only thepixel group unit motion compensation, the video decoding apparatus 100 may performmemory access operations and multiplication operations as much as the number ofmemory accesses the number of multiplication operations of each case according toTable 1 below. In this regard, it is assumed that a length T of a signal filter is 8, a length Tof a gradient filter is 6, a size of a pixel group is LxL, a size of a block is NxN, and awindow size of each pixel 2M+1 is 5.
[00496]
[00497] [Table 1]
[00498] That is, when the video decoding apparatus 100 performs pixel group unitmotion compensation, the video decoding apparatus 100 may perform divisionoperations by decreasing the number of division operations by a value obtained bydividing a value by L^2, the value corresponding to a size of a pixel group.
[00499] FIG. 9A is a diagram for describing processes of adding an offset value afterfiltering is performed, and determining a gradient value in a horizontal or verticaldirection by performing de-scaling, according to an embodiment.
[00500] Referring to FIG. 9A, the video decoding apparatus 100 may determine agradient value in a horizontal or vertical direction by performing filtering on a pixel, inwhich a predetermined direction component is at an integer location, by using a first 1Dfilter and a second 1D filter. However, a value obtained by performing the filtering on thepixel, in which the predetermined direction component is at the integer location, by usingthe first 1D filter or the second 1D filter may exceed a predetermined range. Thisphenomenon is referred to as an overflow phenomenon. Coefficients of a 1D filter maybe determined to be integers for performing an integer operation, instead of aninaccurate and complicated fractional operation. The coefficients of the 1D filter may bescaled to be determined as the integers. When filtering is performed by using the scaledcoefficients of the 1D filter, it is possible to perform an integer operation, but compared toa case where filtering is performed by using not-scaled coefficients of the 1D filter, avalue on which the filtering has been performed may be increased such that the overflowphenomenon may occur. Accordingly, to prevent the overflow phenomenon, de-scalingmay be performed after the filtering is performed by using the 1D filter. In this regard, thede-scaling may include bit-shifting to the right by the number of de-scaling bits. Thenumber of de-scaling bits may be determined based on a maximum bit number of aregister for a filtering operation and a maximum bit number of a temporal buffer thatstores a result of the filtering operation, while maximizing accuracy of calculation. Inparticular, the number of de-scaling bits may be determined based on an internal bitdepth, a scaled bit number for an interpolation filter, and a scaled bit number for agradient filter.
[00501] Hereinafter, performing of de-scaling during processes of generating aninterpolation filtered value in a vertical direction by first performing filtering on a pixel atan integer location by using an interpolation filter in the vertical direction so as todetermine a gradient value in a horizontal direction and then performing filtering on theinterpolation filtered value in the vertical direction by using the gradient filter in thehorizontal direction will be described.
[00502] According to Equation 12 described above, the video decoding apparatus 100may first perform filtering on a pixel at an integer location by using an interpolation filterin a vertical direction so as to determine a gradient value in a horizontal direction. In thisregard, shift1 may be determined to be b-8. In this regard, b may indicate an internal bitdepth of an input image. Hereinafter, with reference to Table 2, a bit depth (Reg Bitdepth)of a register and a bit depth (Temp Bitdepth) of a temporary buffer when de-scaling isactually performed based on shift1 will now be described.
[00503] [Table 2]
[00504] In this regard, values of variables in Table 2 may be determined according toEquation 39.
[00505] [Equation 39]
[00506]
[00507] In this regard, Min(I) may indicate a minimum value of a pixel value Idetermined by an internal bit depth, and Max(I) may indicate a maximum value of thepixel value I determined by the internal bit depth. FilterSumPos indicates a maximumvalue of the sum of positive filter coefficients, and FilterSumNeg indicates a minimumvalue of the sum of negative filter coefficients.
[00508] For example, when it is assumed that a gradient filter FracFilter in a 1 / 4 pel unitin FIG. 7C is used, FilterSumPos may be 88 and FilterSumNeg may be -24.
[00509] A function Ceiling(x) may be a function that outputs a smallest integer fromamong integers equal to or greater than x, with respect to a real number x. offset1 is anoffset value added to a value on which filtering has been performed so as to prevent around-off error that may occur while de-scaling is performed by using shift1, and offset1may be determined to be 2^(shift1 - 1).
[00510] Referring to Table 2, when the internal bit depth b is 8, the bit depth (RegBitdepth) of the register may be 16, when the internal bit depth b is 9, the bit depth of theregister may be 17, and when the internal bit depth b is 10, 11, 12, and 16, the bit depthof the register may be 18, 19, 20, and 24. When a register used to perform filtering is a32-bit register, because bit depths of all registers in Table 2 do not exceed 32, anoverflow phenomenon does not occur.
[00511] Equally, when the internal bit depths b are 8, 9, 10, 11, 12, and 16, the bitdepths (Temp BitDepth) of the temporary buffers are all 16. When a temporary bufferused to store a value on which filtering is performed and then de-scaling is performed isa 16-bit buffer, because bit depths of all temporary buffers in Table 2 are 16 and thus donot exceed 16, an overflow phenomenon does not occur.
[00512] According to Equation 12, the video decoding apparatus 100 may generate aninterpolation filtered value in a vertical direction by first performing filtering on a pixel atan integer location by using a vertical direction interpolation filter so as to determine agradient value in a horizontal direction, and then may perform filtering on theinterpolation filtered value in the vertical direction by using a gradient filter in thehorizontal direction, according to Equation 13. In this regard, shift2 may be determined tobe p + q - shift1. In this reg ard, p may indicate bit number scaled with respect to aninterpolation filter including the filter coefficients illustrated in FIG. 7C, and q may indicatea bit number scaled with respect to a gradient filter including filter coefficients illustratedin FIG. 7A. For example, p may be 6 and q may be 4, and therefore, shift2=18 - b.
[00513] The reason why shift2 is determined as described above is becauseshift1+shift2, i.e., the total sum of the number of de-scaled bits, should be equal to thesum (p + q) of the number of bits up-scaled with respect to a filter such that, even when afilter coefficient is up-scaled and when the filter coefficient is not up-scaled, final filteringresult values are the same.
[00514] Hereinafter, with reference to Table 3, a bit depth (Reg Bitdepth) of a registerand a bit depth (Temp Bitdepth) of a temporary buffer when de-scaling is actuallyperformed based on shift2 will now be described.
[00515] [Table 3]
[00516] In this regard, values of variables in Table 3 may be determined according toEquation 36.
[00517] [Equation 40]
[00518]
[00519] In this regard, TempMax may indicate TempMax of Table 3 and TempMin mayindicate TempMin of Table 3. FilterSumPos indicates a maximum value of the sum ofpositive filter coefficients and FilterSumNeg indicates a minimum value of the sum ofnegative filter coefficients. For example, when it is assumed that a gradient filtergradFilter in a 1 / 4 pel unit illustrated in FIG. 7C is used, FilterSumPos may be 68 andFilterSumNeg may be -68.
[00520] offset2 is an offset value added to a value on which filtering has beenperformed so as to prevent a round-off error that may occur while de-scaling isperformed by using shift2, and offset2 may be determined to be 2^(shift2 - 1).
[00521] shift1 and shift2 may be determined as described above, but the disclosure isnot limited thereto, and thus, shift1 and shift2 may be variously determined in such amanner that the sum of shift1 and shift2 may be equal to the sum of scaled bit numberswith respect to a filter. In this regard, values of shift1 and shift2 may be determined basedon the premise that an overflow phenomenon does not occur. shift1 and shift2 may bedetermined based on an internal bit depth of an input image and a scaled bit number withrespect to a filter.
[00522] However, shift1 and shift2 may not be necessarily determined to allow the sumof shift1 and shift2 to be equal to the sum of scaled bit numbers with respect to a filter. Forexample, shift1 may be determined to be d-8, but shift2 may be determined to be a fixednumber.
[00523] When shift1 is the same as previous and shift2 is a fixed number of 7, OutMax,OutMin, and Temp Bitdepth described with reference to Table 3 may be changed.Hereinafter, a bit depth (Temp Bitdepth) of a temporary buffer will now be described withreference to Table 4.[Table 4]
[00524] Unlike Table 3, in Table 4, the bit depths (Temp Bitdepth) of the temporarybuffers are equal as 16 in all b, and when it is assumed that resultant data is stored byusing a 16-bit temporary buffer, the bit depth (Temp Bitdepth) of the temporary buffer issmaller than 16 such that an overflow phenomenon does not occur with respect tointernal bit depths of all input images. On the other hand, referring to Table 3, in a casewhere internal bit depths of input images are 12 and 16, when it is assumed thatresultant data is stored by using a 16-bit temporary buffer, the bit depth (Temp Bitdepth)of the temporary buffer is greater than 16 such that an overflow phenomenon may occur.
[00525] When shift2 is a fixed number, a resultant value obtained by performing filteringwithout using a scaled filter coefficient may be different from a resultant value obtainedby performing filtering by using the scaled filter coefficient and then performingde-scaling. In this case, it would be obvious to one of ordinary skill in the art that it isrequired to additionally perform de-scaling.
[00526] Hereinabove, performing of de-scaling during processes of generating aninterpolation filtered value in a vertical direction by first performing filtering on a pixel atan integer location by using the vertical direction interpolation filter so as to determine agradient value in a horizontal direction, and then performing filtering on the interpolationfiltered value in the vertical direction by using a gradient filter in the horizontal directionhas been described, but it would be obvious to one of ordinary skill in the art thatde-scaling may be performed in a similar manner when filtering is performed on a pixel,in which a predetermined direction component is an integer, so as to determine gradientvalues in horizontal and vertical directions via a combination of various 1D filters.
[00527] FIG. 9B is a diagram for describing processes of adding an offset value afterfiltering is performed, and determining a gradient value in a horizontal or verticaldirection by performing de-scaling, according to another embodiment.
[00528] Referring to FIG. 9B, the video decoding apparatus 100 may perform filteringbased on an input of fractional pixels and integer pixels of a reference pixel. In thisregard, it is assumed that the fractional pixels of the reference picture have beendetermined by applying a 1D filter in horizontal and vertical directions on the integerpixels of the reference pixel.
[00529] The video decoding apparatus 100 may determine a gradient value in ahorizontal direction or a vertical direction by performing filtering on a pixel, in which apredetermined directional location component is a fraction, and the integer pixels, byusing the 1D filter in the horizontal direction or the vertical direction. However, a valueobtained by performing filtering, by using a first 1D filter, on the pixel, in which thepredetermined directional location component is a fraction, and the integer pixel, mayexceed a predetermined range. This phenomenon is referred to as an overflowphenomenon. Coefficients of the 1D filter may be determined to be integers forperforming an integer operation, instead of an inaccurate and complicated fractionaloperation. The coefficients of the 1D filter may be scaled to be determined as theintegers. When filtering is performed by using the scaled coefficients of the 1D filter, it ispossible to perform an integer operation, but compared to a case where filtering isperformed by using not-scaled coefficients of the 1D filter, a value on which the filteringhas been performed may be increased such that the overflow phenomenon may occur.Accordingly, to prevent the overflow phenomenon, de-scaling may be performed afterthe filtering is performed by using the 1D filter. In this regard, the de-scaling may includebit-shifting to the right by the number of de-scaling bits (shift1). The number of de-scalingbits may be determined based on a maximum bit number of a register for a filteringoperation and a maximum bit number of a temporal buffer that stores a result of thefiltering operation, while maximizing accuracy of calculation. In particular, the number ofde-scaling bits may be determined based on an internal bit depth and a scaled bitnumber for a gradient filter.
[00530] FIG. 9C is a diagram for describing a necessary range for determining ahorizontal direction displacement vector and a vertical direction displacement vectorduring processes of performing pixel-unit motion compensation on a current block.
[00531] Referring to FIG. 9C, during processes of performing pixel-unit motioncompensation on a reference block 910 corresponding to the current block, the videodecoding apparatus 100 may determine a displacement vector per unit time in ahorizontal direction and a displacement vector per unit time in a vertical direction in apixel 915 by using a window 920 around the pixel 915 located in the upper-left in thereference block 910. In this regard, the displacement vector per unit time in thehorizontal direction or the vertical direction may be determined by using a pixel value anda gradient value of a pixel located in a range exceeding the reference block 910. Equally,during processes of determining a horizontal direction displacement vector and a verticaldirection displacement vector with respect to a pixel located at a boundary of thereference block 910, the video decoding apparatus 100 may determine a pixel value anda gradient value of a pixel located in a range exceeding the reference block 910.Therefore, the video decoding apparatus 100 may determine a horizontal directiondisplacement vector and a vertical direction displacement vector by using a block 925 ofwhich range is larger than the reference block 910. For example, when a size of thecurrent block is AxB, and a size of a window for each pixel is (2M+1)x(2N+1), the size ofa range for determining a horizontal direction displacement vector and a vertical directiondisplacement vector may be (A+2M)x(B+2N).
[00532] FIGS. 9D to 9E are diagrams for determining a range of a region used inprocesses of performing pixel-unit motion compensation, according to variousembodiments.
[00533] Referring to FIG. 9D, during processes of performing pixel-unit motioncompensation, the video decoding apparatus 100 may determine a horizontal directiondisplacement vector and a vertical direction displacement vector with respect to each ofpixels included in a reference block 930, based on a block 935 having a range that isenlarged by a size of a window of a pixel located at a boundary of the reference block930. However, when a horizontal direction displacement vector and a vertical directiondisplacement vector are determined, the video decoding apparatus 100 requires a pixelvalue and a gradient value of a pixel located in the block 935, and to obtain the pixelvalue and the gradient value, the video decoding apparatus 100 may use aninterpolation filter or a gradient filter. During a process of using the interpolation filter orthe gradient filter for the boundary pixel of the block 935, a pixel value of a neighboringpixel may be used, and thus, a pixel located outside a block boundary may be used.Therefore, pixel-unit motion compensation may be performed by using a block 940 ofwhich range is additionally enlarged based on a value obtained by subtracting 1 from thenumber of taps of the interpolation filter or the gradient filter. Therefore, in a casewhere a size of a block is NxN, a size of a window for each pixel is (2M+1)x(2M+1), anda length of the interpolation filter or the gradient filter is T, (N+2M+T-1)x(N+2M+T-1) maybe possible.
[00534] Referring to FIG. 9E, during processes of performing pixel-unit motioncompensation, the video decoding apparatus 100 may determine a horizontal directiondisplacement vector and a vertical direction displacement vector with respect to each ofpixels by using a pixel value and a gradient value of a pixel located in a reference block945, without enlarging the reference block 945 based on a size of a window of a pixellocated at a boundary of the reference block 945. Particular processes, performed by thevideo decoding apparatus 100, of determining a displacement vector per unit time in ahorizontal direction and a displacement vector per unit time in a vertical direction withoutenlarging a reference block will be described with reference to FIG. 9E. In this regard, aninterpolation filter or a gradient filter may be used with respect to the reference block 945so as to obtain the pixel value and the gradient value of the pixel, and pixel-unit motioncompensation may be performed by using an enlarged block 950. Therefore, in a casewhere a size of a block is NxN, a size of a window for each pixel is (2M+1)x(2M+1), anda length of the interpolation filter or the gradient filter is T, (N+T-1)x(N+T-1) may bepossible.
[00535] FIG. 9F is a diagram for describing processes of determining a horizontaldirection displacement vector and a vertical direction displacement vector, withoutenlarging a reference block.
[00536] Referring to FIG. 9F, for a pixel located outside a boundary of a reference block955, the video decoding apparatus 100 may adjust a location of the pixel to a location ofan available pixel that is closest to the pixel and is from among pixels located within theboundary of the reference block 955, and may determine a pixel value and a gradientvalue of the pixel located outside the boundary to be a pixel value and a gradient value ofthe available pixel at the closest location. In this regard, the video decoding apparatus100 may adjust the location of the pixel located outside the boundary of the referenceblock 955 to the location of the available pixel at the closest location according to anequation:(Equation) and an equation:(Equation).
[00537] In this regard, i' may indicate an x-coordinate value of a pixel, j' may indicate ay-coordinate value of the pixel, and H and W may indicate a height and a width of areference block. In this regard, an upper-left location in the reference block is assumedto be (0, 0). When the upper-left location in the reference block is (xP, yP), a location of afinal pixel may be (i'+xP, j'+yP).
[00538] Referring back to FIG. 9D, locations of pixels that are in the block 935 enlargedby a size of a window of each pixel and are located outside the boundary of thereference block 930 may be adjusted to locations of pixels adjacent to the inside of theboundary of the reference block 930, and thus, as illustrated in FIG. 9E, the videodecoding apparatus 100 may determine a horizontal direction displacement vector and avertical direction displacement vector with respect to each of pixels in the referenceblock 945 by using a pixel value and a gradient value of the reference block 945.
[00539] Therefore, the video decoding apparatus 100 performs pixel-unit motioncompensation without enlarging the reference block 945 by a size of a window of eachpixel, such that the number of memory accesses for referring to a pixel value may bedecreased, the number of multiplication operations may be decreased, and thuscomplexity in operations may be decreased.
[00540] When the video decoding apparatus 100 performs block-unit motioncompensation (as in an operation according to the High Efficiency Video Coding (HEVC)standard), performs pixel-unit motion compensation with block enlargement by a windowsize, and performs the pixel-unit motion compensation without the block enlargement,the video decoding apparatus 100 may perform memory access operations andmultiplication operations as much as the number of memory accesses the number ofmultiplication operations of each case according to Table 5 below. In this regard, it isassumed that a length of a signal (interpolation) filter is 8, a length of a gradient filter is 6,a size of a block is NxN, and a window size of each pixel 2M+1 is 5.
[00541] [Table 5]
[00542] In the block-unit motion compensation according to the HEVC standard, an8-tap interpolation filter is used for one sample, and therefore, 8 neighboring samplesare required. Accordingly, when a size of a reference block is NxN, (N+7)x(N+7)reference samples are required for 8-tap interpolation, and because bi-directional motionprediction compensation is to be performed, two reference blocks are to be used, andtherefore, in the block-unit motion compensation according to the HEVC standard,2*(N+7)x(N+7) times of memory accesses are required as in Table 4. In a case wherethe pixel-unit motion compensation is performed with block enlargement, in M=2, whenthe pixel-unit motion compensation is performed on a block having an enlarged size of(N+4)x(N+4) by using an 8-tap interpolation filter or a grandient filter, (N+4+7)x(N+4+7)reference samples are required, and because bi-directional motion predictioncompensation is to be performed, two reference blocks are to be used, and therefore, inthe pixel-unit motion compensation performed with block enlargement,2*(N+4+7)x(N+4+7) times of memory accesses are required as in Table 5.
[00543] In a case where the pixel-unit motion compensation is performed without blockenlargement, because there is no enlargement with respect to a block, (N+7)x(N+7)reference samples are required as in the block-unit motion compensation according tothe HEVC standard, and because bi-directional motion prediction compensation is to beperformed, two reference blocks are to be used, and therefore, in the pixel-unit motioncompensation performed without block enlargement, 2*(N+7)x(N+7) times of memoryaccesses are required as in Table 5.
[00544] In the block-unit motion compensation according to the HEVC standard, the8-tap interpolation filter is used for one sample, and therefore, a sample required toperform first interpolation in a horizontal direction is (N+7)xN sample. A sample requiredto perform second interpolation in a vertical direction is NxN sample. The number ofmultiplication operations required for each 8-tap interpolation filter is 8, and becausebi-directional motion prediction compensation is to be performed, two reference blocksare to be used, and therefore, in the block-unit motion compensation according to theHEVC standard, 2*8*{(N+7)*N+N*N) times of multiplication operations are required as inTable 5.
[00545] In a case where the pixel-unit motion compensation is performed with blockenlargement, a size of a block is enlarged to perform the pixel-unit motion compensation,and therefore, to determine a pixel value at a fractional pixel location by using the 8-tapinterpolation filter with respect to the block having the enlarged size of (N+4)x(N+4), thetotal sum of 2*8*{(N +4+7) x (N+4) + (N+4) x (N+4)} times of multiplication operations arerequired as in Table 5.
[00546] In a case where the pixel unit-motion compensation is performed with blockenlargement, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine agradient value in a horizontal direction or a vertical direction. Because the size of theblock has been enlarged, the total sum of 2*6*{(N+4+5)x(N+4)+(N+4)x(N+4)}*2 times ofmultiplication operations are required as in Table 5 so as to determine the gradient valueby using the 6-tap gradient filter and the 6-tap interpolation filter with respect to the blockhaving the enlarged size of (N+4)x(N+4).
[00547] In a case where the pixel-unit motion compensation is performed without blockenlargement, because there is no block enlargement, (N+7)x(N+7) reference samplesare required as in the block-unit motion compensation according to the HEVC standard,and because bi-directional motion prediction compensation is to be performed, tworeference blocks are to be used, and therefore, in the pixel-unit motion compensationperformed without block enlargement, the total sum of 2*8*{(N+7) x N + N x N } times ofmultiplication operations are required to determine the pixel value at the fractional pixellocation by using the 8-tap interpolation filter with respect to a block with a NxN size.
[00548] When the pixel-unit motion compensation is performed without blockenlargement, a 6-tap gradient filter and a 6-tap interpolation filter are used to determine agradient value in a horizontal direction or a vertical direction. To determine a gradientvalue by using the 6-tap gradient filter and the 6-tap interpolation filter with respect to theblock with a NxN size, the total sum of 2*6*{(N+5)xN+NxN}*2 times of multiplicationoperations are required as in Table 5.
[00549] FIGS. 9G to 9I are diagrams for describing processes of determining ahorizontal direction displacement vector and a vertical direction displacement vectorwithout enlarging a reference block, according to other embodiments.
[00550] As described above with reference to FIG. 9D, during processes of performingpixel-unit motion compensation, the video decoding apparatus 100 may determine ahorizontal direction displacement vector and a vertical direction displacement vector foreach of pixels included in a reference block 930, based on a block 935 of which range isenlarged by a size of a window of a pixel located at a boundary of the reference block930. For example, in a case where the size of the window is (2M+1)x(2M+1), when awindow is applied to the pixel located at the boundary of the reference block 930, thevideo decoding apparatus 100 may determine a horizontal direction displacement vectorand a vertical direction displacement vector for each of the pixels by referring to a pixelvalue and a gradient value of a pixel exceeding the reference block 930 by M.
[00551] Hereinafter, a method according to another embodiment, performed by thevideo decoding apparatus 100, of determining values (s1 to s6 of Equation 33) fordetermining a horizontal direction displacement vector and a vertical directiondisplacement vector with respect to each pixel by using only a pixel value and a gradientvalue of a reference block corresponding to a current block, without referring to a pixelvalue and a gradient value of a pixel outside the reference block, will now be described.Here, it is assumed that a size of a window is 5x5. For convenience, only pixels locatedin a horizontal direction with respect to a current pixel will now be described. It would beobvious to one of ordinary skill in the art that a weight may be equally determined withrespect to pixels located in a vertical direction with respect to the current pixel.
[00552] The video decoding apparatus 100 performs a calculation on a function aboutpixel values (P0(I',j') and P1(I',j')), a gradient value in a horizontal or vertical direction( (Equation),(Equation) ,(Equation) , and(Equation) ),and a difference (τ0, τ1) between POCs of a reference picture and a current picture withrespect to each of pixels included in a window with respect to a current pixel for which ahorizontal direction displacement vector or a vertical direction displacement vector is tobe determined, and determines the values (s1 to s6 of Equation 33) for determining ahorizontal direction displacement vector and a vertical direction displacement vector withrespect to each pixel by adding resultant values of the calculation performed on eachpixel.
[00553] In this regard, the video decoding apparatus 100 may determine the values (s1to s6 of Equation 33) for determining a horizontal direction displacement vector and avertical direction displacement vector with respect to each pixel by allocating a sameweight to each pixel, multiplying the weight by each of the resultant values of thecalculation performed on each pixel, and then by summing values thereof.
[00554] Referring to FIG. 9G, during a process of determining a horizontal directiondisplacement vector and a vertical direction displacement vector with respect to a currentpixel 961 in a current block 960, the video decoding apparatus 100 may determine pixelsin a window to have a same weight of 1. The video decoding apparatus 100 maydetermine values (s1 to s6 of Equation 33) for determining a horizontal directiondisplacement vector and a vertical direction displacement vector with respect to a currentpixel by multiplying weights determined for respective pixels by the resultant values ofrespective calculations for respective pixels and then by summing resultant valuesthereof.
[00555] Referring to FIG. 9H, when a current pixel 971 is immediately adjacent to aboundary of a current block 970 in the current block 970, the video decoding apparatus100 may determine a weight for pixel to be 3, the pixel being located inside the currentblock 970 while being adjacent to the boundary of the current block 970. The videodecoding apparatus 100 may determine a weight for other pixels 973 to be 1.
[00556] Referring to FIG. 9H, when a current pixel 981 is near a boundary of a currentblock 980 in the current block 980 (when the current pixel 981 is distant from theboundary by one pixel), the video decoding apparatus 100 may determine a weight forpixels 982 to be 0, the pixels being located outside the boundary of the current block 980,and may determine a weight for pixels 983 to be 2, the pixels being immediately adjacentto the boundary of the current block 980. The video decoding apparatus 100 maydetermine a weight for other pixels 984 to be 1.
[00557] As described with reference to FIGS. 9G to 9I, the video decoding apparatus100 may determine values (s1 to s6 of Equation 33) for determining a horizontal directiondisplacement vector and a vertical direction displacement vector with respect to eachpixel by using a pixel value and a gradient value of pixel located in a reference block,without using a pixel value and a gradient value of a pixel outside the reference blockcorresponding to a current block, by allocating different weights to pixels in a window,according to locations of a current pixel.
[00558] FIG. 9J is a diagram for describing processes of determining a horizontaldirection displacement vector and a vertical direction displacement vector with respect toeach pixel by applying an exponential smoothing technique in vertical and horizontaldirections by referring to a pixel value and a gradient value of a reference block, withoutenlargement of a block, according to an embodiment.
[00559] Referring to FIG. 9J, the video decoding apparatus 100 may perform acalculation on a function about pixel values (P0(I',j') and P1(I',j')) of a correspondingreference pixel included in a corresponding reference block, a gradient value in ahorizontal or vertical direction ((Equation) ,(Equation) ,(Equation), and(Equation) ) of the corresponding reference pixelincluded in a pixel of the corresponding reference block, and a difference (τ0, τ1)between POCs of a reference picture and a current picture with respect to each pixelincluded in a current block 990.
[00560] The video decoding apparatus 100 may determine values (s1 to s6;σk(k=1,2,3,4,5,6)) for determining a horizontal direction displacement vector and a verticaldirection displacement vector with respect to a current pixel by summing a resultantvalue of a function calculation for the current pixel and a resultant value of a functioncalculation for a neighboring pixel. That is, the values (s1 to s6) for determining thehorizontal direction displacement vector and the vertical direction displacement vectorwith respect to the current pixel may be expressed as a weighted average of values bycalculations with respect to the current pixel and the neighboring pixel according toEquation 41 below. In this regard, coordinates of locations of pixels included in a windowΩ are (i',j'). Also, W[i',j'] may indicate a weight for the pixels included in the window Ω. Inthis regard, a size of the window Ω may be (2M+1)x(2M+1)(where, M is an integer). Also,a value of function Ak[i',j'] may be a value of a function about pixel values (P0(I',j') andP1(I',j');I[I', j'](0,1)) of a corresponding reference pixel with respect to pixels at a (I',j'')location included in the window Ω, a gradient value in a horizontal or vertical direction((Equation) ,(Equation) ,(Equation) , and(Equation);Ix,y[i',j'](0,1)) of the corresponding reference pixel included in a pixelof the corresponding reference block, and the difference (τ0, τ1) between POCs of thereference picture and the current picture.
[00561] [Equation 41]
[00562] (Equation)
[00563] For example, the video decoding apparatus 100 may determine values (s1 tos6) for determining a horizontal direction displacement vector and a vertical directiondisplacement vector with respect to the current pixel by changelessly summing aresultant value of a function calculation for the current pixel and resultant values offunction calculations for pixels in a window with respect to the current pixel. When it isassumed that a size of the window is 5x5, it is required to refer to resultant values ofcalculations about pixels in a window 992 with respect to a pixel 991 so as to determinevalues required to determine a horizontal direction displacement vector and a verticaldirection displacement vector of the pixel 991 located at a boundary of a current block990, and thus, it is required to refer to resultant values of function calculations withrespect to pixels of a block 996 that is enlarged by 2 pixels in horizontal and verticaldirections, compared to the current block 990. The value of function Ak[i',j'] may bedetermined as Table 6 below.
[00564] [Table 6]
[00565] In this regard, Ix(0) [i',j'] and Ix(1) [i',j'] may be respectivelyand(Equation) , Iy(0) [i',j'] and Iy(1) [i',j'] may be respectivelyand (Equation), and I(0) [i',j'] and I(1) [i',j'] may be respectively P0(I',j') and P1(I',j').
[00566] The video decoding apparatus 100 may perform averaging by applying anexponential smoothing technique to Ak[i',j' ] in vertical and horizontal directions so as todetermine a weighted average with respect to Ak [i',j' ].
[00567] Referring to FIG. 9J, the video decoding apparatus 100 may apply theexponential smoothing technique to Ak[i',j' ] in the horizontal direction. Hereinafter,processes, performed by the video decoding apparatus 100, of determining a weightedaverage value with respect to Ak [i',j' ] by applying the exponential smoothing techniqueto Ak[i',j' ] in the horizontal direction will now be described in detail.
[00568] First, contents in which the video decoding apparatus 100 performs averagingin a right direction by applying the exponential smoothing technique will now bedescribed. The video decoding apparatus 100 may perform averaging on Ak[i',j' ] in theright direction by using Pseudocode 1 as below. In this regard, H may indicate a height ofa current block, W may indicate a width of the current block, and Stride may indicate adistance between a line and a next line in a 1D array. That is, two-dimensional (2D) arrayA[i,j] may be expressed as A[i+j*Stride] of 1D array.
[00569] [Pseudocode 1](Equation)
[00570] (Equation)
[00571] For example, when a size of the current block is 4x4, and A[0], A[1], A[2], andA[3] are a, b, c, and d, resultant values A'[0] of a process using Pseudocode 1 may bethat 2a, A'[1] may be A[0] / 2+b =a+b, A'[2] may be A[1] / 2+c=a / 2+b / 2+c, and A'[3] may beA[2] / 2+d=a / 4+b / 4+c / 2+d.
[00572] Hereinafter, contents in which the video decoding apparatus 100 performsaveraging in a left direction by applying the exponential smoothing technique will now bedescribed. The video decoding apparatus 100 may perform averaging on Ak[i',j' ] in theleft direction by using Pseudocode 2 as below. In this regard, H may indicate a height ofa current block, W may indicate a width of the current block, and Stride may indicate adistance between a line and a next line in a 1D array. That is, 2D array A[i,j] may beexpressed as A[i+j*Stride] of 1D array.
[00573] [Pseudocode 2](Equation)
[00574] (Equation)
[00575] For example, when a size of the current block is 4x4, and A'[0], A'[1], A'[2], andA'[3] that have been averaged in the right direction 2a, a+b, a / 2+b / 2+c, anda / 4+b / 4+c / 2+d, resultant values of a process using Pseudocode 2 may be that A[3] maybe 2*A'[3]= a / 2+b / 2+c+2d, A[2] may be A'[2]+A[3] / 2= a / 2+b / 2+c+a / 4+b / 4+c / 2+d=3a / 4+3b / 4+3c / 2+d, A[1] may be A'[1]+A[2] / 2… , and A[0] may beA'[0]+A[1] / 2.
[00576] Hereinafter, contents in which the video decoding apparatus 100 performsaveraging in a down direction by applying the exponential smoothing technique will nowbe described. The video decoding apparatus 100 may perform averaging on Ak[i',j' ] inthe down direction by using Pseudocode 3 as below. In this regard, H may indicate aheight of a current block, W may indicate a width of the current block, and Stride mayindicate a distance between a line and a next line in a 1D array. That is, 2D array A[i,j]may be expressed as A[i+j*Stride] of 1D array.
[00577] [Pseudocode 3](Equation)
[00578]
[00579] Hereinafter, contents in which the video decoding apparatus 100 performsaveraging in an up direction by applying the exponential smoothing technique will nowbe described. The video decoding apparatus 100 may perform averaging on Ak[i',j' ] inthe up direction by using Pseudocode 4 as below. In this regard, H may indicate a heightof a current block, W may indicate a width of the current block, and Stride may indicate adistance between a line and a next line in a 1D array. That is, 2D array A[i,j] may beexpressed as A[i+j*Stride] of 1D array.
[00580] [Pseudocode 4](Equation)
[00581]
[00582] That is, the video decoding apparatus 100 may perform averaging on Ak[i',j'] inthe vertical and horizontal directions by using the exponential smoothing technique, anda resultant value of each pixel which is determined by performing the averaging maybecome σk [i,j] that is a value required to determine a horizontal direction displacementvector or a vertical direction displacement vector with respect to each pixel. In this regard,when the averaging is performed by using the exponential smoothing technique, acurrent pixel has a largest weight with respect to a location of the current pixel, and as apixel becomes distant from the current pixel, a weight with respect to the pixel isexponentially decreased.
[00583] Therefore, the video decoding apparatus 100 may determine σk[i,j] that is aweighted average required to determine a horizontal direction displacement vector or avertical direction displacement vector with respect to each pixel, by performing theaveraging on the current block 990 in the vertical and horizontal directions by using theexponential smoothing technique. That is, the video decoding apparatus 100 maydetermine a horizontal direction displacement vector or a vertical direction displacementvector with respect to each pixel by referring to only a pixel value and a gradient value ofa reference block corresponding to the current block 990, without referring to a pixelvalue and a gradient value of a reference block corresponding to a block 996 enlargedfrom the current block 990.
[00584] FIG. 9K is a diagram for describing processes of determining a pixel value andgradient values in horizontal and vertical directions of a reference pixel in a referenceblock by using a filter so as to perform motion compensation on a current block,according to an embodiment.
[00585] Referring to FIG. 9K, the video decoding apparatus 100 may perform pixel-unitand block-unit motion compensations on the current block by using a pixel value andgradient values of a reference pixel in a reference block corresponding to the currentblock. Therefore, to perform the pixel-unit and block-unit motion compensations on thecurrent block, the pixel value and the gradient values of the reference pixel in thereference block corresponding to the current block have to be determined. In this regard,a unit of the pixel value and the gradient values of the reference pixel in the referenceblock may be a fractional pixel unit. For example, the unit of the pixel value and thegradient values of the reference pixel in the reference block may be a 1 / 16 pel unit.
[00586] The video decoding apparatus 100 may perform filtering on a pixel value of aninteger pixel in the reference block so as to determine the pixel value and the gradientvalues in fractional pixel units of the reference pixel in the reference block.
[00587] First, the video decoding apparatus 100 may determine the pixel value of thereference pixel in the reference block by applying a horizontal 8-tap signal filter (alsoreferred to as an interpolation filter) and a vertical 8-tap signal filter to the pixel value ofthe integer pixel in the reference block.
[00588] The video decoding apparatus 100 may determine a pixel value of thereference pixel, which has a location component in a fractional pixel unit in a horizontaldirection, by performing filtering by applying the horizontal direction 8-tap signal filter to apixel value of the integer pixel in the reference block, and may store the determined pixelvalue in a buffer. The video decoding apparatus 100 may determine a pixel value of thereference pixel, which has a location component in a fractional pixel unit in a verticaldirection, by applying the vertical direction 8-tap signal filter to a pixel value of thereference pixel which has a location component in an integer unit in the vertical direction.
[00589] The video decoding apparatus 100 may determine a gradient value in thehorizontal direction of the reference pixel in the reference block by performing filtering byfirst applying a vertical direction 6-tap signal filter to the pixel value of the integer pixel inthe reference block and then applying a horizontal direction 6-tap gradient filter thereto.
[00590] That is, the video decoding apparatus 100 may determine the pixel value of thereference pixel, which has the location component in the fractional pixel unit in thevertical direction, by performing filtering by applying the vertical direction 6-tap signalfilter to the pixel value of the integer pixel in the reference block, and may store thedetermined pixel value in the buffer.
[00591] The video decoding apparatus 100 may determine the gradient value in thehorizontal direction of the reference pixel, which has the location component in thefractional pixel unit in the horizontal direction, by applying the horizontal direction 6-tapgradient filter to a pixel value of the reference pixel which has a location component in aninteger unit in the horizontal direction.
[00592] The video decoding apparatus 100 may determine a gradient value in thehorizontal direction of the reference pixel in the reference block by performing filtering byfirst applying a vertical direction 6-tap signal filter to the pixel value of the integer pixel inthe reference block and then applying a horizontal direction 6-tap gradient filter thereto.
[00593] That is, the video decoding apparatus 100 may determine the gradient value ofthe reference pixel, which has the location component in the fractional pixel unit in thevertical direction, by performing filtering by applying the vertical direction 6-tap signalfilter to the pixel value of the integer pixel in the reference block, and may store thedetermined gradient value in the buffer.
[00594] The video decoding apparatus 100 may determine the gradient value in thevertical direction of the reference pixel, which has the location component in thefractional pixel unit in the horizontal direction, by applying the horizontal direction 6-tapgradient filter to a gradient value in the vertical direction of the reference pixel which hasthe location component in the integer unit in the horizontal direction.
[00595] That is, the video decoding apparatus 100 may apply two 1D filters torespectively determine the pixel value of the reference pixel in the reference block, thegradient value in the horizontal direction of the reference pixel in the reference block, andthe gradient value in the vertical direction of the reference pixel in the reference block,and in this regard, a multiplication operation may be performed on a coefficient of eachfilter and a value about a corresponding pixel. For example, to determine the gradientvalue in the horizontal direction of the reference pixel in the reference block, the 6-tapsignal filter and the 6-tap gradient filter may be used, and 12 multiplication operationsmay be performed on each pixel. Also, to determine the gradient value in the verticaldirection of the reference pixel in the reference block, the 6-tap signal filter and the 6-tapgradient filter may be used, and 12 multiplication operations may be performed on eachpixel.
[00596] FIG. 9I is a diagram for describing processes of determining a pixel value andgradient values in horizontal and vertical directions of a reference pixel in a referenceblock by using a filter so as to perform motion compensation on a current block,according to another embodiment.
[00597] Referring to FIG. 9I, the video decoding apparatus 100 may perform pixel-unitand block-unit motion compensations on the current block by using a pixel value andgradient values of a reference pixel in a reference block corresponding to the currentblock. Therefore, to perform the pixel-unit and block-unit motion compensations on thecurrent block, the pixel value and the gradient values of the reference pixel in thereference block corresponding to the current block have to be determined. In this regard,a unit of the pixel value and the gradient values of the reference pixel in the referenceblock may be a fractional pixel unit. For example, the unit of the pixel value and thegradient values of the reference pixel in the reference block may be a 1 / 16 pel unit.
[00598] The video decoding apparatus 100 may perform filtering on a pixel value of aninteger pixel in the reference block so as to determine the pixel value and the gradientvalues in fractional pixel units of the reference pixel in the reference block.
[00599] Unlike to the descriptions about FIG. 9K, the video decoding apparatus 100may determine a gradient value in the horizontal direction of the reference pixel in thereference block by first applying a horizontal 8-tap signal filter and a vertical 8-tap signalfilter to the pixel value of the integer pixel in the reference block and then applying ahorizontal 5-tap gradient filter to the pixel value of the reference pixel in the referenceblock. Also, the video decoding apparatus 100 may determine a gradient value in thevertical direction of the reference pixel in the reference block by applying a vertical 5-tapgradient filter to the pixel value of the reference pixel in the reference block.
[00600] The video decoding apparatus 100 may determine the gradient value in thehorizontal direction or the vertical direction of the reference pixel in the reference blockby determining a pixel value of the reference pixel, which has a location of a fractionalpixel unit, by applying two 1D signal (interpolation) filters, and then by parallel applyingtwo 1D gradient filters to the pixel value of the reference pixel at the location of thefractional pixel unit.
[00601] The video decoding apparatus may determine a gradient value Ix(k) in thehorizontal direction of the reference pixel in the reference block by applying thehorizontal direction 5-tap gradient filter (filter coefficients are {9, -48, 0, 48, 9}; but thefilter coefficients are not limited thereto) to the pixel value of the reference pixelaccording to Equation 42 below. In this regard, k may have a value of one of 0 and 1 thatrespectively indicate reference pictures 0 and 1. I(k)[i,j] may be a pixel value at a (i,j)location of the reference pixel in the reference block. i may indicate a location componentin the horizontal direction of a pixel and j may indicate a location component in thevertical direction of the pixel, in which a unit thereof is a fractional pixel unit.
[00602] [Equation 42]
[00603] (Equation)
[00604] Therefore, the video decoding apparatus 100 may determine the gradient valueIx(k) in the horizontal direction of the reference pixel in the reference block by performingonly two multiplication operations on each sample.
[00605] Also, the video decoding apparatus 100 may determine the gradient value Iy(k)in the vertical direction of the reference pixel in the reference block by applying a verticaldirection 5-tap gradient filter (where filter coefficients are {9, -48, 0, 48, 9}; but are notlimited thereto) to the pixel value of the reference pixel. In this regard, k may have avalue of one of 0 and 1 that respectively indicate reference pictures 0 and 1. I(k)[i,j] maybe a pixel value at a (i,j) location of the reference pixel in the reference block. i mayindicate a location component in the horizontal direction of a pixel and j may indicate alocation component in the vertical direction of the pixel, in which a unit thereof is afractional pixel unit.
[00606] [Equation 43]
[00607] (Equation)
[00608] Therefore, the video decoding apparatus 100 may determine the gradient valueIy(k) in the vertical direction of the reference pixel in the reference block by performingonly two multiplication operations on each sample.
[00609] When the video decoding apparatus 100 performs pixel-unit motioncompensation with block enlargement by a window size, and performs the pixel-unitmotion compensation by decreasing a length of a gradient filter, the video decodingapparatus 100 may perform memory access operations and multiplication operations asmuch as the number of memory accesses the number of multiplication operations ofeach case according to Table 7 below. In this regard, it is assumed that a length T of asignal filter is 8, a length T of the gradient filter is 6, a length T of a decreased gradientfilter is 5, a size of a block is NxN, and a window size of each pixel 2M+1 is 5.
[00610] [Table 7]
[00611] That is, because the video decoding apparatus 100 performs two multiplicationoperations with respect to each 1D gradient filter according to Equations 41 and 42, thegradient filter is applied to two reference blocks, and the gradient filter is applied to areference block of (N+4)x(N+4) size that is enlarged based on a window size, a total of2*2*{(N+4)x(N+4)}*2 times of multiplication operations may be performed so as todetermine gradient values of the reference pixel in the horizontal and vertical directions.
[00612] When the video decoding apparatus 100 performs pixel-unit motioncompensation with block enlargement by a window size, and performs pixel group unitmotion compensation by decreasing a length of a gradient filter, without the blockenlargement, the video decoding apparatus 100 may perform memory accessoperations and multiplication operations as much as the number of memory accessesthe number of multiplication operations of each case according to Table 8 below. In thisregard, it is assumed that a length T of a signal filter is 8, a length T of the gradient filteris 6, a length T of a decreased gradient filter is 5, a size of a pixel group is LxL, a size ofa block is NxN, and a window size of each pixel 2M+1 is 5.
[00613] [Table 8]
[00614] Therefore, the number of memory accesses, the number of multiplicationoperations, and the number of division operations of a case where the video decodingapparatus 100 performs the pixel group unit motion compensation by decreasing alength of the gradient filter, without the block enlargement, may be significantlydecreased compared to the number of memory accesses, the number of multiplicationoperations, and the number of division operations of a case where the pixel-unit motioncompensation is performed with the block enlargement.
[00615] Hereinafter, a method of determining a data unit that may be used while thevideo decoding apparatus 100 according to an embodiment decodes an image will bedescribed with reference to FIGS. 10 through 23. Operations of the video encodingapparatus 150 may be similar to or the reverse of various embodiments of operations ofthe video decoding apparatus 100 described below.
[00616] FIG. 10 illustrates a process, performed by the video decoding apparatus 100,of determining at least one coding unit by splitting a current coding unit, according to anembodiment.
[00617] According to an embodiment, the video decoding apparatus 100 maydetermine a shape of a coding unit by using block shape information, and may determinea splitting method of the coding unit by using split shape information. That is, a codingunit splitting method indicated by the split shape information may be determined basedon a block shape indicated by the block shape information used by the video decodingapparatus 100.
[00618] According to an embodiment, the video decoding apparatus 100 may use theblock shape information indicating that the current coding unit has a square shape. Forexample, the video decoding apparatus 100 may determine whether not to split a squarecoding unit, whether to vertically split the square coding unit, whether to horizontally splitthe square coding unit, or whether to split the square coding unit into four coding units,based on the split shape information. Referring to FIG. 10, when the block shapeinformation of a current coding unit 1000 indicates a square shape, a video decoder 100may determine that a coding unit 1010a having the same size as the current coding unit1000 is not split, based on the split shape information indicating not to perform splitting,or may determine coding units 1010b, 1010c, or 1010d split based on the split shapeinformation indicating a predetermined splitting method.
[00619] Referring to FIG. 10, according to an embodiment, the video decodingapparatus 100 may determine two coding units 1010b obtained by splitting the currentcoding unit 1000 in a vertical direction, based on the split shape information indicating toperform splitting in a vertical direction. The video decoding apparatus 100 maydetermine two coding units 1010c obtained by splitting the current coding unit 1000 in ahorizontal direction, based on the split shape information indicating to perform splitting ina horizontal direction. The video decoding apparatus 100 may determine four codingunits 1010d obtained by splitting the current coding unit 1000 in vertical and horizontaldirections, based on the split shape information indicating to perform splitting in verticaland horizontal directions. However, splitting methods of the square coding unit are notlimited to the above-described methods, and the split shape information may indicatevarious methods. Predetermined splitting methods of splitting the square coding unit willbe described in detail below in relation to various embodiments.
[00620] FIG. 11 illustrates a process, performed by the video decoding apparatus 100,of determining at least one coding unit by splitting a non-square coding unit, according toan embodiment.
[00621] According to an embodiment, the video decoding apparatus 100 may use blockshape information indicating that a current coding unit has a non-square shape. Thevideo decoding apparatus 100 may determine whether not to split the non-squarecurrent coding unit or whether to split the non-square current coding unit by using apredetermined splitting method, based on split shape information. Referring to FIG. 11,when the block shape information of a current coding unit 1100 or 1150 indicates anon-square shape, the video decoding apparatus 100 may determine that a coding unit1110 or 1160 having the same size as the current coding unit 1100 or 1150 is not split,based on the split shape information indicating not to perform splitting, or determinecoding units 1120a and 1120b, 1130a to 1130c, 1170a and 1170b, or 1180a to 1180csplit based on the split shape information indicating a predetermined splitting method.Predetermined splitting methods of splitting a non-square coding unit will be described indetail below in relation to various embodiments.
[00622] According to an embodiment, the video decoding apparatus 100 maydetermine a splitting method of a coding unit by using the split shape information and, inthis case, the split shape information may indicate the number of one or more codingunits generated by splitting a coding unit. Referring to FIG. 11, when the split shapeinformation indicates to split the current coding unit 1100 or 1150 into two coding units,the video decoding apparatus 100 may determine two coding units 1120a and1120b, or1170a and 1170b included in the current coding unit 1100 or 1150, by splitting thecurrent coding unit 1100 or 1150 based on the split shape information.
[00623] According to an embodiment, when the video decoding apparatus 100 splitsthe non-square current coding unit 1100 or 1150 based on the split shape information,the location of a long side of the non-square current coding unit 1100 or 1150 may beconsidered. For example, the video decoding apparatus 100 may determine a plurality ofcoding units by dividing a long side of the current coding unit 1100 or 1150, inconsideration of the shape of the current coding unit 1100 or 1150.
[00624] According to an embodiment, when the split shape information indicates to splita coding unit into an odd number of blocks, the video decoding apparatus 100 maydetermine an odd number of coding units included in the current coding unit 1100 or1150. For example, when the split shape information indicates to split the current codingunit 1100 or 1150 into three coding units, the video decoding apparatus 100 may split thecurrent coding unit 1100 or 1150 into three coding units 1130a, 1130b, and 1130c, or1180a, 1180b, and 1180c. According to an embodiment, the video decoding apparatus100 may determine an odd number of coding units included in the current coding unit1100 or 1150, and not all the determined coding units may have the same size. Forexample, a predetermined coding unit 1130b or 1180b from among the determined oddnumber of coding units 1130a, 1130b, and 1130c, or 1180a, 1180b, and 1180c mayhave a size different from the size of the other coding units 1130a and 1130c, or 1180aand 1180c. That is, coding units which may be determined by splitting the current codingunit 1100 or 1150 may have multiple sizes and, in some cases, all of the odd number ofcoding units 1130a, 1130b, and 1130c, or 1180a, 1180b, and 1180c may have differentsizes.
[00625] According to an embodiment, when the split shape information indicates to splita coding unit into an odd number of blocks, the video decoding apparatus 100 maydetermine an odd number of coding units included in the current coding unit 1100 or1150, and may put a predetermined restriction on at least one coding unit from amongthe odd number of coding units generated by splitting the current coding unit 1100 or1150. Referring to FIG. 11, the video decoding apparatus 100 may allow a decodingmethod of the coding unit 1130b or 1180b to be different from that of the other codingunits 1130a and 1130c, or 1180a and 1180c, wherein the coding unit 1130b or 1180b isat a center location from among the three coding units 1130a, 1130b, and 1130c, or1180a, 1180b, and 1180c generated by splitting the current coding unit 1100 or 1150.For example, the video decoding apparatus 100 may restrict the coding unit 1130b or1180b at the center location to be no longer split or to be split only a predeterminednumber of times, unlike the other coding units 1130a and 1130c, or 1180a and 1180c.
[00626] FIG. 12 illustrates a process, performed by the video decoding apparatus 100,of splitting a coding unit based on at least one of block shape information and split shapeinformation, according to an embodiment.
[00627] According to an embodiment, the video decoding apparatus 100 maydetermine to split or not to split a square first coding unit 1200 into coding units, basedon at least one of the block shape information and the split shape information. Accordingto an embodiment, when the split shape information indicates to split the first coding unit1200 in a horizontal direction, the video decoding apparatus 100 may determine asecond coding unit 1210 by splitting the first coding unit 1200 in a horizontal direction. Afirst coding unit, a second coding unit, and a third coding unit used according to anembodiment are terms used to understand a relation before and after splitting a codingunit. For example, a second coding unit may be determined by splitting a first coding unit,and a third coding unit may be determined by splitting the second coding unit. It will beunderstood that the structure of the first coding unit, the second coding unit, and the thirdcoding unit follows the above descriptions.
[00628] According to an embodiment, the video decoding apparatus 100 maydetermine to split or not to split the determined second coding unit 1210 into coding units,based on at least one of the block shape information and the split shape information.Referring to FIG. 12, the video decoding apparatus 100 may or may not split thenon-square second coding unit 1210, which is determined by splitting the first coding unit1200, into one or more third coding units 1220a, or 1220b, 1220c, and 1220d based onat least one of the block shape information and the split shape information. The videodecoding apparatus 100 may obtain at least one of the block shape information and thesplit shape information, and determine a plurality of various-shaped second coding units(e.g., 1210) by splitting the first coding unit 1200, based on the obtained at least one ofthe block shape information and the split shape information, and the second coding unit1210 may be split by using the splitting method of the first coding unit 1200, based on atleast one of the block shape information and the split shape information. According to anembodiment, when the first coding unit 1200 is split into the second coding units 1210based on at least one of the block shape information and the split shape information ofthe first coding unit 1200, the second coding unit 1210 may also be split into the thirdcoding units 1220a, or 1220b, 1220c, and 1220d based on at least one of the blockshape information and the split shape information of the second coding unit 1210. That is,a coding unit may be recursively split based on at least one of the block shapeinformation and the split shape information of each coding unit. Therefore, a squarecoding unit may be determined by splitting a non-square coding unit, and a non-squarecoding unit may be determined by recursively splitting the square coding unit. Referringto FIG. 12, a predetermined coding unit from among an odd number of third coding units1220b, 1220c, and 1220d determined by splitting the non-square second coding unit1210 (e.g., a coding unit at a center location or a square coding unit) may be recursivelysplit. According to an embodiment, the square third coding unit 1220c from among theodd number of third coding units 1220b, 1220c, and 1220d may be split in a horizontaldirection into a plurality of fourth coding units. A non-square fourth coding unit 1240 fromamong the plurality of fourth coding units may be split into a plurality of coding units. Forexample, the non-square fourth coding unit 1240 may be split into an odd number ofcoding units 1250a, 1250b, and 1250c.
[00629] A method that may be used to recursively split a coding unit will be describedbelow in relation to various embodiments.
[00630] According to an embodiment, the video decoding apparatus 100 maydetermine to split each of the third coding units 1220a, or 1220b, 1220c, and 1220d intocoding units or not to split the second coding unit 1210, based on at least one of theblock shape information and the split shape information. According to an embodiment,the video decoding apparatus 100 may split the non-square second coding unit 1210 intothe odd number of third coding units 1220b, 1220c, and 1220d. The video decodingapparatus 100 may put a predetermined restriction on a predetermined third coding unitfrom among the odd number of third coding units 1220b, 1220c, and 1220d. For example,the video decoding apparatus 100 may restrict the third coding unit 1220c at a centerlocation from among the odd number of third coding units 1220b, 1220c, and 1220d tobe no longer split or to be split a settable number of times. Referring to FIG. 12, the videodecoding apparatus 100 may restrict the third coding unit 1220c, which is at the centerlocation from among the odd number of third coding units 1220b, 1220c, and 1220dincluded in the non-square second coding unit 1210, to be no longer split, to be split byusing a predetermined splitting method (e.g., split into only four coding units or split byusing a splitting method of the second coding unit 1210), or to be split only apredetermined number of times (e.g., split only n times (where n>0)). However, therestrictions on the third coding unit 1220c at the center location are not limited to theabove-described examples, and may include various restrictions for decoding the thirdcoding unit 1220c at the center location differently from the other third coding units1220b and 1220d.
[00631] According to an embodiment, the video decoding apparatus 100 may obtain atleast one of the block shape information and the split shape information, which is used tosplit a current coding unit, from a predetermined location in the current coding unit.
[00632] FIG. 13 illustrates a method, performed by the video decoding apparatus 100,of determining a predetermined coding unit from among an odd number of coding units,according to an embodiment. Referring to FIG. 13, at least one of block shapeinformation and split shape information of a current coding unit 1300 may be obtainedfrom a sample of a predetermined location from among a plurality of samples included inthe current coding unit 1300 (e.g., a sample 1340 of a center location). However, thepredetermined location in the current coding unit 1300, from which at least one of theblock shape information and the split shape information may be obtained, is not limited tothe center location in FIG. 13, and may include various locations included in the currentcoding unit 1300 (e.g., top, bottom, left, right, top left, bottom left, top right, and bottomright locations). The video decoding apparatus 100 may obtain at least one of the blockshape information and the split shape information from the predetermined location anddetermine to split or not to split the current coding unit into various-shaped andvarious-sized coding units.
[00633] According to an embodiment, when the current coding unit is split into apredetermined number of coding units, the video decoding apparatus 100 may selectone of the coding units. Various methods may be used to select one of a plurality ofcoding units, as will be described below in relation to various embodiments.
[00634] According to an embodiment, the video decoding apparatus 100 may split thecurrent coding unit into a plurality of coding units, and may determine a coding unit at apredetermined location.
[00635] FIG. 13 illustrates a method, performed by the video decoding apparatus 100,of determining a coding unit of a predetermined location from among an odd number ofcoding units, according to an embodiment.
[00636] According to an embodiment, the video decoding apparatus 100 may useinformation indicating locations of the odd number of coding units, to determine a codingunit at a center location from among the odd number of coding units. Referring to FIG. 13,the video decoding apparatus 100 may determine an odd number of coding units 1320a,1320b, and 1320c by splitting the current coding unit 1300. The video decodingapparatus 100 may determine a coding unit 1320b at a center location by usinginformation about locations of the odd number of coding units 1320a to 1320c. Forexample, the video decoding apparatus 100 may determine the coding unit 1320b of thecenter location by determining the locations of the coding units 1320a, 1320b, and 1320cbased on information indicating locations of predetermined samples included in thecoding units 1320a, 1320b, and 1320c. In detail, the video decoding apparatus 100 maydetermine the coding unit 1320b at the center location by determining the locations of thecoding units 1320a, 1320b, and 1320c based on information indicating locations of topleft samples 1330a, 1330b, and 1330c of the coding units 1320a, 1320b, and 1320c.
[00637] According to an embodiment, the information indicating the locations of the topleft samples 1330a, 1330b, and 1330c, which are included in the coding units 1320a,1320b, and 1320c, respectively, may include information about locations or coordinatesof the coding units 1320a, 1320b, and 1320c in a picture. According to an embodiment,the information indicating the locations of the top left samples 1330a, 1330b, and 1330c,which are included in the coding units 1320a, 1320b, and 1320c, respectively, mayinclude information indicating widths or heights of the coding units 1320a, 1320b, and1320c included in the current coding unit 1300, and the widths or heights maycorrespond to information indicating differences between the coordinates of the codingunits 1320a, 1320b, and 1320c in the picture. That is, the video decoding apparatus 100may determine the coding unit 1320b at the center location by directly using theinformation about the locations or coordinates of the coding units 1320a, 1320b, and1320c in the picture, or by using the information about the widths or heights of the codingunits, which correspond to the difference values between the coordinates.
[00638] According to an embodiment, information indicating the location of the top leftsample 1330a of the upper coding unit 1320a may include coordinates (xa, ya),information indicating the location of the top left sample 1330b of the middle coding unit1320b may include coordinates (xb, yb), and information indicating the location of the topleft sample 1330c of the lower coding unit 1320c may include coordinates (xc, yc). Thevideo decoding apparatus 100 may determine the middle coding unit 1320b by using thecoordinates of the top left samples 1330a, 1330b, and 1330c which are included in thecoding units 1320a, 1320b, and 1320c, respectively. For example, when the coordinatesof the top left samples 1330a, 1330b, and 1330c are sorted in an ascending ordescending order, the coding unit 1320b including the coordinates (xb, yb) of the sample1330b at a center location may be determined as a coding unit at a center location fromamong the coding units 1320a, 1320b, and 1320c determined by splitting the currentcoding unit 1300. However, the coordinates indicating the locations of the top leftsamples 1330a, 1330b, and 1330c may include coordinates indicating absolute locationsin the picture, or may use coordinates (dxb, dyb) indicating a relative location of the topleft sample 1330b of the middle coding unit 1320b and coordinates (dxc, dyc) indicatinga relative location of the top left sample 1330c of the lower coding unit 1320c withreference to the location of the top left sample 1330a of the upper coding unit 1320a. Amethod of determining a coding unit at a predetermined location by using coordinates ofa sample included in the coding unit, as information indicating a location of the sample, isnot limited to the above-described method, and may include various arithmetic methodscapable of using the coordinates of the sample.
[00639] According to an embodiment, the video decoding apparatus 100 may split thecurrent coding unit 1300 into a plurality of coding units 1320a, 1320b, and 1320c, andmay select one of the coding units 1320a, 1320b, and 1320c based on a predeterminedcriterion. For example, the video decoding apparatus 100 may select the coding unit1320b, which has a size different from that of the others, from among the coding units1320a, 1320b, and 1320c.
[00640] According to an embodiment, the video decoding apparatus 100 maydetermine the widths or heights of the coding units 1320a, 1320b, and 1320c by usingthe coordinates (xa, ya) indicating the location of the top left sample 1330a of the uppercoding unit 1320a, the coordinates (xb, yb) indicating the location of the top left sample1330b of the middle coding unit 1320b, and the coordinates (xc, yc) indicating thelocation of the top left sample 1330c of the lower coding unit 1320c. The video decodingapparatus 100 may determine the respective sizes of the coding units 1320a, 1320b,and 1320c by using the coordinates (xa, ya), (xb, yb), and (xc, yc) indicating thelocations of the coding units 1320a, 1320b, and 1320c.
[00641] According to an embodiment, the video decoding apparatus 100 maydetermine the width of the upper coding unit 1320a to be xb-xa and determine the heightthereof to be yb-ya. According to an embodiment, the video decoding apparatus 100may determine the width of the middle coding unit 1320b to be xc-xb and determine theheight thereof to be yc-yb. According to an embodiment, the video decoding apparatus100 may determine the width or height of the lower coding unit 1320c by using the widthor height of the current coding unit 1300 or the widths or heights of the upper and middlecoding units 1320a and 1320b. The video decoding apparatus 100 may determine acoding unit, which has a size different from that of the others, based on the determinedwidths and heights of the coding units 1320a to 1320c. Referring to FIG. 13, the videodecoding apparatus 100 may determine the middle coding unit 1320b, which has a sizedifferent from the size of the upper and lower coding units 1320a and 1320c, as thecoding unit of the predetermined location. However, the above-described method,performed by the video decoding apparatus 100, of determining a coding unit having asize different from the size of the other coding units merely corresponds to an example ofdetermining a coding unit at a predetermined location by using the sizes of coding units,which are determined based on coordinates of samples, and thus various methods ofdetermining a coding unit at a predetermined location by comparing the sizes of codingunits, which are determined based on coordinates of predetermined samples, may beused.
[00642] However, locations of samples considered to determine locations of codingunits are not limited to the above-described top left locations, and information aboutarbitrary locations of samples included in the coding units may be used.
[00643] According to an embodiment, the video decoding apparatus 100 may select acoding unit at a predetermined location from among an odd number of coding unitsdetermined by splitting the current coding unit, considering the shape of the currentcoding unit. For example, when the current coding unit has a non-square shape, a widthof which is longer than a height, the video decoding apparatus 100 may determine thecoding unit at the predetermined location in a horizontal direction. That is, the videodecoding apparatus 100 may determine one of coding units at different locations in ahorizontal direction and put a restriction on the coding unit. When the current coding unithas a non-square shape, a height of which is longer than a width, the video decodingapparatus 100 may determine the coding unit at the predetermined location in a verticaldirection. That is, the video decoding apparatus 100 may determine one of coding unitsat different locations in a vertical direction and may put a restriction on the coding unit.
[00644] According to an embodiment, the video decoding apparatus 100 may useinformation indicating respective locations of an even number of coding units, todetermine the coding unit at the predetermined location from among the even number ofcoding units. The video decoding apparatus 100 may determine an even number ofcoding units by splitting the current coding unit, and may determine the coding unit at thepredetermined location by using the information about the locations of the even numberof coding units. An operation related thereto may correspond to the operation ofdetermining a coding unit at a predetermined location (e.g., a center location) fromamong an odd number of coding units, which has been described in detail above inrelation to FIG. 13, and thus detailed descriptions thereof are not provided here.
[00645] According to an embodiment, when a non-square current coding unit is splitinto a plurality of coding units, predetermined information about a coding unit at apredetermined location may be used in a splitting operation to determine the coding unitat the predetermined location from among the plurality of coding units. For example, thevideo decoding apparatus 100 may use at least one of block shape information and splitshape information, which is stored in a sample included in a coding unit at a centerlocation, in a splitting operation to determine the coding unit at the center location fromamong the plurality of coding units determined by splitting the current coding unit.
[00646] Referring to FIG. 13, the video decoding apparatus 100 may split the currentcoding unit 1300 into a plurality of coding units 1320a, 1320b, and 1320c based on atleast one of the block shape information and the split shape information, and maydetermine a coding unit 1320b at a center location from among the plurality of the codingunits 1320a, 1320b, and 1320c. Furthermore, the video decoding apparatus 100 maydetermine the coding unit 1320b at the center location, in consideration of a location fromwhich at least one of the block shape information and the split shape information isobtained. That is, at least one of the block shape information and the split shapeinformation of the current coding unit 1300 may be obtained from the sample 1340 at acenter location of the current coding unit 1300 and, when the current coding unit 1300 issplit into the plurality of coding units 1320a, 1320b, and 1320c based on at least one ofthe block shape information and the split shape information, the coding unit 1320bincluding the sample 1340 may be determined as the coding unit at the center location.However, information used to determine the coding unit at the center location is notlimited to at least one of the block shape information and the split shape information, andvarious types of information may be used to determine the coding unit at the centerlocation.
[00647] According to an embodiment, predetermined information for identifying thecoding unit at the predetermined location may be obtained from a predetermined sampleincluded in a coding unit to be determined. Referring to FIG. 13, the video decodingapparatus 100 may use at least one of the block shape information and the split shapeinformation, which is obtained from a sample at a predetermined location in the currentcoding unit 1300 (e.g., a sample at a center location of the current coding unit 1300) todetermine a coding unit at a predetermined location from among the plurality of thecoding units 1320a, 1320b, and 1320c determined by splitting the current coding unit1300 (e.g., a coding unit at a center location from among a plurality of split coding units).That is, the video decoding apparatus 100 may determine the sample at thepredetermined location by considering a block shape of the current coding unit 1300,determine the coding unit 1320b including a sample, from which predeterminedinformation (e.g., at least one of the block shape information and the split shapeinformation) may be obtained, from among the plurality of coding units 1320a, 1320b,and 1320c determined by splitting the current coding unit 1300, and may put apredetermined restriction on the coding unit 1320b. Referring to FIG. 13, according to anembodiment, the video decoding apparatus 100 may determine the sample 1340 at thecenter location of the current coding unit 1300 as the sample from which thepredetermined information may be obtained, and may put a predetermined restriction onthe coding unit 1320b including the sample 1340, in a decoding operation. However, thelocation of the sample from which the predetermined information may be obtained is notlimited to the above-described location, and may include arbitrary locations of samplesincluded in the coding unit 1320b to be determined for a restriction.
[00648] According to an embodiment, the location of the sample from which thepredetermined information may be obtained may be determined based on the shape ofthe current coding unit 1300. According to an embodiment, the block shape informationmay indicate whether the current coding unit has a square or non-square shape, and thelocation of the sample from which the predetermined information may be obtained maybe determined based on the shape. For example, the video decoding apparatus 100may determine a sample located on a boundary for dividing at least one of a width andheight of the current coding unit in half, as the sample from which the predeterminedinformation may be obtained, by using at least one of information about the width of thecurrent coding unit and information about the height of the current coding unit. Asanother example, when the block shape information of the current coding unit indicates anon-square shape, the video decoding apparatus 100 may determine one of samplesadjacent to a boundary for dividing a long side of the current coding unit in half, as thesample from which the predetermined information may be obtained.
[00649] According to an embodiment, when the current coding unit is split into aplurality of coding units, the video decoding apparatus 100 may use at least one of theblock shape information and the split shape information to determine a coding unit at apredetermined location from among the plurality of coding units. According to anembodiment, the video decoding apparatus 100 may obtain at least one of the blockshape information and the split shape information from a sample at a predeterminedlocation in a coding unit, and split the plurality of coding units, which are generated bysplitting the current coding unit, by using at least one of the split shape information andthe block shape information, which is obtained from the sample of the predeterminedlocation in each of the plurality of coding units. That is, a coding unit may be recursivelysplit based on at least one of the block shape information and the split shape information,which is obtained from the sample at the predetermined location in each coding unit. Anoperation of recursively splitting a coding unit has been described above in relation toFIG. 12, and thus detailed descriptions thereof will not be provided here.
[00650] According to an embodiment, the video decoding apparatus 100 maydetermine one or more coding units by splitting the current coding unit, and maydetermine an order of decoding the one or more coding units, based on a predeterminedblock (e.g., the current coding unit).
[00651] FIG. 14 illustrates an order of processing a plurality of coding units when thevideo decoding apparatus 100 determines the plurality of coding units by splitting acurrent coding unit, according to an embodiment.
[00652] According to an embodiment, the video decoding apparatus 100 maydetermine second coding units 1410a and 1410b by splitting a first coding unit 1400 in avertical direction, determine second coding units 1430a and 1430b by splitting the firstcoding unit 1400 in a horizontal direction, or determine second coding units 1450a to1450d by splitting the first coding unit 1400 in vertical and horizontal directions, based onblock shape information and split shape information.
[00653] Referring to FIG. 14, the video decoding apparatus 100 may determine toprocess the second coding units 1410a and 1410b, which are determined by splitting thefirst coding unit 1400 in a vertical direction, in a horizontal direction order 1410c. Thevideo decoding apparatus 100 may determine to process the second coding units 1430aand 1430b, which are determined by splitting the first coding unit 1400 in a horizontaldirection, in a vertical direction order 1430c. The video decoding apparatus 100 maydetermine to process the second coding units 1450a to 1450d, which are determined bysplitting the first coding unit 1400 in vertical and horizontal directions, in a predeterminedorder for processing coding units in a row and then processing coding units in a next row(e.g., in a raster scan order or Z-scan order 1450e).
[00654] According to an embodiment, the video decoding apparatus 100 mayrecursively split coding units. Referring to FIG. 14, the video decoding apparatus 100may determine a plurality of coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b,1450c, and 1450d by splitting the first coding unit 1400, and may recursively split each ofthe determined plurality of coding units 1410a, 1410b, 1430a, 1430b, 1450a, 1450b,1450c, and 1450d. A splitting method of the plurality of coding units 1410a, 1410b,1430a, 1430b, 1450a, 1450b, 1450c, and 1450d may correspond to a splitting method ofthe first coding unit 1400. As such, each of the plurality of coding units 1410a, 1410b,1430a, 1430b, 1450a, 1450b, 1450c, and 1450d may be independently split into aplurality of coding units. Referring to FIG. 14, the video decoding apparatus 100 maydetermine the second coding units 1410a and 1410b by splitting the first coding unit1400 in a vertical direction, and may determine to independently split or not to split eachof the second coding units 1410a and 1410b.
[00655] According to an embodiment, the video decoding apparatus 100 maydetermine third coding units 1420a and 1420b by splitting the left second coding unit1410a in a horizontal direction, and may not split the right second coding unit 1410b.
[00656] According to an embodiment, a processing order of coding units may bedetermined based on an operation of splitting a coding unit. In other words, a processingorder of split coding units may be determined based on a processing order of codingunits immediately before being split. The video decoding apparatus 100 may determine aprocessing order of the third coding units 1420a and 1420b determined by splitting theleft second coding unit 1410a, independently of the right second coding unit 1410b.Because the third coding units 1420a and 1420b are determined by splitting the leftsecond coding unit 1410a in a horizontal direction, the third coding units 1420a and1420b may be processed in a vertical direction order 1420c. Because the left and rightsecond coding units 1410a and 1410b are processed in the horizontal direction order1410c, the right second coding unit 1410b may be processed after the third coding units1420a and 1420b included in the left second coding unit 1410a are processed in thevertical direction order 1420c. An operation of determining a processing order of codingunits based on a coding unit before being split is not limited to the above-describedexample, and various methods may be used to independently process coding units,which are split and determined to various shapes, in a predetermined order.
[00657] FIG. 15 illustrates a process, performed by the video decoding apparatus 100,of determining that a current coding unit is to be split into an odd number of coding units,when the coding units are not processable in a predetermined order, according to anembodiment.
[00658] According to an embodiment, the video decoding apparatus 100 maydetermine whether the current coding unit is split into an odd number of coding units,based on obtained block shape information and split shape information. Referring to FIG.15, a square first coding unit 1500 may be split into non-square second coding units1510a and 1510b, and the second coding units 1510a and 1510b may be independentlysplit into third coding units 1520a and 1520b, and 1520c to 1520e. According to anembodiment, the video decoding apparatus 100 may determine a plurality of third codingunits 1520a and 1520b by splitting the left second coding unit 1510a in a horizontaldirection, and may split the right second coding unit 1510b into an odd number of thirdcoding units 1520c to 1520e.
[00659] According to an embodiment, the video decoding apparatus 100 maydetermine whether any coding unit is split into an odd number of coding units, bydetermining whether the third coding units 1520a and 1520b, and 1520c to 1520e areprocessable in a predetermined order. Referring to FIG. 15, the video decodingapparatus 100 may determine the third coding units 1520a and 1520b, and 1520c to1520e by recursively splitting the first coding unit 1500. The video decoding apparatus100 may determine whether any of the first coding unit 1500, the second coding units1510a and 1510b, and the third coding units 1520a and 1520b, and 1520c, 1520d, and1520e are split into an odd number of coding units, based on at least one of the blockshape information and the split shape information. For example, the right second codingunit 1510b may be split into an odd number of third coding units 1520c, 1520d, and1520e. A processing order of a plurality of coding units included in the first coding unit1500 may be a predetermined order (e.g., a Z-scan order 1530), and the video decodingapparatus 100 may decide whether the third coding units 1520c, 1520d, and 1520e,which are determined by splitting the right second coding unit 1510b into an odd numberof coding units, satisfy a condition for processing in the predetermined order.
[00660] According to an embodiment, the video decoding apparatus 100 maydetermine whether the third coding units 1520a and 1520b, and 1520c, 1520d, and1520e included in the first coding unit 1500 satisfy the condition for processing in thepredetermined order, and the condition relates to whether at least one of a width andheight of the second coding units 1510a and 1510b is divided in half along a boundary ofthe third coding units 1520a and 1520b, and 1520c, 1520d, and 1520e. For example, thethird coding units 1520a and 1520b determined by dividing the height of the non-squareleft second coding unit 1510a in half satisfy the condition. However, because boundariesof the third coding units 1520c, 1520d, and 1520e determined by splitting the rightsecond coding unit 1510b into three coding units do not divide the width or height of theright second coding unit 1510b in half, it may be determined that the third coding units1520c, 1520d, and 1520e do not satisfy the condition. When the condition is not satisfiedas described above, the video decoding apparatus 100 may decide disconnection of ascan order, and determine that the right second coding unit 1510b is split into an oddnumber of coding units, based on a result of the decision. According to an embodiment,when a coding unit is split into an odd number of coding units, the video decodingapparatus 100 may put a predetermined restriction on a coding unit at a predeterminedlocation among the split coding units. The restriction or the predetermined location hasbeen described above in relation to various embodiments, and thus detailed descriptionsthereof will not be provided here.
[00661] FIG. 16 illustrates a process, performed by the video decoding apparatus 100,of determining at least one coding unit by splitting a first coding unit 1600, according toan embodiment. According to an embodiment, the video decoding apparatus 100 maysplit the first coding unit 1600, based on at least one of block shape information and splitshape information, which is obtained by the obtainer 110. The square first coding unit1600 may be split into four square coding units, or may be split into a plurality ofnon-square coding units. For example, referring to FIG. 16, when the block shapeinformation indicates that the first coding unit 1600 has a square shape and the splitshape information indicates to split the first coding unit 1600 into non-square codingunits, the video decoding apparatus 100 may split the first coding unit 1600 into aplurality of non-square coding units. In detail, when the split shape information indicatesto determine an odd number of coding units by splitting the first coding unit 1600 in ahorizontal direction or a vertical direction, the video decoding apparatus 100 may splitthe square first coding unit 1600 into an odd number of coding units, e.g., second codingunits 1610a, 1610b, and 1610c determined by splitting the square first coding unit 1600in a vertical direction or second coding units 1620a, 1620b, and 1620c determined bysplitting the square first coding unit 1600 in a horizontal direction.
[00662] According to an embodiment, the video decoding apparatus 100 maydetermine whether the second coding units 1610a, 1610b, 1610c, 1620a, 1620b, and1620c included in the first coding unit 1600 satisfy a condition for processing in apredetermined order, and the condition relates to whether at least one of a width andheight of the first coding unit 1600 is divided in half along a boundary of the secondcoding units 1610a, 1610b, 1610c, 1620a, 1620b, and 1620c. Referring to FIG. 16,because boundaries of the second coding units 1610a, 1610b, and 1610c determined bysplitting the square first coding unit 1600 in a vertical direction do not divide the width ofthe first coding unit 1600 in half, it may be determined that the first coding unit 1600 doesnot satisfy the condition for processing in the predetermined order. In addition, becauseboundaries of the second coding units 1620a, 1620b, and 1620c determined by splittingthe square first coding unit 1600 in a horizontal direction do not divide the width of thefirst coding unit 1600 in half, it may be determined that the first coding unit 1600 does notsatisfy the condition for processing in the predetermined order. When the condition is notsatisfied as described above, the video decoding apparatus 100 may decidedisconnection of a scan order, and may determine that the first coding unit 1600 is splitinto an odd number of coding units, based on a result of the decision. According to anembodiment, when a coding unit is split into an odd number of coding units, the videodecoding apparatus 100 may put a predetermined restriction on a coding unit at apredetermined location from among the split coding units. The restriction or thepredetermined location has been described above in relation to various embodiments,and thus detailed descriptions thereof will not be provided herein.
[00663] According to an embodiment, the video decoding apparatus 100 maydetermine various-shaped coding units by splitting a first coding unit.
[00664] Referring to FIG. 16, the video decoding apparatus 100 may split the squarefirst coding unit 1600 or a non-square first coding unit 1630 or 1650 into various-shapedcoding units.
[00665] FIG. 17 illustrates that a shape into which a second coding unit is splittable bythe video decoding apparatus 100 is restricted when the second coding unit having anon-square shape, which is determined by splitting a first coding unit 1700, satisfies apredetermined condition, according to an embodiment.
[00666] According to an embodiment, the video decoding apparatus 100 maydetermine to split the square first coding unit 1700 into non-square second coding units1710a, 1710b, 1720a, and 1720b, based on at least one of block shape information andsplit shape information, which is obtained by the obtainer 105. The second coding units1710a, 1710b, 1720a, and 1720b may be independently split. As such, the videodecoding apparatus 100 may determine to split or not to split the first coding unit 1700into a plurality of coding units, based on at least one of the block shape information andthe split shape information of each of the second coding units 1710a, 1710b, 1720a, and1720b. According to an embodiment, the video decoding apparatus 100 may determinethird coding units 1712a and 1712b by splitting the non-square left second coding unit1710a, which is determined by splitting the first coding unit 1700 in a vertical direction, ina horizontal direction. However, when the left second coding unit 1710a is split in ahorizontal direction, the video decoding apparatus 100 may restrict the right secondcoding unit 1710b to not be split in a horizontal direction in which the left second codingunit 1710a is split. When third coding units 1714a and 1714b are determined by splittingthe right second coding unit 1710b in a same direction, because the left and right secondcoding units 1710a and 1710b are independently split in a horizontal direction, the thirdcoding units 1712a, 1712b, 1714a, and 1714b may be determined. However, this caseserves equally as a case in which the video decoding apparatus 100 splits the firstcoding unit 1700 into four square second coding units 1730a, 1730b, 1730c, and 1730d,based on at least one of the block shape information and the split shape information, andmay be inefficient in terms of image decoding.
[00667] According to an embodiment, the video decoding apparatus 100 maydetermine third coding units 1722a, 1722b, 1724a, and 1724b by splitting the non-squaresecond coding unit 1720a or 1720b, which is determined by splitting a first coding unit11300 in a horizontal direction, in a vertical direction. However, when a second codingunit (e.g., the upper second coding unit 1720a) is split in a vertical direction, for theabove-described reason, the video decoding apparatus 100 may restrict the othersecond coding unit (e.g., the lower second coding unit 1720b) to not be split in a verticaldirection in which the upper second coding unit 1720a is split.
[00668] FIG. 18 illustrates a process, performed by the video decoding apparatus 100,of splitting a square coding unit when split shape information indicates that the squarecoding unit is not to be split into four square coding units, according to an embodiment.
[00669] According to an embodiment, the video decoding apparatus 100 maydetermine second coding units 1810a, 1810b, 1820a, 1820b, etc. by splitting a firstcoding unit 1800, based on at least one of block shape information and split shapeinformation. The split shape information may include information about various methodsof splitting a coding unit but, the information about various splitting methods may notinclude information for splitting a coding unit into four square coding units. According tosuch split shape information, the video decoding apparatus 100 may not split the firstsquare coding unit 1800 into four square second coding units 1830a, 1830b, 1830c, and1830d. The video decoding apparatus 100 may determine the non-square secondcoding units 1810a, 1810b, 1820a, 1820b, etc., based on the split shape information.
[00670] According to an embodiment, the video decoding apparatus 100 mayindependently split the non-square second coding units 1810a, 1810b, 1820a, 1820b,etc. Each of the second coding units 1810a, 1810b, 1820a, 1820b, etc. may berecursively split in a predetermined order, and this splitting method may correspond to amethod of splitting the first coding unit 1800, based on at least one of the block shapeinformation and the split shape information.
[00671] For example, the video decoding apparatus 100 may determine square thirdcoding units 1812a and 1812b by splitting the left second coding unit 1810a in ahorizontal direction, and may determine square third coding units 1814a and 1814b bysplitting the right second coding unit 1810b in a horizontal direction. Furthermore, thevideo decoding apparatus 100 may determine square third coding units 1816a, 1816b,1816c, and 1816d by splitting both of the left and right second coding units 1810a and1810b in a horizontal direction. In this case, coding units having the same shape as thefour square second coding units 1830a, 1830b, 1830c, and 1830d split from the firstcoding unit 1800 may be determined.
[00672] As another example, the video decoding apparatus 100 may determine squarethird coding units 1822a and 1822b by splitting the upper second coding unit 1820a in avertical direction, and may determine square third coding units 1824a and 1824b bysplitting the lower second coding unit 1820b in a vertical direction. Furthermore, thevideo decoding apparatus 100 may determine square third coding units 1822a, 1822b,1824a, and 1824b by splitting both of the upper and lower second coding units 1820aand 1820b in a vertical direction. In this case, coding units having the same shape as thefour square second coding units 1830a, 1830b, 1830c, and 1830d split from the firstcoding unit 1800 may be determined.
[00673] FIG. 19 illustrates that a processing order between a plurality of coding unitsmay be changed depending on a process of splitting a coding unit, according to anembodiment.
[00674] According to an embodiment, the video decoding apparatus 100 may split afirst coding unit 1900, based on block shape information and split shape information.When the block shape information indicates a square shape and the split shapeinformation indicates to split the first coding unit 1900 in at least one of horizontal andvertical directions, the video decoding apparatus 100 may determine second codingunits 1910a, 1910b, 1920a, 1920b, 1930a, 1930b, 1930c, and 1930d by splitting the firstcoding unit 1900. Referring to FIG. 19, the non-square second coding units 1910a,1910b, 1920a, and 1920b determined by splitting the first coding unit 1900 in only ahorizontal direction or vertical direction may be independently split based on the blockshape information and the split shape information of each coding unit. For example, thevideo decoding apparatus 100 may determine third coding units 1916a, 1916b, 1916c,and 1916d by splitting the second coding units 1910a and 1910b, which are generatedby splitting the first coding unit 1900 in a vertical direction, in a horizontal direction, andmay determine third coding units 1926a, 1926b, 1926c, and 1926d by splitting thesecond coding units 1920a and 1920b, which are generated by splitting the first codingunit 1900 in a horizontal direction, in a horizontal direction. An operation of splitting thesecond coding units 1910a, 1910b, 1920a, and 1920b has been described above inrelation to FIG. 17, and thus detailed descriptions thereof will not be provided herein.
[00675] According to an embodiment, the video decoding apparatus 100 may processcoding units in a predetermined order. An operation of processing coding units in apredetermined order has been described above in relation to FIG. 14, and thus detaileddescriptions thereof will not be provided herein. Referring to FIG. 19, the video decodingapparatus 100 may determine four square third coding units 1916a, 1916b, 1916c, and1916d, and 1926a, 1926b, 1926c, and 1926d by splitting the square first coding unit1900. According to an embodiment, the video decoding apparatus 100 may determineprocessing orders of the third coding units 1916a, 1916b, 1916c, and 1916d, and 1926a,1926b, 1926c, and 1926d based on a splitting method of the first coding unit 1900.
[00676] According to an embodiment, the video decoding apparatus 100 maydetermine the third coding units 1916a, 1916b, 1916c, and 1916d by splitting the secondcoding units 1910a and 1910b generated by splitting the first coding unit 1900 in avertical direction, in a horizontal direction, and may process the third coding units 1916a,1916b, 1916c, and 1916d in a processing order 1917 for initially processing the thirdcoding units 1916a and 1916b , which are included in the left second coding unit 1910a,in a vertical direction and then processing the third coding unit 1916c and 1916d, whichare included in the right second coding unit 1910b, in a vertical direction.
[00677] According to an embodiment, the video decoding apparatus 100 maydetermine the third coding units 1926a, 1926b, 1926c, and 1926d by splitting the secondcoding units 1920a and 1920b generated by splitting the first coding unit 1900 in ahorizontal direction, in a vertical direction, and may process the third coding units 1926a,1926b, 1926c, and 1926d in a processing order 1927 for initially processing the thirdcoding units 1926a and 1926b, which are included in the upper second coding unit1920a, in a horizontal direction and then processing the third coding unit 1926c and1926d, which are included in the lower second coding unit 1920b, in a horizontaldirection.
[00678] Referring to FIG. 19, the square third coding units 1916a, 1916b, 1916c, and1916d, and 1926a, 1926b, 1926c, and 1926d may be determined by splitting the secondcoding units 1910a, 1910b, 1920a, and 1920b, respectively. Although the second codingunits 1910a and 1910b are determined by splitting the first coding unit 1900 in a verticaldirection differently from the second coding units 1920a and 1920b which aredetermined by splitting the first coding unit 1900 in a horizontal direction, the third codingunits 1916a, 1916b, 1916c, and 1916d, and 1926a, 1926b, 1926c, and 1926d splittherefrom eventually show same-shaped coding units split from the first coding unit 1900.As such, by recursively splitting a coding unit in different manners based on at least oneof the block shape information and the split shape information, the video decodingapparatus 100 may process a plurality of coding units in different orders even when thecoding units are eventually determined to be the same shape.
[00679] FIG. 20 illustrates a process of determining a depth of a coding unit as a shapeand size of the coding unit change, when the coding unit is recursively split such that aplurality of coding units are determined, according to an embodiment.
[00680] According to an embodiment, the video decoding apparatus 100 maydetermine the depth of the coding unit, based on a predetermined criterion. For example,the predetermined criterion may be the length of a long side of the coding unit. When thelength of a long side of a coding unit before being split is 2n times (n>0) the length of along side of a split current coding unit, the video decoding apparatus 100 may determinethat a depth of the current coding unit is increased from a depth of the coding unit beforebeing split, by n. In the following description, a coding unit having an increased depth isexpressed as a coding unit of a deeper depth.
[00681] Referring to FIG. 20, according to an embodiment, the video decodingapparatus 100 may determine a second coding unit 2002 and a third coding unit 2004 ofdeeper depths by splitting a square first coding unit 2000 based on block shapeinformation indicating a square shape (for example, the block shape information may beexpressed as '0: SQUARE'). Assuming that the size of the square first coding unit 2000is 2Nx2N, the second coding unit 2002 determined by dividing a width and height of thefirst coding unit 2000 to 1 / 21 may have a size of NxN. Furthermore, the third coding unit2004 determined by dividing a width and height of the second coding unit 2002 to 1 / 2may have a size of N / 2xN / 2. In this case, a width and height of the third coding unit 2004are 1 / 22 times those of the first coding unit 2000. When a depth of the first coding unit2000 is D, a depth of the second coding unit 2002, the width and height of which are 1 / 21times those of the first coding unit 2000, may be D+1, and a depth of the third coding unit2004, the width and height of which are 1 / 22 times those of the first coding unit 2000,may be D+2.
[00682] According to an embodiment, the video decoding apparatus 100 maydetermine a second coding unit 2012 or 2022 and a third coding unit 2014 or 2024 ofdeeper depths by splitting a non-square first coding unit 2010 or 2020 based on blockshape information indicating a non-square shape (for example, the block shapeinformation may be expressed as '1: NS_VER' indicating a non-square shape, a heightof which is longer than a width, or as '2: NS_HOR' indicating a non-square shape, awidth of which is longer than a height).
[00683] The video decoding apparatus 100 may determine a second coding unit 2002,2012, or 2022 by dividing at least one of a width and height of the first coding unit 2010having a size of Nx2N. That is, the video decoding apparatus 100 may determine thesecond coding unit 2002 having a size of NxN or the second coding unit 2022 having asize of NxN / 2 by splitting the first coding unit 2010 in a horizontal direction, or maydetermine the second coding unit 2012 having a size of N / 2xN by splitting the first codingunit 2010 in horizontal and vertical directions.
[00684] According to an embodiment, the video decoding apparatus 100 maydetermine the second coding unit 2002, 2012, or 2022 by dividing at least one of a widthand height of the first coding unit 2020 having a size of 2NxN. That is, the videodecoding apparatus 100 may determine the second coding unit 2002 having a size ofNxN or the second coding unit 2012 having a size of N / 2xN by splitting the first codingunit 2020 in a vertical direction, or may determine the second coding unit 2022 having asize of NxN / 2 by splitting the first coding unit 2020 in horizontal and vertical directions.
[00685] According to an embodiment, the video decoding apparatus 100 maydetermine a third coding unit 2004, 2014, or 2024 by dividing at least one of a width andheight of the second coding unit 2002 having a size of NxN. That is, the video decodingapparatus 100 may determine the third coding unit 2004 having a size of N / 2xN / 2, thethird coding unit 2014 having a size of N / 22xN / 2, or the third coding unit 2024 having asize of N / 2xN / 22 by splitting the second coding unit 2002 in vertical and horizontaldirections.
[00686] According to an embodiment, the video decoding apparatus 100 maydetermine the third coding unit 2004, 2014, or 2024 by dividing at least one of a widthand height of the second coding unit 2012 having a size of N / 2xN. That is, the videodecoding apparatus 100 may determine the third coding unit 2004 having a size ofN / 2xN / 2 or the third coding unit 2024 having a size of N / 2xN / 22 by splitting the secondcoding unit 2012 in a horizontal direction, or may determine the third coding unit 2014having a size of N / 22xN / 2 by splitting the second coding unit 2012 in vertical andhorizontal directions.
[00687] According to an embodiment, the video decoding apparatus 100 maydetermine the third coding unit 2004, 2014, or 2024 by dividing at least one of a widthand height of the second coding unit 2014 having a size of NxN / 2. That is, the videodecoding apparatus 100 may determine the third coding unit 2004 having a size ofN / 2xN / 2 or the third coding unit 2014 having a size of N / 22xN / 2 by splitting the secondcoding unit 2012 in a vertical direction, or may determine the third coding unit 2024having a size of N / 2xN / 22 by splitting the second coding unit 2022 in vertical andhorizontal directions.
[00688] According to an embodiment, the video decoding apparatus 100 may split thesquare coding unit 2000, 2002, or 2004 in a horizontal or vertical direction. For example,the video decoding apparatus 100 may determine the first coding unit 2010 having a sizeof Nx2N by splitting the first coding unit 2000 having a size of 2Nx2N in a verticaldirection, or may determine the first coding unit 2020 having a size of 2NxN by splittingthe first coding unit 2000 in a horizontal direction. According to an embodiment, when adepth is determined based on the length of the longest side of a coding unit, a depth of acoding unit determined by splitting the first coding unit 2000, 2002 or 2004 having a sizeof 2Nx2N in a horizontal or vertical direction may be the same as the depth of the firstcoding unit 2000, 2002 or 2004.
[00689] According to an embodiment, a width and height of the third coding unit 2014 or2024 may be 1 / 22 times those of the first coding unit 2010 or 2020. When a depth of thefirst coding unit 2010 or 2020 is D, a depth of the second coding unit 2012 or 2014, thewidth and height of which are 1 / 2 times those of the first coding unit 2010 or 2020, maybe D+1, and a depth of the third coding unit 2014 or 2024, the width and height of whichare 1 / 22 times those of the first coding unit 2010 or 2020, may be D+2.
[00690] FIG. 21 illustrates depths determinable based on shapes and sizes of codingunits, and part indexes (PIDs) for distinguishing the coding units, according to anembodiment.
[00691] According to an embodiment, the video decoding apparatus 100 maydetermine various-shape second coding units by splitting a square first coding unit 2100.Referring to FIG. 21, the video decoding apparatus 100 may determine second codingunits 2102a and 2102b, 2104a and 2104b, and 2106a, 2106b, 2106c, and 2106d bysplitting the first coding unit 2100 in at least one of vertical and horizontal directionsbased on split shape information. That is, the video decoding apparatus 100 maydetermine the second coding units 2102a and 2102b, 2104a and 2104b, and 2106a,2106b, 2106c, and 2106d, based on the split shape information of the first coding unit2100.
[00692] According to an embodiment, a depth of the second coding units 2102a and2102b, 2104a and 2104b, and 2106a, 2106b, 2106c, and 2106d, which are determinedbased on the split shape information of the square first coding unit 2100, may bedetermined based on the length of a long side thereof. For example, because the lengthof a side of the square first coding unit 2100 equals the length of a long side of thenon-square second coding units 2102a and 2102b, and 2104a and 2104b, the firstcoding unit 2100 and the non-square second coding units 2102a and 2102b, and 2104aand 2104b may have the same depth, e.g., D. However, when the video decodingapparatus 100 splits the first coding unit 2100 into the four square second coding units2106a, 2106b, 2106c, and 2106d based on the split shape information, because thelength of a side of the square second coding units 2106a, 2106b, 2106c, and 2106d is1 / 2 times the length of a side of the first coding unit 2100, a depth of the second codingunits 2106a, 2106b, 2106c, and 2106d may be D+1 which is deeper than the depth D ofthe first coding unit 2100 by 1.
[00693] According to an embodiment, the video decoding apparatus 100 maydetermine a plurality of second coding units 2112a and 2112b, and 2114a, 2114b, and2114c by splitting a first coding unit 2110, a height of which is longer than a width, in ahorizontal direction based on the split shape information. According to an embodiment,the video decoding apparatus 100 may determine a plurality of second coding units2122a and 2122b, and 2124a, 2124b, and 2124c by splitting a first coding unit 2120, awidth of which is longer than a height, in a vertical direction based on the split shapeinformation.
[00694] According to an embodiment, a depth of the second coding units 2112a and2112b, 2114a, 2114b, and 2116a, 2116b, 2116c, and 2116d, which are determinedbased on the split shape information of the non-square first coding unit 2110 or 2120,may be determined based on the length of a long side thereof. For example, because thelength of a side of the square second coding units 2112a and 2112b is 1 / 2 times thelength of a long side of the first coding unit 2110 having a non-square shape, a height ofwhich is longer than a width, a depth of the square second coding units 2102a, 2102b,2104a, and 2104b is D+1 which is deeper than the depth D of the non-square first codingunit 2110 by 1.
[00695] Furthermore, the video decoding apparatus 100 may split the non-square firstcoding unit 2110 into an odd number of second coding units 2114a, 2114b, and 2114cbased on the split shape information. The odd number of second coding units 2114a,2114b, and 2114c may include the non-square second coding units 2114a and 2114cand the square second coding unit 2114b. In this case, because the length of a long sideof the non-square second coding units 2114a and 2114c and the length of a side of thesquare second coding unit 2114b are 1 / 2 times the length of a long side of the firstcoding unit 2110, a depth of the second coding units 2114a, 2114b, and 2114c may beD+1 which is deeper than the depth D of the non-square first coding unit 2110 by 1. Thevideo decoding apparatus 100 may determine depths of coding units split from the f...
Claims
1. A video decoding method comprising: obtaining a first extended reference block in a first reference picture and a second extended reference block in a second reference picture, wherein the first extended reference block comprises a first reference block and a first portion extended from the first reference block, and the second extended reference block comprises a second reference block and a second portion extended from the second reference block; determining a displacement vector of a pixel group of a current block, the pixel group comprising at least one pixel adjacent to an inside of a boundary of the current block, by using a gradient value of at least one reference pixel in the first extended reference block and a gradient value of at least one reference pixel in the second extended reference block, wherein the first portion of the first extended reference block is used to calculate a gradient value of at least one reference pixel in the first reference block, and the second portion of the second extended reference block is used to calculate a gradient value of at least one reference pixel in the second reference block; obtaining a predicted pixel value of the current block by performing optical flow based compensation on the current block by using the gradient value of the at least one reference pixel in the first reference block, the gradient value of the at least one reference pixel in the second reference block, and the displacement vector of the pixel group; and reconstructing the current block based on the predicted pixel value, wherein a size of the pixel group is 4x4.
2. A video encoding method comprising: obtaining a predicted pixel value of a current block by performing optical flow based compensation on the current block by using a gradient value of at least one reference pixel in a first reference block, a gradient value of at least one reference pixel in a second reference block, and a displacement vector of a pixel group of the current block, the pixel group comprising at least one pixel adjacent to an inside of a boundary of the current block; and encoding the current block based on the predicted pixel value, wherein a first extended reference block in a first reference picture and a second extended reference block in a second reference picture are obtained, wherein the first extended reference block comprises the first reference block and a first portion extended from the first reference block, and the second extended reference block comprises the second reference block and a second portion extended from the second reference block, wherein the displacement vector of the pixel group of the current block is determined by using a gradient value of at least one reference pixel in the first extended reference block and a gradient value of at least one reference pixel in the second extended reference block, wherein the first portion of the first extended reference block is used to calculate the gradient value of the at least one reference pixel in the first reference block, and the second portion of the second extended reference block is used to calculate the gradient value of the at least one reference pixel in the second reference block, wherein a size of the pixel group is 4x4.
3. A non-transitory computer-readable medium for recording a bitstream, the bitstream comprising: encoding results of a current block, wherein the encoding results are obtained by: obtaining a predicted pixel value of the current block by performing optical flow based compensation on the current block by using a gradient value of at least one reference pixel in a first reference block, a gradient value of at least one reference pixel in a second reference block, and a displacement vector of a pixel group of the current block, the pixel group comprising at least one pixel adjacent to an inside of a boundary of the current block; and encoding the current block based on the predicted pixel value, wherein a first extended reference block in a first reference picture and a second extended reference block in a second reference picture are obtained, wherein the first extended reference block comprises the first reference block and a first portion extended from the first reference block, and the second extended reference block comprises the second reference block and a second portion extended from the second reference block, wherein the displacement vector of the pixel group of the current block is determined by using a gradient value of at least one reference pixel in the first extended reference block and a gradient value of at least one reference pixel in the second extended reference block, wherein the first portion of the first extended reference block is used to calculate the gradient value of the at least one reference pixel in the first reference block, and the second portion of the second extended reference block is used to calculate the gradient value of the at least one reference pixel in the second reference block, wherein a size of the pixel group is 4x4.
