Image encoding / decoding method and image transmission method
By adaptively adjusting motion vector resolution and using interpolation filters, the method efficiently derives motion information for high-resolution images, improving inter-frame prediction encoding/decoding efficiency.
Patent Information
- Application Number
- CN202411061853.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-09-24
- Filing Date
- 2019-09-11
- Publication Date
- 2024-12-03
AI Technical Summary
Existing image compression technologies face challenges in efficiently deriving motion information and adapting the resolution or accuracy of motion vectors for high-resolution, high-quality images, particularly in inter-frame prediction.
The method and apparatus adaptively adjust the resolution of motion vectors by determining motion information based on predefined inter-frame modes, using merge, AMVP, or intra-block copy modes, and apply interpolation filters to enhance motion compensation efficiency.
This approach efficiently derives motion information and improves inter-frame prediction encoding/decoding by adaptively adjusting motion vector resolution and accuracy, enhancing the overall encoding/decoding efficiency.
Smart Images

Figure CN119071507A_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with application number 201980058797.1 filed on September 11, 2019 and invention name “Image encoding / decoding method and device using inter-frame prediction”. Technical Field
[0002] The invention relates to an image encoding / decoding method and device. Background Art
[0003] Recently, demands for high-resolution, high-quality images such as high-definition (HD) images and ultra-high-definition (UHD) images are increasing in various application fields, and thus, efficient image compression technology is being discussed.
[0004] There are various techniques: for example, an inter-frame prediction technique that predicts a pixel value included in a current picture based on a picture before or after the current picture using a video compression technique; an intra-frame prediction technique that predicts a pixel value included in a current picture by using pixel information in the current picture; an entropy coding technique that assigns a short code to a value with a high frequency of occurrence and a long code to a value with a low frequency of occurrence. Image data can be effectively compressed by using such an image compression technique, and the image data can be transmitted or stored. Summary of the invention
[0005] Technical issues
[0006] An object of the present invention is to provide a method and apparatus for deriving motion information according to a predetermined inter-frame mode.
[0007] An object of the present invention is to provide a method and apparatus for adaptively adjusting the resolution or accuracy of a motion vector.
[0008] The object of the present invention is to provide a motion compensation method and device based on interpolation.
[0009] Technical Solutions
[0010] The image encoding / decoding method and apparatus of the present invention can obtain motion information of a current block and perform motion compensation on the current block based on the motion information.
[0011] In the image encoding / decoding method and device of the present invention, the motion information of the current block can be obtained based on the predefined inter-frame mode in the decoding device, and the predefined inter-frame mode may include at least one of a merge mode, an AMVP mode, an affine mode or an intra-block copy mode (IBC mode).
[0012] In the image encoding / decoding method and apparatus of the present invention, deriving motion information may include: determining the resolution of the motion vector of the current block; deriving a motion vector prediction value corresponding to the resolution; rounding the motion vector difference of the current block based on the resolution; and deriving the motion vector of the current block based on the derived motion vector prediction value and the rounded motion vector difference.
[0013] In the image encoding / decoding method and apparatus of the present invention, the resolution can be determined as a resolution candidate specified by a predetermined index from among a plurality of resolution candidates predefined in the decoding apparatus.
[0014] In the image encoding / decoding method and apparatus of the present invention, the plurality of predefined resolution candidates may include at least one of 1 / 16 pixel, 1 / 8 pixel, 1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels or 4 pixels.
[0015] In the image encoding / decoding method and apparatus of the present invention, a plurality of resolution candidates may be differently defined for each predefined inter-mode.
[0016] In the image encoding / decoding method and apparatus of the present invention, motion information may be derived from a merge candidate list of a current block, and the merge candidate list may include at least one of a spatial merge candidate, a temporal merge candidate, or a combined merge candidate.
[0017] In the image encoding / decoding method and apparatus of the present invention, the combined merge candidate may be obtained by weighted averaging of motion information of a plurality of merge candidates pre-added to the merge candidate list.
[0018] In the image encoding / decoding method and apparatus of the present invention, a combined merge candidate may be derived by considering prediction directions of a plurality of merge candidates.
[0019] In the image encoding / decoding method and apparatus of the present invention, performing motion compensation may include: determining an interpolation filter for a current block; determining a position of a reference integer pixel; and deriving a predicted pixel for the current block by applying the interpolation filter to the reference integer pixel.
[0020] In the image encoding / decoding method and apparatus of the present invention, when the fractional pixel to be interpolated corresponds to a 1 / 2 pixel, one of a plurality of interpolation filters predefined in the decoding apparatus may be selectively used.
[0021] In the image encoding / decoding method and apparatus of the present invention, the interpolation filter of the current block may be determined as one of a plurality of interpolation filters in consideration of the determined resolution.
[0022] In the image encoding / decoding method and apparatus of the present invention, the position of the reference integer pixel may be determined in consideration of the position of the sub-picture.
[0023] Beneficial Effects
[0024] According to the present invention, motion information according to the inter-frame mode can be efficiently derived.
[0025] The present invention can adaptively adjust the resolution or accuracy of the motion vector.
[0026] The present invention can improve the efficiency of motion compensation by selectively using an interpolation filter.
[0027] The present invention can improve the efficiency of inter-frame prediction encoding / decoding through adaptive block partitioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 is a block diagram showing an image encoding apparatus according to an embodiment of the present invention.
[0029] Figure 2 is a block diagram illustrating an image decoding apparatus according to an exemplary embodiment of the present invention.
[0030] Figure 3 Block partition types according to an embodiment to which the present invention is applied are shown.
[0031] Figure 4 A block partitioning method based on a tree structure is shown as an embodiment to which the present invention is applied.
[0032] Figure 5 An inter-frame prediction method is shown as an embodiment to which the present invention is applied.
[0033] Figure 6 A method of deriving motion information according to a merge mode is shown as an embodiment of the present invention.
[0034] Figure 7 A method of deriving motion information based on an affine model is shown as an embodiment to which the present invention is applied.
[0035] Figure 8 A method of deriving affine candidates from control point vectors of spatial / temporal neighboring blocks is shown as an embodiment to which the present invention is applied.
[0036] Fig. 9 A method of adaptively adjusting the resolution of a motion vector is shown as an embodiment to which the present invention is applied.
[0037] Fig.10 An inter-frame prediction method based on interpolation is shown as an embodiment to which the present invention is applied. DETAILED DESCRIPTION
[0038] The image encoding / decoding method and apparatus of the present invention can obtain motion information of a current block and perform motion compensation on the current block based on the motion information.
[0039] In the image encoding / decoding method and device of the present invention, the motion information of the current block can be obtained based on the inter-frame mode predefined in the decoding device, and the predefined inter-frame mode may include at least one of a merge mode, an AMVP mode, an affine mode or an intra-block copy mode (IBC mode).
[0040] In the image encoding / decoding method and apparatus of the present invention, deriving motion information may include: determining the resolution of the motion vector of the current block; deriving a motion vector prediction value corresponding to the resolution; rounding the motion vector difference of the current block based on the resolution; and deriving the motion vector of the current block based on the derived motion vector prediction value and the rounded motion vector difference.
[0041] In the image encoding / decoding method and apparatus of the present invention, the resolution can be determined as a resolution candidate specified by a predetermined index from among a plurality of resolution candidates predefined in the decoding apparatus.
[0042] In the image encoding / decoding method and apparatus of the present invention, the plurality of predefined resolution candidates may include at least one of 1 / 16 pixel, 1 / 8 pixel, 1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels or 4 pixels.
[0043] In the image encoding / decoding method and apparatus of the present invention, a plurality of resolution candidates may be differently defined for each predefined inter-mode.
[0044] In the image encoding / decoding method and apparatus of the present invention, the motion information may be derived from a merge candidate list of the current block, and the merge candidate list may include at least one of a spatial merge candidate, a temporal merge candidate, or a combined merge candidate.
[0045] In the image encoding / decoding method and apparatus of the present invention, the combined merge candidate can be obtained by weighted averaging of motion information of a plurality of merge candidates pre-added to the merge candidate list.
[0046] In the image encoding / decoding method and apparatus of the present invention, the prediction directions of multiple merging candidates may be considered to derive a combined merging candidate.
[0047] In the image encoding / decoding method and apparatus of the present invention, performing motion compensation may include: determining an interpolation filter for a current block; determining a position of a reference integer pixel; and deriving a predicted pixel for the current block by applying the interpolation filter to the reference integer pixel.
[0048] In the image encoding / decoding method and apparatus of the present invention, when the fractional pixel to be interpolated corresponds to a 1 / 2 pixel, one of a plurality of interpolation filters predefined in the decoding apparatus may be selectively used.
[0049] In the image encoding / decoding method and apparatus of the present invention, the interpolation filter of the current block may be determined as one of a plurality of interpolation filters in consideration of the determined resolution.
[0050] In the image encoding / decoding method and apparatus of the present invention, the position of the reference integer pixel may be determined in consideration of the position of the sub-picture.
[0051] Embodiments of the present invention
[0052] The present invention may be changed and modified in various ways and may be described with reference to different exemplary embodiments, some of which will be described and shown in the accompanying drawings. However, these embodiments are not intended to limit the present invention, but are understood to include all modifications, equivalents and alternatives that belong to the spirit and technical scope of the present invention. In the accompanying drawings, the same reference numerals always refer to the same elements.
[0053] Although the terms first, second, etc. may be used to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the teachings of the present invention, a first element may be referred to as a second element, and similarly a second element may be referred to as a first element. The term "and / or" includes any and all combinations of multiple associated listed items.
[0054] It will be understood that when an element is referred to as being "connected to" or "coupled to" another element, the element may be directly connected or coupled to the other element or an intervening element. In contrast, when an element is referred to as being "directly connected to" or "directly coupled to" another element, there are no intervening elements.
[0055] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are also intended to include plural forms. It will also be understood that when used in this specification, the terms "including" and / or "having" specify the presence of stated features, wholes, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, parts and / or their groups.
[0056] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals refer to the same elements throughout, and redundant descriptions of the same elements will be omitted herein.
[0057] Figure 1 is a block diagram showing an image encoding apparatus according to an embodiment of the present invention.
[0058] Reference Figure 1 The image encoding device 100 includes a picture division unit (110), a prediction unit (120, 125), a transform unit (130), a quantization unit (135), a rearrangement unit (160), an entropy encoding unit (165), an inverse quantization unit (140), an inverse transform unit (145), a filter unit (150) and a memory (155).
[0059] Figure 1 Each of the elements shown in the figure is shown independently to represent different characteristic functions in the encoding device, but it is not intended that each element is composed of a separate hardware or a software element. That is, for ease of description, the elements are arranged independently, wherein at least two elements can be combined into a single element, or a single element can be divided into multiple elements to perform functions. It should be noted that, without departing from the essence of the present invention, embodiments in which some elements are integrated into a combined element and / or elements are divided into multiple separate elements are included within the scope of the present invention.
[0060] Some elements are not essential to the substantial functions of the present invention, but are optional constituent elements that can only be used to improve performance. The present invention can be implemented by only including constituent elements that are essential to the embodiments of the present invention except for the constituent elements that are only used to improve performance. Structures that only include essential constituent elements except for the optional constituent elements that are only used to improve performance belong to the scope of the present invention.
[0061] The picture division unit (110) may divide an input picture into at least one processing unit. In this case, the processing unit may be a prediction unit (PU), a transform unit (TU), or a coding unit (CU). The picture division unit (110) may divide a picture into a plurality of combinations of coding units, prediction units, and transform units, and select a combination of coding units, prediction units, and transform units based on a predetermined criterion (e.g., a cost function) to encode the picture.
[0062] For example, a picture can be divided into multiple coding units. In order to divide a picture into coding units, a recursive tree structure such as a quadtree structure can be used. An image or the largest coding block (maximum coding unit) as a root can be divided into other coding units, and can be divided into as many child nodes as the number of divided coding units. Coding units that are no longer divided according to certain restrictions become leaf nodes. That is, when it is assumed that only square division is possible for a coding unit, a coding unit can be divided into up to four different coding units.
[0063] In the embodiments of the present invention, a coding unit may be used to refer not only to a coding unit but also to a decoding unit.
[0064] The prediction unit may be a block divided in a shape such as at least one square or rectangle having the same size within one coding unit, or one prediction unit among the prediction units divided within one coding unit may have a different shape and / or size from another prediction unit.
[0065] When a prediction unit on which intra prediction is performed based on a coding unit is not a minimum coding unit, intra prediction may be performed without being split into a plurality of prediction units N×N.
[0066] The prediction unit (120, 125) may include an inter-frame prediction unit (120) for performing inter-frame prediction and an intra-frame prediction unit (125) for performing intra-frame prediction. The prediction unit (120, 125) may determine which of inter-frame prediction and intra-frame prediction is performed on the PU, and may determine the specific information of the determined prediction method (e.g., intra-frame prediction mode, motion vector, and reference picture). Here, the processing unit on which the prediction is performed may be different from the processing unit for which the prediction method and specific information are determined. For example, the prediction method and prediction mode may be determined for each PU, while the prediction may be performed for each TU. The residual value (residual block) between the generated prediction block and the original block may be input to the transform unit (130). In addition, the prediction mode information, motion vector information, etc. used for the prediction may be encoded by the entropy encoding unit (165) together with the residual value and sent to the decoding device. When a specific encoding mode is used, the original block may be encoded and sent to the decoding device without generating the prediction block by the prediction unit (120, 125).
[0067] The inter prediction unit (120) may predict the PU based on information about at least one of a previous picture of the current picture and a subsequent picture of the current picture. In some cases, the inter prediction unit (120) may predict the PU based on information about a partial coding region in the current picture. The inter prediction unit (120) may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.
[0068] The reference picture interpolation unit may be provided with reference picture information from the memory (155), and the reference picture interpolation unit may generate pixel information less than or equal to an integer pixel on the reference picture. In the case of luma pixels, an 8-tap interpolation filter based on DCT with a variable filter coefficient may be used to generate pixel information in units of 1 / 4 pixels less than or equal to an integer pixel. In the case of chroma pixels, a 4-tap interpolation filter based on DCT with a variable filter coefficient may be used to generate pixel information in units of 1 / 8 pixels less than or equal to an integer pixel.
[0069] The motion prediction unit may perform motion prediction based on the reference picture interpolated by the reference picture interpolation unit. Various methods such as a full search based block matching algorithm (FBMA), a three-step search (TSS) algorithm, and a new three-step search (NTS) algorithm may be used to calculate the motion vector. Based on the interpolated pixel, the motion vector has a motion vector value in units of 1 / 2 or 1 / 4 pixels. The motion prediction unit may use different motion prediction methods to predict the current PU. Various methods such as skip mode, merge mode, advanced motion vector prediction (AMVP) mode, and intra-block copy mode may be used as motion prediction methods.
[0070] The intra prediction unit (125) may generate a PU based on information about reference pixels adjacent to the current block. When the reference pixel is a pixel on which inter prediction has been performed because the block adjacent to the current PU is a block on which inter prediction has been performed, information about the reference pixel on which intra prediction has been performed in the block may be replaced with information about the reference pixel on which inter prediction has been performed in the block. That is, when the reference pixel is unavailable, information about the unavailable reference pixel may be replaced with information about at least one reference pixel among the available reference pixels.
[0071] The prediction mode of the intra prediction includes a directional prediction mode in which reference pixel information is used according to the prediction direction and a non-directional prediction mode in which information about the direction is not used when performing prediction. The mode for predicting luminance information and the mode for predicting chrominance information may be different from each other. In addition, the chrominance information may be predicted using the intra prediction mode information for predicting luminance information or the predicted luminance signal information.
[0072] When the size of the prediction unit is the same as the size of the transform unit when performing intra prediction, intra prediction of the prediction unit can be performed based on the pixels on the left, the pixels on the upper left, and the pixels on the top of the prediction unit. However, when the size of the prediction unit is different from the size of the transform unit when performing intra prediction, intra prediction can be performed using reference pixels determined based on the transform unit. In addition, intra prediction using N×N partitioning can be used only for the minimum coding unit.
[0073] In the intra prediction method, a prediction block can be generated by applying an adaptive intra smoothing (AIS) filter to reference pixels according to a prediction mode. Different types of AIS filters can be applied to reference pixels. In the intra prediction method, the intra prediction mode of the current PU can be predicted based on the intra prediction mode of a PU adjacent to the current PU. When the prediction mode of the current PU is predicted using mode information predicted based on the adjacent PU, when the current PU has the same intra prediction mode as the adjacent PU, predetermined flag information can be used to send information indicating that the current PU has the same prediction mode as the adjacent PU. When the current PU has different prediction modes from the adjacent PU, information about the prediction mode of the current block can be encoded by entropy coding.
[0074] A residual block including residual information may be generated. The residual information is a difference between a prediction unit generated by the prediction unit (120, 125) and an original block of the prediction unit. The generated residual block may be input to the transform unit (130).
[0075] The transform unit (130) may transform a residual block including residual information between a prediction unit generated by the prediction unit (120, 125) and an original block by using a transform type such as DCT (Discrete Cosine Transform), DST (Discrete Sine Transform) or KLT. Whether to apply DCT, DST or KLT to transform the residual block may be determined based on intra prediction mode information of the prediction unit used to generate the residual block.
[0076] The quantization unit (135) may quantize the value transformed to the frequency domain by the transformation unit (130). The quantization coefficient may be changed according to the importance or block of the image. The value output from the quantization unit (135) may be provided to the inverse quantization unit (140) and the rearrangement unit (160).
[0077] The rearrangement unit (160) may perform rearrangement on coefficient values used for quantizing the residual.
[0078] The rearrangement unit (160) may change the coefficients of a two-dimensional (2D) block into coefficients of a one-dimensional (1D) vector by a coefficient scanning method. For example, the rearrangement unit (160) may scan the DC coefficient into coefficients in a high frequency region using a Z-shaped scanning method and change it into a one-dimensional vector form. Depending on the size of the transform unit and the intra-frame prediction mode, a vertical scan that scans the two-dimensional block shape coefficients in a column direction or a horizontal scan that scans the two-dimensional block shape coefficients in a row direction may be used instead of the Z-shaped scan. That is, according to the size of the transform unit and the intra-frame prediction mode, it may be determined which of the Z-shaped scan, the vertical direction scan, and the horizontal direction scan to use.
[0079] The entropy encoding unit (165) may perform entropy encoding based on the value obtained by the rearrangement unit (160). For entropy encoding, various encoding methods may be used, such as exponential Golomb coding, context adaptive variable length coding (CAVLC), or context adaptive binary arithmetic coding (CABAC).
[0080] The entropy coding unit (165) can encode various information from the rearrangement unit (160) and the prediction unit (120, 125), such as residual coefficient information and block type information of the coding unit, prediction mode information, partition unit information, prediction unit information, transmission unit information, motion vector information, reference frame information, block interpolation information and filtering information.
[0081] The entropy encoding unit (165) may entropy encode coefficients of the CU input from the rearrangement unit (160).
[0082] The inverse quantization unit (140) and the inverse transform unit (145) dequantize the value quantized by the quantization unit (135) and inversely transform the value transformed by the transform unit (130). A reconstructed block may be generated by adding a residual value to the predicted PU. The residual value may be generated by the inverse quantization unit (140) and the inverse transform unit (145). The predicted PU may be predicted by an intra prediction unit, a motion vector prediction unit, and a motion compensation unit of the prediction unit (120, 125).
[0083] The filter unit (150) may include at least one of a deblocking filter, an offset unit, and an adaptive loop filter (ALF).
[0084] The deblocking filter can remove block distortion generated by the boundaries between blocks in the reconstructed picture. It can be determined whether to apply the deblocking filter to the current block based on the pixels included in several rows or columns of the block. When the deblocking filter is applied to the block, a strong filter or a weak filter can be applied according to the required deblocking filter strength. When horizontal filtering and vertical filtering are performed when applying the deblocking filter, horizontal filtering and vertical filtering can be performed in parallel.
[0085] The offset unit may apply an offset in pixels relative to the original image to the deblocked filtered image. The area to which the offset may be applied may be determined after dividing the pixels of the picture into a predetermined number of areas. The offset may be applied to the determined area in consideration of edge information about each pixel or a method of applying the offset to the determined area.
[0086] ALF can perform filtering based on the comparison result of the filtered reconstructed image with the original image. The pixels included in the image can be divided into predetermined groups, the filter to be applied to each group can be determined, and differential filtering can be performed for each group. Information about whether ALF is applied can be transmitted by each coding unit (CU), and the shape and filter coefficient of ALF to be applied to each block can be varied. In addition, ALF having the same form (fixed form) can be applied to the block regardless of the characteristics of the block.
[0087] The memory (155) may store the reconstructed block or the reconstructed picture output from the filter unit (150), and when inter-frame prediction is performed, the stored reconstructed block or the reconstructed picture may be provided to the prediction unit (120, 125).
[0088] Figure 2 is a block diagram illustrating an image decoding apparatus according to an exemplary embodiment of the present invention.
[0089] Reference Figure 2 , the image decoding device (200) may include an entropy decoding unit (210), a rearrangement unit (215), a dequantization unit (220), an inverse transform unit (225), a prediction unit (230, 235), a filter unit (240) and a memory (245).
[0090] When an image bit stream is input from the image encoding device, the input bit stream may be decoded in a process opposite to that of the image encoding device.
[0091] The entropy decoding unit (210) may perform entropy decoding in a process opposite to the process of performing entropy encoding in the entropy encoding unit of the image encoding device. For example, various methods such as exponential Golomb coding, CAVLC or CABAC may be applied corresponding to the method performed by the image encoding device.
[0092] The entropy decoding unit (210) can decode information associated with intra-frame prediction and inter-frame prediction performed by the encoding device.
[0093] The rearrangement unit (215) may perform rearrangement on the bit stream entropy-decoded by the entropy decoding unit (210) based on the rearrangement method of the encoding device. The rearrangement unit (215) may reconstruct and rearrange coefficients of the 1D vector into coefficients of the 2D block. Information about coefficient scanning performed by the encoding device may be provided to the rearrangement unit (215), and the rearrangement unit (215) may perform rearrangement using a method of inversely scanning the coefficients based on the scanning order performed by the encoding device.
[0094] The dequantization unit (220) may perform dequantization based on the quantization parameter provided by the encoding device and the rearrangement coefficient of the block.
[0095] The inverse transform unit (225) can perform inverse transforms, i.e., inverse DCT, inverse DST, and inverse KLT, relative to the transforms (i.e., DCT, DST, and KLT) performed by the transform unit on the quantization result performed by the image encoding device. The inverse transform can be performed based on the transmission unit determined by the image encoding device. The inverse transform unit (225) of the image decoding device can selectively perform transform techniques (e.g., DCT, DST, KLT) according to multiple pieces of information such as a prediction method, a size of a current block, and a prediction direction.
[0096] The prediction unit (230, 235) may generate a prediction block based on the provided information for generating the prediction block and information about a previously decoded block or picture. The information for generating the prediction block may be provided from the entropy decoding unit (210). The information about a previously decoded block or picture may be provided from the memory (245).
[0097] As described above, when intra prediction is performed in the same manner as the operation of the image encoding device, if the size of the prediction unit is the same as the size of the transform unit, intra prediction for the prediction unit may be performed based on the pixels on the left, the pixels on the upper left, and the pixels on the top of the prediction unit. However, when intra prediction is performed, if the size of the prediction unit is different from the size of the transform unit, intra prediction for the prediction unit may be performed using reference pixels determined based on the transform unit. In addition, intra prediction using N×N partitioning may be used only for the minimum coding unit.
[0098] The prediction unit (230, 235) may include a prediction unit determination unit, an inter-frame prediction unit, and an intra-frame prediction unit. The prediction unit determination unit may receive various information such as prediction unit information, prediction mode information of an intra-frame prediction method, and motion prediction related information of an inter-frame prediction method from the entropy decoding unit (210), and may determine a prediction unit for a current coding unit. The prediction unit determination unit may determine which of inter-frame prediction and intra-frame prediction is to be performed on the prediction unit. The inter-frame prediction unit (230) may perform inter-frame prediction on the current prediction unit based on information about at least one of a previous picture and a subsequent picture of a current picture including the current prediction unit. In this article, the inter-frame prediction unit (230) may use information required for inter-frame prediction of the current prediction unit provided from the image encoding device. Inter-frame prediction may be performed based on information of a pre-reconstructed partial area in the current picture including the current prediction unit.
[0099] In order to perform inter prediction, whether a motion prediction method of a prediction unit included in a coding unit is a skip mode, a merge mode, an AMVP mode, or an intra block copy mode may be determined in units of coding units.
[0100] The intra prediction unit (235) can generate a prediction block based on pixel information in the current picture. When the prediction unit is a prediction unit on which intra prediction is performed, intra prediction can be performed based on intra prediction mode information about the prediction unit provided from the image encoding device. The intra prediction unit (235) may include an AIS (adaptive intra smoothing) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter performs filtering on the reference pixels of the current block. The AIS filter can determine whether to apply the filter based on the prediction mode of the current prediction unit. AIS filtering can be performed on the reference pixels of the current block using information about the AIS filter provided from the image encoding device and the prediction mode for the prediction unit. When the prediction mode for the current block is a mode in which AIS filtering is not performed, the AIS filter may not be applied.
[0101] When the prediction mode for the prediction unit indicates a prediction mode for performing intra-frame prediction based on pixel values obtained by interpolating reference pixels, the reference pixel interpolation unit may generate reference pixels in units of fractional pixels smaller than integer pixels (i.e., full pixels) by interpolating reference pixels. When the prediction mode for the current prediction unit indicates a prediction mode for generating a prediction block without interpolating reference pixels, the reference pixels may not be interpolated. When the prediction mode for the current block is a DC mode, a DC filter may generate a prediction block by filtering.
[0102] The reconstructed block or the reconstructed picture may be provided to the filter unit (240). The filter unit (240) includes a deblocking filter, an offset unit, and an ALF.
[0103] The image encoding device may provide information about whether to apply a deblocking filter to a corresponding block or picture and information about which of a strong filter and a weak filter to apply when using the deblocking filter. The information about the deblocking filter from the image encoding device may be provided to the deblocking filter of the image decoding device, and the deblocking filter may perform deblocking filtering on the corresponding block.
[0104] The offset unit may apply an offset to the reconstructed picture based on information about an offset type and an offset value applied to the picture during encoding.
[0105] ALF may be applied to the coding unit based on information on whether ALF is applied and ALF coefficient information, etc., provided from the encoding device. The ALF information may be included in and provided in a specific parameter set.
[0106] The memory (245) may store a reconstructed picture or a reconstructed block used as a reference picture or a reference block, and may provide the reconstructed picture to the output unit.
[0107] As described above, in the embodiments of the present invention, for convenience of description, a coding unit is used as only an encoding unit, but the coding unit may be a unit that performs not only encoding but also decoding.
[0108] Figure 3 Block partition types according to an embodiment to which the present invention is applied are shown.
[0109] A block (hereinafter referred to as a first block) may be divided into a plurality of sub-blocks (hereinafter referred to as a second block) by at least one of vertical lines or horizontal lines. The number of each of the vertical lines and the horizontal lines may be one, two or more. Here, the first block may be a coding block (CU) as a basic unit of image encoding / decoding, a prediction block (PU) as a basic unit of prediction encoding / decoding, or a transform block (TU) as a basic unit of transform encoding / decoding. The first block may be a square block or a non-square block.
[0110] The partitioning of the first block may be performed based on a quadtree, a binary tree, a ternary tree, etc., and the reference Figure 3 Describe in detail.
[0111] Figure 3A shows a quadtree partition (QT). QT is a partition type that partitions a first block into four second blocks. For example, when a 2N×2N first block is partitioned by QT, the first block may be partitioned into four second blocks having a size of N×N. QT may be limited to be applicable to only square blocks, but may also be applicable to non-square blocks.
[0112] Figure 3 B shows a horizontal binary tree (hereinafter referred to as horizontal BT) partition. Horizontal BT is a partition type that divides a first block into two second blocks through a horizontal line. The partition can be performed symmetrically or asymmetrically. For example, when a 2N×2N first block is divided based on horizontal BT, the first block can be divided into two second blocks with a height ratio of (a:b). Here, a and b can be the same value, and a can be greater than or less than b.
[0113] Figure 3 C shows a vertical binary tree (hereinafter referred to as vertical BT) partition. Vertical BT is a partition type that divides a first block into two second blocks through a vertical line. The partition can be performed symmetrically or asymmetrically. For example, when a 2N×2N first block is divided based on vertical BT, the first block can be divided into two second blocks with a width ratio of (a:b). Here, a and b can be the same value, and a can be greater than or less than b.
[0114] Figure 3 D shows a horizontal ternary tree (hereinafter referred to as horizontal TT) partition. Horizontal TT is a partition type that divides a first block into three second blocks by two horizontal lines. For example, when a 2N×2N first block is divided based on horizontal TT, the first block may be divided into three second blocks with a height ratio of (a:b:c). Here, a, b, and c may be the same value. Alternatively, a and c may be the same, and b may be greater than or less than a.
[0115] Figure 3 E shows a vertical ternary tree (hereinafter referred to as vertical TT) partition. Vertical TT is a partition type that divides a first block into three second blocks by two vertical lines. For example, when a 2N×2N first block is divided based on vertical TT, the first block can be divided into three second blocks with a width ratio of (a:b:c). Here, a, b, and c can be the same value or different values. Alternatively, a and c can be the same, and b can be greater than or less than a. Alternatively, a and b can be the same, and c can be greater than or less than a. Alternatively, b and c are the same, and a can be greater than or less than b.
[0116] The above-mentioned division may be performed based on division information signaled from the encoding device. The division information may include at least one of division type information, division direction information, or division ratio information.
[0117] The partition type information may specify any one of the partition types predefined in the encoding / decoding device. The predefined partition types may include at least one of QT, horizontal BT, vertical BT, horizontal TT, vertical TT, or non-partition mode (no partitioning). Alternatively, the partition type information may mean information about whether QT, BT, or TT is applied, and may be encoded in the form of a tag or an index. In the case of BT or TT, the partition direction information may indicate whether it is horizontal partitioning or vertical partitioning. In the case of BT or TT, the partition ratio information may indicate the ratio of the width and / or height of the second block.
[0118] Figure 4 A tree-structure based block partitioning method according to an embodiment of the present invention is shown.
[0119] Assumptions Figure 4 The block (400) shown is a square block of size 8N×8N and a partition depth of k (hereinafter referred to as a first block). When the partition information of the first block indicates QT partitioning, the first block can be divided into four sub-blocks (hereinafter referred to as a second block). The size of the second block can be 4N×4N, and the partition depth can be (k+1).
[0120] The four second blocks may be divided again based on QT, BT, TT or non-division mode. For example, when the division information of the second block indicates a horizontal binary tree (horizontal BT), the second block is divided into Figure 4 At this time, the third block may have a size of 4N×2N and may have a partition depth of (k+2).
[0121] The third block may be further divided based on QT, BT, TT or non-division mode. For example, when the division information of the third block indicates a vertical binary tree (vertical BT), the third block is divided into Figure 4 The two sub-blocks (411, 412) shown in FIG. 4A and 4B are shown in FIG. 4B . At this time, the size of the sub-block (411, 412) may be 2N×2N and the division depth may be (k+3). Alternatively, when the division information of the third block indicates a horizontal binary tree (horizontal BT), the third block may be divided into Figure 4 The two sub-blocks (413, 414) are shown. In this case, the sub-block (413, 414) may have a size of 4N×N and a partition depth of (k+3).
[0122] The partitioning may be performed independently or in parallel with adjacent blocks, or may be performed sequentially according to a predetermined priority order.
[0123] The division information of the current block can be determined based on at least one of the division information of the upper block of the current block or the division information of the adjacent block. For example, when the second block is divided based on the horizontal BT and the upper third block is divided based on the vertical BT, it is not necessary to divide the lower third block based on the vertical BT. If the lower third block is divided by the vertical BT, this is the same as the result of dividing the second block by QT. Therefore, the encoding of the division information of the lower third block (especially the division direction information) can be skipped, and the decoding device can be set so that the lower third block is divided in the horizontal direction.
[0124] The upper block may mean a block having a smaller partition depth than the partition depth of the current block. For example, when the partition depth of the current block is (k+2), the partition depth of the upper block may be (k+1). The neighboring block may be a block adjacent to the top or left side of the current block. The neighboring block may be a block having the same partition depth as the current block.
[0125] The above division can be performed repeatedly until the minimum unit of encoding / decoding. When divided into the minimum unit, the division information of the block is no longer notified by signal from the encoding device. The information about the minimum unit may include at least one of the size or shape of the minimum unit. The size of the minimum unit may be represented by width, height, minimum or maximum value of width and height, sum of width and height, number of pixels or division depth. The information about the minimum unit may be notified by signal in at least one of a video sequence, a picture, a slice or a block unit. Alternatively, the information about the minimum unit may be a value predefined in the encoding / decoding device. Information about the minimum unit may be notified by signal for each of CU, PU and TU. Information about a minimum unit may be equally applied to CU, PU and TU.
[0126] Figure 5 An inter-frame prediction method is shown as an embodiment to which the present invention is applied.
[0127] Reference Figure 5 , the motion information of the current block can be obtained (S500).
[0128] The motion information may be derived based on an inter-frame mode predefined in the encoding / decoding apparatus. The predefined inter-frame mode may include at least one of a merge mode, an AMVP mode, an affine mode, or an intra-frame block copy mode. Here, depending on the inter-frame mode, the motion information may be interpreted differently as a motion vector predictor, a motion vector, a control point vector predictor, a control point vector, a block vector, etc.
[0129] In merge mode, the motion information of the current block can be set to be the same as the motion information of the merge candidate. In this way, the motion information is derived by merging with the merge candidate, and no additional motion vector difference (mvd) is signaled. Figure 6 A method for deriving motion information based on a merge mode is described in detail.
[0130] In AMVP mode, the motion vector of the neighboring block can be set as the motion vector prediction value of the current block. To this end, a candidate list including the motion vectors of the spatial / temporal neighboring blocks can be constructed, and the index of any one of the multiple motion vectors of the specified candidate list can be signaled. At the same time, unlike the merge mode, the motion vector can be reconstructed by adding the motion vector prediction value to the motion vector difference signaled.
[0131] In addition, in addition to the translational motion model that only considers translational motion, an additional motion model may be used. For example, a motion model that considers movement such as rotation, perspective, and zooming in / out as well as parallel movement may be used. This may be referred to as an affine mode. In the affine mode, motion information may be derived in units of predetermined sub-blocks based on the control point vector of the current block. Figure 7 and Figure 8 This is described in detail.
[0132] In the case of the IBC mode, the IBC mode is similar to the AMVP mode in that the motion vector of the neighboring block is set as the block vector prediction value (BVP) of the current block, and the block vector is reconstructed using the signaled block vector difference (BVD). However, there is the following difference: the IBC mode performs motion compensation based on a pre-reconstructed area in the same picture as the current block, while the AMVP mode performs motion compensation based on a pre-reconstructed area in a different picture from the current block.
[0133] Meanwhile, the motion information derivation in step S500 may also include a process of adjusting the resolution or accuracy of the motion information, which will be referred to as Figure 5 Describe in detail.
[0134] Reference Figure 5 , motion compensation may be performed on the current block based on the derived motion information (S510).
[0135] A reference block according to a motion vector of the current block may be determined, and reconstructed pixels of the determined reference block may be set as predicted pixels of the current block. The reference block may be determined by applying a predetermined interpolation filter to integer pixels and / or fractional pixels in a reference picture. Figure 6 Describes the detailed interpolation method.
[0136] Meanwhile, in the case of affine mode, a reference block may be specified for each subblock of the current block. The reference block of each subblock may belong to one reference picture. That is, the subblocks belonging to the current block may share one reference picture. Alternatively, a reference picture index may be independently set for each subblock of the current block.
[0137] Meanwhile, the derived motion vector may include at least one of an L0 motion vector and an L1 motion vector. When the motion vector includes an L0 motion vector and an L1 motion vector, the decoding apparatus may perform unidirectional prediction by setting one of the L0 motion vector and the L1 motion vector to 0.
[0138] The setting may be selectively performed in consideration of at least one of the size / shape of the block or the aforementioned inter-mode.Here, the block may mean a current block or a sub-block of the current block.
[0139] For example, if the size of the current block (or a sub-block of the current block) is less than or equal to a predetermined threshold size, unidirectional prediction may be performed by setting one of the L0 motion vector or the L1 motion vector to 0, otherwise, bidirectional prediction may be performed by using the L0 motion vector and the L1 motion vector. Here, the threshold size may be defined as a block size in which at least one of the width and the height is 4, 8, or 16.
[0140] Alternatively, if the type of the subblock of the current block is non-square or triangle, unidirectional prediction may be performed by setting one of the L0 motion vector or the L1 motion vector to 0, otherwise, bidirectional prediction may be performed by using the L0 motion vector and the L1 motion vector.
[0141] Alternatively, when the current block is encoded in the affine mode or the IBC mode, unidirectional prediction may be performed by setting one of the L0 motion vector or the L1 motion vector to 0, otherwise, bidirectional prediction may be performed by using the L0 motion vector and the L1 motion vector.
[0142] Figure 6 A method of deriving motion information according to a merge mode is shown as an embodiment of the present invention.
[0143] Reference Figure 6 , a merge candidate list of the current block can be constructed (S600).
[0144] The merge candidate list may include at least one of a spatial merge candidate or a temporal merge candidate of the current block.
[0145] The motion information of the spatial merge candidate can be derived from the motion information of the spatial neighboring blocks of the current block. Here, the spatial neighboring block is a block belonging to the same picture as the current block, and may mean a block adjacent to the current block. The spatial neighboring block may include a block adjacent to at least one of the left, top, right, bottom, left, or top left of the current block. The top left neighboring block can be used only when at least one of the blocks adjacent to the left, top, right, and bottom left is not available.
[0146] The motion information of the temporal merge candidate can be derived from the motion information of the temporal neighboring blocks of the current block. The temporal neighboring block is a block belonging to a different picture from the current block, and can be defined as a block at the same position as the current block. Here, the block at the same position may mean at least one of the following blocks: a block (BR) adjacent to the lower right corner of the current block, a block (CTR) including the position of the center sample of the current block, or a block (TL) including the position of the upper left sample of the current block. Alternatively, the block at the same position may mean a block including a position shifted by a predetermined disparity vector from the position of the upper left sample of the current block. Here, the disparity vector can be determined based on any one of the motion vectors of the spatial neighboring blocks described above. Alternatively, the disparity vector can be determined based on a combination of at least two of the motion vectors of the above-mentioned spatial neighboring blocks. The combination may mean operations such as maximum, minimum, median, and weighted average. For example, the disparity vector may be set to the motion vector of the left neighboring block or the upper neighboring block. Alternatively, the disparity vector may be set to the median or average between the motion vector of the left neighboring block and the motion vector of the lower left neighboring block.
[0147] The reference picture index and motion vector of the temporal merge candidate may be derived from the reference picture index and motion vector of the temporal neighboring block, respectively. Alternatively, the motion vector of the temporal merge candidate may be derived as the motion vector of the temporal neighboring block, and the reference picture index of the temporal merge candidate may be set to a default value (e.g., 0) pre-assigned to the decoding device, regardless of the temporal neighboring block.
[0148] The merge candidate list may also include a combined merge candidate. The combined merge candidate may be derived by combining n merge candidates belonging to the pre-generated merge candidate list.
[0149] Here, n may be an integer of 2, 3, 4 or more. The number n of merge candidates to be combined may be a fixed value pre-assigned to the encoding / decoding device, or may be encoded and signaled by the encoding device. Signaling may be performed in at least one unit of a sequence, a picture, a slice, a tile, a sub-tile (brick), or a predetermined block. The number n of merge candidates to be combined may be variably determined based on the number of remaining merge candidates. Here, the number of remaining merge candidates may mean the difference between the maximum number of merge candidates that can be included in the merge candidate list and the current number of merge candidates in the merge candidate list. The maximum number may be a number pre-assigned to the encoding / decoding device, or may be encoded and signaled by the encoding device. The current number may mean the number of merge candidates constructed before adding the combined merge candidate. For example, when the number of remaining merge candidates is 1, two merge candidates may be used, and when the number of remaining merge candidates is greater than 1, three or more merge candidates may be used.
[0150] The positions of the n merge candidates may be predetermined positions in the merge candidate list. For example, an index (0 to (k-1)) may be assigned to each merge candidate belonging to the merge candidate list. Here, k may mean the total number of merge candidates included in the merge candidate list. In this case, the positions of the n merge candidates may correspond to index 0 to index (n-1) in the merge candidate list. Alternatively, the n merge candidates may be determined by considering the prediction direction of each merge candidate included in the merge candidate list. For example, from among the merge candidates included in the merge candidate list, only merge candidates that are bidirectionally predicted may be selectively used, or only merge candidates that are unidirectionally predicted may be selectively used.
[0151] The combined merge candidate may be derived using both the spatial merge candidate and the temporal merge candidate, or may be derived using only one of the spatial merge candidate or the temporal merge candidate. For example, the combined merge candidate may be limited to be derived using only the spatial merge candidate. In this case, the number of merge candidates to be combined may be limited to the number of spatial merge candidates belonging to the pre-generated merge candidate list.
[0152] A combined merge candidate may be added after the spatial / temporal merge candidate in the merge candidate list. That is, the index of the combined merge candidate may be greater than the index of the spatial / temporal merge candidate. Alternatively, the combined merge candidate may be added between the spatial merge candidate and the temporal merge candidate in the merge candidate list. That is, the index of the combined merge candidate may be greater than the index of the spatial merge candidate and less than the index of the temporal merge candidate. Alternatively, the position of the combined merge candidate may be variably determined in consideration of the prediction direction of the combined merge candidate. The position of the combined merge candidate in the merge candidate list may be rearranged according to whether the prediction direction of the combined merge candidate is bidirectional prediction. For example, when the combined merge candidate is bidirectional prediction, an index less than the spatial merge candidate or the temporal merge candidate may be assigned, otherwise, an index greater than the spatial merge candidate or the temporal merge candidate may be assigned.
[0153] Hereinafter, for convenience of description, a method of deriving a combined merge candidate based on two merge candidates will be described.
[0154] The motion information of the combined merge candidate can be derived by weighted averaging the motion information of the first merge candidate and the second merge candidate. Here, the weights of the weighted average are [1:1], [1:2], [1:3], [2:3], etc., but are not limited to this. The weights can be predefined in the encoding / decoding device or derived from the decoding device. In this case, the weights can be derived by considering at least one of the distance between the reference picture of the merge candidate and the current picture or the prediction direction of the merge candidate. Alternatively, the motion information of the combined merge candidate can be derived by obtaining the motion information in the L0 direction from the first merge candidate and the motion information in the L1 direction from the second merge candidate and combining them. The motion information of the combined merge candidate can be derived based on at least one of the above-mentioned derivation methods, and this can be performed by considering the prediction direction of the merge candidates to be combined as described later.
[0155] In the present specification, the motion information may include at least one of a prediction direction flag, a reference picture index, or a motion vector. Each of the motion information may be defined for L0 prediction and L1 prediction. Here, L0 prediction refers to prediction with reference to reference picture list L0, and L1 prediction means prediction with reference to reference picture list L1.
[0156] 1. When the first merging candidate and the second merging candidate are unidirectional predictions
[0157] (Case 1) When the first merge candidate is L0 prediction and the second merge candidate is L1 prediction, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the first merge candidate. The prediction direction flag in the L0 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the motion vector of the first merge candidate. The reference picture index in the L1 direction of the combined merge candidate may be derived as the reference picture index of the second merge candidate. The prediction direction flag in the L1 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the motion vector of the second merge candidate.
[0158] (Case 2) When the first merge candidate is L1 prediction and the second merge candidate is L0 prediction, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the second merge candidate. The prediction direction flag in the L0 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the motion vector of the second merge candidate. The reference picture index in the L1 direction of the combined merge candidate may be derived as the reference picture index of the first merge candidate. The prediction direction flag in the L1 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the motion vector of the first merge candidate.
[0159] (Case 3) When the first merge candidate and the second merge candidate are L0 predictions, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of any one of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index in the L0 direction of the combined merge candidate. The prediction direction flag in the L0 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the weighted average of the motion vectors of the first merge candidate and the second merge candidate. The reference picture index in the L1 direction of the combined merge candidate may be derived as -1, the prediction direction flag in the L1 direction may be derived as 0, and the motion information in the L1 direction may be derived as 0.
[0160] (Case 4) When the first merge candidate and the second merge candidate are L1 predictions, the reference picture index of the combined merge candidate in the L0 direction may be derived as -1, the prediction direction flag in the L0 direction may be derived as 0, and the motion information in the L0 direction may be derived as 0. The reference picture index of the combined merge candidate in the L1 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index of the combined merge candidate in the L1 direction. The prediction direction flag in the L1 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the weighted average of the motion vectors of the first merge candidate and the second merge candidate.
[0161] 2. When both the first merging candidate and the second merging candidate are bidirectional predictions
[0162] (Case 5) The reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index in the L0 direction of the combined merge candidate. The prediction direction mark in the L0 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the weighted average of the motion vectors of the first merge candidate and the second merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index of the combined merge candidate in the L1 direction. The prediction direction mark in the L1 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the weighted average of the motion vectors of the first merge candidate and the second merge candidate.
[0163] 3. When the first merge candidate is bidirectionally predicted and the second merge candidate is unidirectionally predicted
[0164] (Case 6) When the second merge candidate is an L0 prediction, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index in the L0 direction of the combined merge candidate. The prediction direction mark in the L0 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the weighted average of the motion vectors of the first merge candidate and the second merge candidate. The reference picture index in the L1 direction of the combined merge candidate may be derived as the reference picture index of the first merge candidate. The prediction direction mark in the L1 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the motion vector of the first merge candidate.
[0165] (Case 7) When the second merge candidate is an L1 prediction, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the first merge candidate. The prediction direction mark in the L0 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the motion vector of the first merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index of the combined merge candidate in the L1 direction. The prediction direction mark in the L1 direction of the combined merge candidate may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the weighted average of the motion vectors of the first merge candidate and the second merge candidate.
[0166] Reference Figure 6 , the motion information of the current block can be obtained according to the merge candidate list (S610).
[0167] Specifically, a merge index of the current block may be signaled. The merge index may specify any one of a plurality of merge candidates belonging to a merge candidate list. A merge candidate having the same index as the merge index may be extracted, and the motion information of the current block may be derived using the motion information of the extracted merge candidate. For example, the motion vector, reference picture index, and prediction direction information of the current block may be set to be the same as the motion vector, reference picture index, and prediction direction information of the extracted merge candidate.
[0168] Figure 7 A method of deriving motion information based on an affine model is shown as an embodiment to which the present invention is applied.
[0169] Reference Figure 7, a candidate list for predicting motion information of the current block may be generated ( S700 ).
[0170] The candidate list may include one or more candidates based on an affine model (hereinafter referred to as affine candidates). An affine candidate may mean a candidate having a control point vector. A control point vector means a motion vector of a control point for an affine model, and the control point vector may be defined for a corner position of a block (e.g., at least one of the upper left corner, the upper right corner, the lower left corner, or the lower right corner).
[0171] The affine candidate may include at least one of a spatial candidate, a temporal candidate, or a construction candidate. Here, the spatial candidate may be derived from a vector of a neighboring block spatially adjacent to the current block, and the temporal candidate may be derived from a vector of a neighboring block temporally adjacent to the current block. Here, the neighboring block may mean a block encoded using an affine model. The vector may mean a motion vector or a control point vector. Meanwhile, the construction candidate may be derived based on a combination of motion vectors of spatial / temporal neighboring blocks of the current block.
[0172] Will refer to Figure 8 A method of deriving spatial / temporal candidates based on vectors of spatial / temporal neighboring blocks is described in detail.
[0173] The above-mentioned multiple affine candidates can be arranged in the candidate list based on a predetermined priority. For example, multiple affine candidates can be arranged in the candidate list in the order of spatial candidates, temporal candidates, and construction candidates. Alternatively, multiple affine candidates can be arranged in the candidate list in the order of temporal candidates, spatial candidates, and construction candidates. However, the present invention is not limited to this, and the temporal candidates can also be arranged after the construction candidates. Alternatively, some of the construction candidates can be arranged before the spatial candidates, and the remaining construction candidates can be arranged after the spatial candidates.
[0174] A control point vector of the current block may be derived based on the candidate list and the candidate index (S710).
[0175] The candidate index may mean an index that is encoded to derive a control point vector of the current block. The candidate index may specify any one of the multiple affine candidates included in the candidate list. Here, the affine candidate may mean the position of the control point vector of the affine candidate or the spatial / temporal candidate. The control point vector of the current block may be derived using the affine candidate specified by the candidate index.
[0176] Specifically, it is assumed that the type of the affine model of the current block is 4 parameters (i.e., when it is determined that the current block uses two control point vectors). In this case, when the affine candidate specified by the candidate index has three control point vectors, only two control point vectors (e.g., control point vectors of Idx=0, 1) may be selected from among the three control point vectors and may be set as the control point vectors of the current block. Alternatively, the three control point vectors of the specified affine candidate may be set as the control point vectors of the current block. In this case, the type of the affine model of the current block may be updated to 6 parameters.
[0177] On the other hand, it is assumed that the type of the affine model of the current block is 6 parameters (i.e., when it is determined that the current block uses three control point vectors). In this case, when the affine candidate specified by the candidate index has two control point vectors, one additional control point vector may be generated, and the two control point vectors of the affine candidate and the additional control point vector may be set as the control point vector of the current block. The additional control point vector may be derived based on at least one of the two control point vectors of the affine candidate, the size of the current / neighboring block, or the position information of the current / neighboring block.
[0178] Alternatively, the two control point vectors of the specified affine candidate may be set as the control point vectors of the current block. In this case, the type of the affine model of the current block may be updated to 4 parameters.
[0179] Alternatively, the control point vector of the specified affine candidate may be set as the control point vector prediction value (cpmvp) of the current block. In this case, the control point vector difference (cpmvd) may be signaled for the current block. The number of control point vector differences signaled may be determined according to the type of the affine model of the current block. That is, when the type of the affine model of the current block is 4 parameters, the mvd_coding() function may be called twice to obtain the difference of two control point vectors. Similarly, when the type of the affine model of the current block is 6 parameters, the mvd_coding() function may be called three times to obtain the difference of three control point vectors. The control point vector (cpmv) may be reconstructed using the control point vector prediction value (cpmvp) and the control point vector difference (cpmvd) of the current block.
[0180] A motion vector of the current block may be derived based on the control point vector of the current block (S720).
[0181] The motion vector can be derived in units of the current block or sub-block. To this end, the current block can be divided into a plurality of N×M sub-blocks. Here, the N×M sub-blocks can be in the form of rectangles (N>M or N<M) or squares (N=M). The values of N and M can be 4, 8, 16, 32 or more.
[0182] For example, the size / shape of the sub-block may be a fixed size / shape predefined in the decoding device. For example, the size / shape of the sub-block may be a square such as 4×4, 8×8, 16×16, etc., or a non-square such as 2×4, 2×8, 4×8, 4×16, etc. Alternatively, the sub-block may be defined as a block whose sum of width and height is 8, 12, 16 or more. Alternatively, the sub-block may be defined as a block whose product of width and height is an integer of 16, 32, 64 or more.
[0183] Alternatively, the size / shape of the sub-block can be variably derived based on the properties of the above-mentioned blocks. For example, when the size of the current block is greater than or equal to a predetermined threshold size, the current block can be divided into units of a first sub-block (e.g., 8×8, 16×16), otherwise, the current block can be divided into units of a second sub-block (e.g., 4×4).
[0184] Alternatively, the information about the size / shape of the sub-block may be encoded by the encoding device and signaled. The information indicates at least one of the size or shape of the sub-block, and the information may be signaled in at least one level of a sequence, a picture, a tile group, a tile, and a CTU.
[0185] The control point representing the current block may include an upper left position and an upper right position. However, the present invention is not limited thereto, and the control point may include three points of the upper left position, the upper right position and the lower left position, or may further include a plurality of additional points.
[0186] When two control points are used, the motion vector of each sub-block of the current block can be derived by using a first control point vector corresponding to the first control point, a second control point vector corresponding to the second control point, the position (x, y) of the sub-block, or at least one of the size (width or height) of the current block.
[0187] When three control points are used, the motion vector of each sub-block of the current block can be derived by using at least one of a first control point vector corresponding to the first control point, a second control point vector corresponding to the second control point, a third control point vector corresponding to the third control point, the position (x, y) of the sub-block, or the size (width or height) of the current block. In this case, a difference vector between the second control point vector and the first control point vector can be used, and a difference vector between the third control point vector and the first control point vector can be used. The difference vector can be calculated for each of the horizontal direction (x-axis direction) and the vertical direction (y-axis direction).
[0188] The motion vector derivation process may also include a process of applying a predetermined offset to a motion vector derived based on a control point vector. The offset may refer to a vector used to improve a pre-derived motion vector. The offset may be determined based on information about at least one of an absolute value or a direction of the offset. The absolute value may be an integer of 1, 2, 3 or more. The direction may include at least one of left, right, up, and down. The information about the absolute value and / or direction of the offset may be encoded by the encoding device and signaled. Alternatively, the absolute value of the offset may be a predetermined fixed value in the decoding device.
[0189] Figure 8 A method of deriving affine candidates based on control point vectors of spatial / temporal neighboring blocks is provided as an embodiment to which the present invention is applied.
[0190] The width and height of the current block (800) are cbW and cbH, respectively, and the position of the current block is (xCb, yCb). The width and height of the spatial neighboring blocks (810 to 850) are nbW and nbH, respectively, and the position of the spatial neighboring blocks is (xNb, yNb). The spatial neighboring blocks may include at least one of the left block (810), the lower left block (840), the upper right block (830), the upper block (820), or the upper left block (850) of the current block. Alternatively, the spatial neighboring blocks may also include at least one of the blocks adjacent to the right side of the upper left block (850) or the blocks adjacent to the lower side.
[0191] The spatial candidate may have n control point vectors (cpMVs). Here, the n value may be an integer of 1, 2, 3 or more. The n value may be determined based on at least one of information about whether to decode in units of subblocks, information about whether to encode the block using an affine model, or information about the type of the affine model (4 parameters or 6 parameters).
[0192] For example, according to the information, when a block is decoded in units of subblocks or a block is a block encoded by an affine model, the block may have two control point vectors. On the other hand, if this is not the case, the block may not perform prediction based on the affine model.
[0193] Alternatively, according to the information, when the block is a block encoded using an affine model and the type of the affine model is 6 parameters, the block may have three control point vectors. On the other hand, if not, the block may not perform prediction based on the affine model.
[0194] The above information can be encoded by the encoding device and notified by signal. Alternatively, all or part of the information can be derived from the decoding device based on the properties of the block. Here, the block can mean the current block or the spatial / temporal neighboring blocks of the current block. The attribute can mean size, shape, position, segmentation type, inter-frame mode, parameters related to residual coefficients, etc. The inter-frame mode can be a mode predefined in the decoding device, and can mean a merge mode, a skip mode, an AMVP mode, an affine model, an intra / inter-frame combination mode, a current picture reference mode, etc. Alternatively, the value of n can be derived from the decoding device based on the above block properties.
[0195] In this embodiment, the n control point vectors may be expressed as a first control point vector (cpMV[0]), a second control point vector (cpMV[1]), a third control point vector (cpMV[2]), ... the nth control point vector (cpMV[n-1]).
[0196] For example, the first control point vector (cpMV[0]), the second control point vector (cpMV[1]), the third control point vector (cpMV[2]), and the fourth control point vector (cpMV[3]) may be vectors corresponding to the positions of the upper left sample, the upper right sample, the lower left sample, and the lower right sample, respectively. Here, it is assumed that the spatial candidate may have three control point vectors, and these three control point vectors may be any control point vectors selected from the first control point vector to the nth control point vector. However, the present invention is not limited to this, and the spatial candidate may have two control point vectors, and these two control point vectors may be any control point vectors selected from the first control point vector to the nth control point vector.
[0197] Hereinafter, a method of deriving control point vectors of spatial candidates will be described.
[0198] 1. When the boundary of the current block is inconsistent with the CTU boundary
[0199] The first control point vector may be derived based on at least one of the first control point vector of the spatially neighboring block, a predetermined difference, position information (xCb, yCb) of the current block, or position information (xNb, yNb) about the spatially neighboring block.
[0200] The number of difference values may be 1, 2, 3 or more. The number of difference values may be variably determined in consideration of the above-mentioned properties of the block, or the number of difference values may be a fixed value pre-assigned to the decoding device. The difference value may be defined as a difference between one control point vector and another control point vector among a plurality of control point vectors. For example, the difference value may include at least one of the following differences: a first difference between the second control point vector and the first control point vector, a second difference between the third control point vector and the first control point vector, a third difference between the fourth control point vector and the third control point vector, or a fourth difference between the fourth control point vector and the second control point vector.
[0201] For example, the first control point vector may be obtained as in the following equation 1.
[0202] [Formula 1]
[0203] cpMvLX[0][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb-yNb))
[0204] cpMvLX[0][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb-yNb))
[0205] In Formula 1, the variables mvScaleHor and mvScaleVer may refer to the first control point vector of the spatially adjacent block, or a value obtained by applying a shift operation of k to the first control point vector. Here, k may be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9 or more. The variables dHorX and dVerX correspond to the x component and y component of the first difference between the second control point vector and the first control point vector, respectively. The variables dHorY and dVerY correspond to the x component and y component of the second difference between the third control point vector and the first control point vector, respectively. The above variables may be derived as in Formula 2 below.
[0206] [Formula 2]
[0207] mvScaleHor=CpMvLX[xNb][yNb][0][0]<<7
[0208] mvScaleVer=CpMvLX[xNb][yNb][0][1]<<7
[0209] dHorX=(CpMvLX[xNb+nNbW-1][yNb][1][0]-CpMvLX[xNb][yNb][0][0])<<(7-log2NbW)
[0210] dVerX=(CpMvLX[xNb+nNbW-1][yNb][1][1]-CpMvLX[xNb][yNb][0][1])<<(7-log2NbW)
[0211] dHorY=(CpMvLX[xNb][yNb+nNbH-1][2][0]-CpMvLX[xNb][yNb][2][0])<<(7-log2NbH)
[0212] dVerY=(CpMvLX[xNb][yNb+nNbH-1][2][1]-CpMvLX[xNb][yNb][2][1])<<(7-log2NbH)
[0213] The second control point vector may be derived based on at least one of the first control point vector of the spatially neighboring block, a predetermined difference, the position information (xCb, yCb) of the current block, the block size (width or height), or the position information (xNb, yNb) of the spatially neighboring block. Here, the block size may mean the size of the current block and / or the spatially neighboring block. The difference is as described in the first control point vector, and a detailed description thereof will be omitted. However, the range and / or number of the difference used in deriving the second control point vector may be different from that of the first control point vector.
[0214] For example, the second control point vector may be obtained as in the following equation 3.
[0215] [Formula 3]
[0216] cpMvLX[1][0]=(mvScaleHor+dHorX*(xCb+cbWidth-xNb)+dHorY*(yCb-yNb))
[0217] cpMvLX[1][1]=(mvScaleVer+dVerX*(xCb+cbWidth-xNb)+dVerY*(yCb-yNb))
[0218] In Formula 3, variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Formula 1, and a detailed description thereof will be omitted.
[0219] The third control point vector may be derived based on at least one of the first control point vector of the spatially neighboring block, a predetermined difference, the position information (xCb, yCb) of the current block, the block size (width or height), or the position information (xNb, yNb) of the spatially neighboring block. Here, the block size may mean the size of the current block and / or the spatially neighboring block. The difference is as described in the first control point vector, and its detailed description will be omitted. However, the range and / or number of the difference used in the process of deriving the third control point vector may be different from that of the first control point vector or the second control point vector.
[0220] For example, the third control point vector may be obtained as in Equation 4 below.
[0221] [Formula 4]
[0222] cpMvLX[2][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb+cbHeight-yNb))
[0223] cpMvLX[2][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb+cbHeight-yNb))
[0224] In Formula 4, variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Formula 1, and a detailed description thereof will be omitted. Meanwhile, through the above process, the nth control point vector of the spatial candidate can be obtained.
[0225] 2. When the boundary of the current block coincides with the boundary of the CTU
[0226] The first control point vector may be derived based on at least one of a motion vector (MV) of a spatially neighboring block, a predetermined difference, position information (xCb, yCb) of a current block, or position information (xNb, yNb) of a spatially neighboring block.
[0227] The motion vector may be a motion vector of a subblock located at the bottom of a spatially adjacent block. The subblock may be located at the leftmost, center, or rightmost among a plurality of subblocks located at the bottom of a spatially adjacent block. Alternatively, the motion vector may mean an average value, a maximum value, or a minimum value of the motion vectors of the subblocks.
[0228] The number of difference values may be 1, 2, 3 or more. The number of difference values may be variably determined in consideration of the above-mentioned properties of the block, or the number of difference values may be a fixed value pre-assigned to the decoding device. The difference value may be defined as a difference between one motion vector and another motion vector among a plurality of motion vectors stored in units of sub-blocks in spatially adjacent blocks. For example, the difference value may mean a difference between a motion vector of a lower right sub-block and a motion vector of a lower left sub-block of a spatially adjacent block.
[0229] For example, the first control point vector may be obtained as in the following equation 5.
[0230] [Formula 5]
[0231] cpMvLX[0][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb-yNb))
[0232] cpMvLX[0][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb-yNb))
[0233] In Formula 5, the variables mvScaleHor and mvScaleVer may refer to the motion vector (MV) of the above-mentioned spatial neighboring block, or a value obtained by applying a shift operation of k to the motion vector. Here, k may be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9 or more.
[0234] The variables dHorX and dVerX correspond to the x component and y component of the predetermined difference, respectively. Here, the difference means the difference between the motion vector of the lower right sub-block and the motion vector of the lower left sub-block in the spatial neighboring block. The variables dHorY and dVerY can be derived based on the variables dHorX and dVerX. The above variables can be derived as in the following formula 6.
[0235] [Formula 6]
[0236] mvScaleHor=MvLX[xNb][yNb+nNbH-1][0]<<7
[0237] mvScaleVer=MvLX[xNb][yNb+nNbH-1][1]<<7
[0238] dHorX=(MvLX[xNb+nNbW-1][yNb+nNbH-1][0]-MvLX[xNb][yNb+nNbH-1][0])<<(7-log2NbW)
[0239] dVerX=(MvLX[xNb+nNbW-1][yNb+nNbH-1][1]-MvLX[xNb][yNb+nNbH-1][1])<<(7-log2NbW)
[0240] dHorY=-dVerX
[0241] dVerY=dHorX
[0242] The second control point vector may be derived based on at least one of a motion vector (MV) of a spatially neighboring block, a predetermined difference, position information (xCb, yCb) of a current block, a block size (width or height), or position information (xNb, yNb) of a spatially neighboring block. Here, the block size may mean the size of the current block and / or the spatially neighboring block. The motion vector and the difference are as described in the first control point vector, and a detailed description thereof will be omitted. However, the position of the motion vector used in the derivation of the second control point vector, the range of the difference, and / or the number may be different from those of the first control point vector.
[0243] For example, the second control point vector may be obtained as in the following equation 7.
[0244] [Formula 7]
[0245] cpMvLX[1][0]=(mvScaleHor+dHorX*(xCb+cbWidth-xNb)+dHorY*(yCb-yNb))
[0246] cpMvLX[1][1]=(mvScaleVer+dVerX*(xCb+cbWidth-xNb)+dVerY*(yCb-yNb))
[0247] In Equation 7, variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 5, and a detailed description thereof will be omitted.
[0248] The third control point vector may be derived based on at least one of a motion vector (MV) of a spatially neighboring block, a predetermined difference, position information (xCb, yCb) of a current block, a block size (width or height), or position information (xNb, yNb) of a spatially neighboring block. Here, the block size may mean the size of the current block and / or the spatially neighboring block. The motion vector and the difference are as described in the first control point vector, and a detailed description thereof will be omitted. However, the position of the motion vector used in the derivation of the third control point vector, the range of the difference, and / or the number may be different from those of the first control point vector or the second control point vector.
[0249] For example, the third control point vector may be obtained as in the following equation 8.
[0250] [Formula 8]
[0251] cpMvLX[2][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb+cbHeight-yNb))
[0252] cpMvLX[2][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb+cbHeight-yNb))
[0253] In Formula 8, variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Formula 5, and a detailed description thereof will be omitted. Meanwhile, through the above process, the nth control point vector of the spatial candidate can be obtained.
[0254] The above process of deriving an affine candidate may be performed for each predefined spatial neighboring block. The predefined spatial neighboring block may include at least one of a left block, a lower left block, an upper right block, an upper block, or an upper left block of the current block.
[0255] Alternatively, the process of deriving affine candidates may be performed for each group of spatially neighboring blocks.Here, the spatially neighboring blocks may be classified into a first group including a left block and a lower left block and a second group including an upper right block, an upper block, and an upper left block.
[0256] For example, an affine candidate may be derived from a spatially adjacent block belonging to the first group. The deriving may be performed until an available affine candidate is found based on a predetermined priority. The priority may be in the order of left block -> lower left block, or lower left block -> left block. Based on the priority, it is determined whether the spatially adjacent block in the first group is a block decoded by prediction based on an affine model, and the first block decoded by prediction based on an affine model may be selected as an affine candidate.
[0257] Similarly, an affine candidate may be derived from the spatially adjacent blocks belonging to the second group. The deriving may be performed until an available affine candidate is found based on a predetermined priority. The priority may be in the order of upper right block->upper block->upper left block, or upper left block->upper block->upper right block. Based on the priority, it is determined whether the spatially adjacent blocks in the second group are blocks decoded by prediction based on an affine model, and the first block decoded by prediction based on an affine model may be selected as an affine candidate.
[0258] The above-mentioned embodiments may be applied to temporally adjacent blocks in the same / similar manner. Here, the temporally adjacent block may be a block that belongs to a different picture than the current block but is at the same position as the current block. The collocated block may be a block that includes the position of the upper left sample of the current block, the center position, or the position of a sample adjacent to the lower right sample of the current block.
[0259] Alternatively, the temporally neighboring block may mean a block at a position shifted by a predetermined disparity vector from a block at the same position.Here, the disparity vector may be determined based on a motion vector of any one of the above-mentioned spatially neighboring blocks of the current block.
[0260] Fig. 9 A method of adaptively adjusting the resolution of a motion vector is shown as an embodiment to which the present invention is applied.
[0261] Reference Fig. 9 , the resolution of the motion vector of the current block can be determined (S900).
[0262] Here, the motion vector may mean a pre-reconstructed motion vector (mv), but may be replaced by a motion vector predictor (mvp) or a motion vector difference (mvd).
[0263] The encoding / decoding device may select any one of a plurality of predefined resolution candidates and determine it as the resolution of the motion vector. The plurality of resolution candidates may include at least one of 1 / 16 pixel, 1 / 8 pixel, 1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels or 4 pixels.
[0264] The selection may be performed in higher-level units such as a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, etc. That is, motion compensation may be performed on all blocks referring to the corresponding higher-level unit based on motion vectors having the same resolution. Meanwhile, the selection may be performed in lower-level units such as coding blocks, prediction blocks, transform blocks, sub-blocks, etc. That is, motion compensation may be performed based on motion vectors having different resolutions for each corresponding lower level.
[0265] The selection may be performed based on an index specifying any one of a plurality of resolution candidates (Implementation 1). In this case, the resolution candidates corresponding to each index may be defined as shown in Table 1, and the index may be an integer in the range of 0 to (the number of resolution candidates - 1).
[0266] [Table 1]
[0267] Index(idx) Resolution Candidates 0 1 / 16 Pixel 1 1 / 4 pixel 2 1 / 2 pixel 3 1 pixel 4 4 pixels
[0268] Table 1 is only an example of the index assigned to each resolution candidate, and does not limit the priority order and the number of resolution candidates. For example, a 1 / 4 pixel may have a higher priority than a 1 / 16 pixel, and an index smaller than the 1 / 16 pixel may be assigned to the 1 / 4 pixel. Alternatively, a 1 / 4 pixel or a 1 / 2 pixel may have an index smaller than the 1 / 16 pixel.
[0269] The number of resolution candidates available for the current block may be at most N, and N may be a natural number of 1, 2, 3, 4, 5 or more. The number may be a fixed value pre-assigned to the encoding / decoding device. Alternatively, the number may be variably determined in consideration of the encoding parameters of the current block. The encoding parameters may be variably determined in consideration of at least one of the size / shape / position of the block, whether bidirectional prediction is performed, the inter-frame mode or the component type. For example, if the size of the current block is less than the threshold size, p resolution candidates may be used, otherwise, q resolution candidates may be used. Here, p may be a natural number less than q. Alternatively, when the inter-frame mode of the current block is an affine mode, the number of resolution candidates may be two or three. Alternatively, when the inter-frame mode of the current block is an AMVP mode, the number of resolution candidates may be 3, 4 or 5. And, when the inter-frame mode of the current block is an IBC mode, the number of resolution candidates may be 1, 2 or 3. Alternatively, it may be set to use the same number of resolution candidates regardless of the above-mentioned block size or the above-mentioned inter-frame mode.
[0270] Alternatively, two indexes can be used to specify any one of a plurality of resolution candidates (implementation 2). The two indexes are referred to as the first index and the second index. The first index is information encoded to specify the resolution of the motion vector. For example, if the first index is a first value, the resolution of the motion vector can be determined as 1 / 4 pixel, and otherwise, the resolution of the motion vector can be determined based on the second index. For example, the inter-frame mode shown in Table 2 below can be considered to define the resolution of the motion vector for each of the first index and the second index.
[0271] [Table 2]
[0272]
[0273] Table 2 is only an example of the candidate resolutions for each second index, and does not limit the corresponding resolution candidates for each second index. For example, in affine mode, 1 / 2 pixel, 1 / 8 pixel, etc. may correspond to second index 0, and 2 pixels, 4 pixels, etc. may correspond to second index 1. Similarly, in the case of IBC mode, 2 pixels may correspond to second index 0 or 1. However, unlike Table 2, the same resolution candidate may be configured regardless of the inter-frame mode, or the same resolution candidate may be configured for the IBC mode and the AMVP mode. The same resolution candidate may be configured with at least two of 1 / 4 pixel, 1 / 2 pixel, 1 pixel, or 4 pixels.
[0274] The first index may be signaled when the inter mode of the current block is AMVP mode or affine mode (condition 1). In addition, the first index may be signaled only when a non-zero motion vector difference or control point vector difference is signaled for the current block (condition 2). The first index may be signaled based on a flag indicating whether adaptive motion vector resolution is used in motion vector encoding / decoding. That is, the first index may be signaled only when the flag is a first value (condition 3). The flag may be signaled at a higher level of at least one of a video parameter set (VPS), a sequence parameter set (SPS), or a picture parameter set (PPS). The first index may be signaled when all conditions 1 to 3 are met, or the first index may be signaled when at least one of conditions 1 to 3 is met. The second index may be signaled only when the resolution of the motion vector is not 1 / 4 pixel according to the first index.
[0275] Reference Fig. 9 , a motion vector prediction value corresponding to the determined resolution can be obtained (S910).
[0276] For example, when the resolution of the motion vector is determined to be 1 / 4 pixel and the resolution of the pre-derived motion vector prediction value is 1 / 16 pixel, the resolution can be adjusted to 1 / 4 pixel by shifting the motion vector prediction value by 2. Alternatively, when the resolution of the motion vector is determined to be 1 pixel and the resolution of the pre-derived motion vector prediction value is 1 / 16 pixel, the resolution can be adjusted to 1 pixel by shifting the motion vector prediction value by 4. That is, by applying a predetermined shift value to the pre-derived vector component, the resolution or accuracy of the vector component can be adjusted, which will be called rounding.
[0277] Assume that the resolution of the motion vector prediction value before rounding is 1 / 16 pixel. In this case, the shift value for rounding can be defined as shown in Table 2.
[0278] In AMVP mode, rounding can be performed after specifying any one of the multiple motion vectors included in the candidate list. That is, rounding can be performed only on one candidate specified by an index without performing rounding according to a determined resolution for each of the multiple candidates constituting the candidate list.
[0279] Alternatively, in AMVP mode, rounding can be performed on each motion vector of the spatial / temporal neighboring blocks, and a candidate list can be constructed based on the rounded motion vectors. In this case, a motion vector specified from the candidate list is set as the motion vector prediction value of the current block, and no additional rounding is performed.
[0280] Likewise, in the affine mode, rounding may be performed after specifying any one of the plurality of candidates included in the candidate list. That is, rounding may be performed only on one candidate specified by an index without performing rounding according to a determined resolution for each of the plurality of candidates constituting the candidate list. Some of the plurality of candidates may be constructed as control point vectors, while other candidates may be constructed as motion vectors.
[0281] Alternatively, in affine mode, rounding may be performed on each vector of the spatial / temporal neighboring blocks, and a candidate list may be constructed based on the rounded vectors. In this case, a vector specified from the candidate list is set as the control point vector prediction value (cpmvp) of the current block, and no additional rounding may be performed on the control point vector prediction value.
[0282] Reference Fig. 9 , based on the determined resolution, the motion vector difference may be rounded (S920).
[0283] The motion vector difference value can be reconstructed based on predetermined coding information. The coding information can be notified by signaling to decode the motion vector difference. When the current block is encoded in AMVP mode, a motion vector difference can be signaled by calling the mvd_coding() function once. Alternatively, when the current block is encoded in affine mode, the motion vector difference of S920 can be understood as a control point vector difference. In this case, when the type of the affine model of the current block is 4 parameters, the mvd_coding() function can be called twice to obtain two control point vector differences. Similarly, when the affine mode type of the current block is 6 parameters, the mvd_coding() function can be called three times to obtain three control point vector differences.
[0284] Based on the resolution determined in S900, the motion vector difference may be rounded. That is, the motion vector difference may be shifted by the shift value defined in Table 2 to adjust the resolution of the motion vector difference. Alternatively, the resolution of the control point vector difference may be adjusted by shifting each of the plurality of control point vector differences by the shift value defined in Table 2.
[0285] Reference Fig. 9 , the motion vector of the current block can be reconstructed based on the rounded motion vector prediction value and the motion vector difference (S930).
[0286] Of course, when the current block is encoded in the affine mode, the control point vector of the current block can be reconstructed based on the rounded control point vector prediction value and the control point vector difference.
[0287] Alternatively, rounding may be performed only on the finally reconstructed motion vector based on the resolution determined in S900. In this case, at least one of the rounding of the motion vector prediction value in S910 and the rounding of the motion vector difference in S920 may be omitted.
[0288] Alternatively, the adaptive motion vector resolution may be limited to be applied only to inter-frame modes (e.g., AMVP mode) in which motion vector differences are signaled, and may be set to apply inter-frame modes (e.g., merge mode) in which motion vector differences are not signaled.
[0289] Fig.10 An inter-frame prediction method based on interpolation is shown as an embodiment to which the present invention is applied.
[0290] Reference Fig.10 , the interpolation filter of the current block can be determined (S1000).
[0291] The encoding / decoding device may define one or more interpolation filter groups. When multiple interpolation filter groups are defined, any one of the multiple interpolation filter groups may be selectively used in consideration of block attributes. Here, the block attributes may include the size, shape, and inter-frame mode of the current block or sub-block.
[0292] For example, if the inter-frame mode of the current block is the affine mode and the current block or subblock is 4×4, the first interpolation filter group shown in Table 3 below may be selected, otherwise, the second interpolation filter group shown in Table 4 below may be selected.
[0293] [Table 3]
[0294]
[0295] [Table 4]
[0296]
[0297]
[0298] The first and second interpolation filter groups define interpolation filters for each fractional pixel position. In this case, the fractional pixel positions are based on a resolution or precision of 1 / 16 pixel.
[0299] The fractional pixel position to be interpolated can be specified by the motion vector of the current block. Here, the motion vector can be reconstructed by the above-mentioned rounding process. However, as shown in Table 4, the second interpolation filter group can define two interpolation filters for a resolution of 1 / 2 pixel. When the specified fractional pixel position to be interpolated is 8, one of the two interpolation filters can be selected according to the variable hpelIfIdx. The variable hpelIfIdx can be derived based on the above-mentioned shift value. For example, if the shift value is 3 (i.e., when the resolution of the motion vector is determined to be 1 / 2 pixel), the variable hpelIfIdx can be derived as 1, otherwise, the variable hpelIfIdx can be derived as 0.
[0300] Reference Fig.10 , a reference integer pixel position may be determined (S1010).
[0301] The reference integer pixel may belong to a reference picture of the current block. The reference picture may be a picture in a different time zone from the current block or a current picture to which the current block belongs.
[0302] The reference integer pixel position is (xInti, yInti), and xInti and yInti may be determined as in Equation 9 below.
[0303] [Formula 9]
[0304] xInti=(xIntL+i-3), i=0…7
[0305] yInti=(yIntL+i-3), i=0...7
[0306] In Equation 9, (xIntL, yIntL) may represent a reference integer pixel position for specifying the position of a fractional pixel to be interpolated (hereinafter referred to as an interpolation target pixel).
[0307] The reference integer pixel position may be determined according to whether the sub-picture is regarded as one picture during encoding / decoding. To this end, a flag indicating whether the sub-picture is regarded as one picture may be used. When the flag is a first value, the sub-picture may be regarded as one picture, otherwise, the sub-picture may not be regarded as one picture. The flag may be signaled for each sub-picture constituting one picture.
[0308] For example, when the flag is the first value, in the xInti calculated in Formula 9, the xInti outside the left boundary of the current sub-image can be replaced with the x coordinate of the left boundary of the sub-image, and the xInti outside the right boundary of the current sub-image can be replaced with the x coordinate of the right boundary of the sub-image. In addition, in the yInti calculated in Formula 9, the yInti outside the upper boundary of the current sub-image can be replaced with the y coordinate of the upper boundary of the sub-image, and the yInti outside the lower boundary of the current sub-image can be replaced with the y coordinate of the lower boundary of the sub-image.
[0309] A picture may be divided into one or more sub-pictures. A sub-picture may include one or more slices, or may include one or more tiles. It may be restricted so that a slice does not span multiple sub-pictures. Alternatively, it may be restricted so that a tile does not span multiple sub-pictures.
[0310] To divide into sub-pictures, a picture can be divided into k grid intervals in the vertical and horizontal directions. Each grid can be assigned an index. Grouping can be performed on each grid with the same index. A group of grids with the same index can be defined as a sub-picture.
[0311] Reference Fig.10 , the predicted pixels of the current block can be obtained by applying an interpolation filter to the determined reference integer pixels (S1020).
[0312] Specifically, when the interpolation target pixel position is a position shifted from the reference integer pixel only in the x-axis direction, a predicted pixel may be derived as the interpolation target pixel by applying an interpolation filter to eight reference integer pixels located on the same horizontal line.
[0313] Alternatively, when the interpolation target pixel position is a position shifted from the reference integer pixel only in the y-axis direction, a predicted pixel may be derived as the interpolation target pixel by applying an interpolation filter to eight reference integer pixels located on the same vertical line.
[0314] Meanwhile, when the interpolation target pixel position is a position shifted from the reference integer pixel in the x-axis and y-axis directions, an interpolation filter may be applied to eight reference integer pixels located on a horizontal line to calculate a reference fractional pixel. The reference fractional pixel may be calculated for each of eight consecutive horizontal pixel lines. A predicted pixel may be derived by applying an interpolation filter to eight reference fractional pixels located on a vertical line.
[0315] Alternatively, when the interpolation target pixel position is a position shifted from the reference integer pixel in the x-axis and y-axis directions, an interpolation filter may be applied to eight reference integer pixels located on a vertical line to calculate a reference fractional pixel. The reference fractional pixel may be calculated for each of eight consecutive vertical pixel lines. The predicted pixel may be derived by applying the interpolation filter to eight reference fractional pixels located on a horizontal line.
[0316] However, the 8-tap interpolation filter mentioned above is only an example, and a k-tap interpolation filter can be used, and k can be a natural number of 2, 3, 4, 5, 6, 7 or more. In this case, the above interpolation method can be applied in the same / similar manner. In addition, although the above example describes the case where multiple interpolation filters for the resolution of a specific pixel (1 / 2 pixel) are defined, multiple interpolation filters can also be applied to one or more of the resolutions of fractional pixels obtained based on a resolution of a predetermined accuracy.
[0317] For clarity of explanation, the exemplary method of the present disclosure is expressed as a series of operations, but this is not intended to limit the order in which the steps are performed, and if necessary, each step can be performed simultaneously or in a different order. To implement the method according to the present disclosure, the exemplary steps may include additional steps, may include other steps and exclude some steps, or may include additional other steps excluding some steps.
[0318] The various embodiments of the present disclosure are not intended to list all possible combinations but are intended to describe representative aspects of the present disclosure, and the contents described in the various embodiments may be applied independently or may be applied in combination of two or more.
[0319] In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, it may be implemented by one or more ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), general purpose processors, controllers, microcontrollers, microprocessors, and the like.
[0320] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that enable operations according to the methods of various embodiments to be performed on a device or computer, as well as non-transitory computer-readable media that store such software or instructions, etc. and are executable on a device or computer.
[0321] Industrial Applicability
[0322] The present invention can be used for encoding / decoding video signals.
Claims
1. A method for decoding an image performed by a decoding device, the method include: determining a flag indicating whether a resolution of a motion vector is adaptive; determining the resolution of the motion vector for the current block based on the indication of the flag; Obtaining a motion vector prediction value corresponding to the resolution; adjusting the motion vector difference of the current block based on the resolution; deriving a motion vector of the current block based on the derived motion vector prediction value and the adjusted motion vector difference; as well as predicting the current block based on the motion vector of the current block, wherein, when the flag indicates that the resolution of the motion vector is adaptive, the resolution is determined to be a resolution candidate specified by a predetermined index from among a plurality of resolution candidates, and The multiple resolution candidates are defined differently based on whether the reference picture of the current block is the current picture.
2. The method according to claim 1, in, The predicting the current block comprises: Determining an interpolation filter for the current block; determining a location of a reference integer pixel; and Predicted pixels of the current block are derived by applying an interpolation filter to the reference integer pixels.
3. The method according to claim 2, in, When the fractional pixel to be interpolated corresponds to a 1 / 2 pixel, one of a plurality of interpolation filters predefined in the decoding apparatus is selectively used.
4. The method according to claim 3, in, The interpolation filter of the current block is determined as one of the plurality of interpolation filters in consideration of the determined resolution.
5. The method according to claim 2, in, The position of the reference integer pixel is determined in consideration of the position of the sub-picture.
6. A method for encoding an image, include: Get the motion vector of the current block; predicting the current block based on the motion vector of the current block; as well as determining a resolution of the motion vector of the current block; Obtaining a motion vector prediction value corresponding to the resolution; adjusting the motion vector difference of the current block based on the resolution; as well as encoding the adjusted motion vector difference into the bitstream, and wherein a flag is determined, the flag indicating whether the resolution of the motion vector is adaptive, wherein, when the flag is determined that the resolution of the motion vector is adaptive, the resolution is determined from among a plurality of resolution candidates, and an index indicating the resolution from among the plurality of resolution candidates is encoded into the bitstream, and The multiple resolution candidates are defined differently based on whether the reference picture of the current block is the current picture.
7. A method for transmitting a bit stream of an image, wherein the bit stream is generated by encoding an image, wherein the method include: Get the motion vector of the current block; predicting the current block based on the motion vector of the current block; as well as determining a resolution of the motion vector of the current block; Obtaining a motion vector prediction value corresponding to the resolution; adjusting the motion vector difference of the current block based on the resolution; as well as encoding the adjusted motion vector difference into the bitstream, and wherein a flag is determined, the flag indicating whether the resolution of the motion vector is adaptive, wherein, when the flag is determined that the resolution of the motion vector is adaptive, the resolution is determined from among a plurality of resolution candidates, and an index indicating the resolution from among the plurality of resolution candidates is encoded into the bitstream, and The multiple resolution candidates are defined differently based on whether the reference picture of the current block is the current picture.
8. A method for decoding an image performed by a decoding device, the method include: determining a merge candidate list for the current block, the merge candidate list comprising a combined merge candidate obtained by combining motion vectors of a first merge candidate and a second merge candidate; determining a merge candidate indicating a reference picture and a motion vector from the merge candidate list; as well as predicting the current block based on the merge candidate, and Wherein, when the number of merging candidates in the merging candidate list is less than a predetermined number, constructing the combined merging candidate, wherein, when neither the first merge candidate nor the second merge candidate includes an L0 motion vector and both include an L1 motion vector, the combined merge candidate is determined to be an L1 directional prediction value, and When neither the first merge candidate nor the second merge candidate includes an L1 motion vector and both include an L0 motion vector, the combined merge candidate is determined to be an L0 direction prediction value.
9. The method according to claim 8, in, When both the first merge candidate and the second merge candidate include an L0 motion vector, determining the L0 motion vector of the combined merge candidate based on an average of the L0 motion vectors of the first merge candidate and the second merge candidate, and When both the first merge candidate and the second merge candidate include an L1 motion vector, the L1 motion vector of the combined merge candidate is determined based on an average of the L1 motion vectors of the first merge candidate and the second merge candidate.
10. The method according to claim 8, in, when the first merge candidate includes an L0 motion vector and the second merge candidate does not include an L0 motion vector, determining an L0 motion vector of the combined merge candidate based on the L0 motion vector of the first merge candidate, when the first merge candidate does not include the L0 motion vector and the second merge candidate includes the L0 motion vector, determining the L0 motion vector of the combined merge candidate based on the L0 motion vector of the second merge candidate, When the first merge candidate includes the L1 motion vector and the second merge candidate does not include the L1 motion vector, determining the L1 motion vector of the combined merge candidate based on the L1 motion vector of the first merge candidate, and When the first merge candidate does not include the L1 motion vector and the second merge candidate includes the L1 motion vector, the L1 motion vector of the combined merge candidate is determined based on the L1 motion vector of the second merge candidate.
11. The method according to claim 8, in, When neither the first merge candidate nor the second merge candidate includes an L0 motion vector, the L0 motion vector of the combined merge candidate is determined to be zero, and When neither the first merge candidate nor the second merge candidate includes an L1 motion vector, the L1 motion vector of the combined merge candidate is determined to be zero.
12. The method according to claim 8, in, When both the first merge candidate and the second merge candidate include an L0 motion vector, the L0 reference picture of the combined merge candidate is determined to be the L0 reference picture of the merge candidate having a smaller index among the first merge candidate and the second merge candidate, and When both the first merge candidate and the second merge candidate include an L1 motion vector, the L1 reference picture of the combined merge candidate is determined to be the L1 reference picture of the merge candidate having a smaller index between the first merge candidate and the second merge candidate.
13. The method according to claim 8, in, When the first merge candidate includes an L0 motion vector and the second merge candidate does not include an L0 motion vector, the L0 reference picture of the combined merge candidate is determined to be the L0 reference picture of the first merge candidate, When the first merge candidate does not include the L0 motion vector and the second merge candidate includes the L0 motion vector, the L0 reference picture of the combined merge candidate is determined as the L0 reference picture of the second merge candidate, When the first merge candidate includes an L1 motion vector and the second merge candidate does not include an L1 motion vector, the L1 reference picture of the combined merge candidate is determined to be the L1 reference picture of the first merge candidate, and When the first merge candidate does not include the L1 motion vector and the second merge candidate includes the L1 motion vector, the L1 reference picture of the combined merge candidate is determined as the L1 reference picture of the second merge candidate.
14. A method for encoding an image performed by an encoding device, the method include: determining a merge candidate list for the current block, the merge candidate list comprising a combined merge candidate obtained by combining motion vectors of a first merge candidate and a second merge candidate; determining a merge candidate indicating a reference picture and a motion vector from the merge candidate list; as well as predicting the current block based on the merge candidate, and Wherein, when the number of merging candidates in the merging candidate list is less than a predetermined number, constructing the combined merging candidate, wherein, when neither the first merge candidate nor the second merge candidate includes an L0 motion vector and both include an L1 motion vector, the combined merge candidate is determined to be an L1 directional prediction value, and When neither the first merge candidate nor the second merge candidate includes an L1 motion vector and both include an L0 motion vector, the combined merge candidate is determined to be an L0 direction prediction value.
15. An image transmission method for transmitting a bit stream, wherein the bit stream is generated by encoding an image based on an image encoding method, wherein the image encoding method include: determining a merge candidate list for the current block, the merge candidate list comprising a combined merge candidate obtained by combining motion vectors of a first merge candidate and a second merge candidate; determining a merge candidate indicating a reference picture and a motion vector from the merge candidate list; as well as predicting the current block based on the merge candidate, and Wherein, when the number of merging candidates in the merging candidate list is less than a predetermined number, constructing the group merging candidate, wherein, when neither the first merge candidate nor the second merge candidate includes an L0 motion vector and both include an L1 motion vector, the combined merge candidate is determined to be an L1 directional prediction value, and When neither the first merge candidate nor the second merge candidate includes an L1 motion vector and both include an L0 motion vector, the combined merge candidate is determined to be an L0 direction prediction value.
