Inter prediction method and apparatus based on merge mode

Abstract

To provide an inter prediction method and apparatus which improves the accuracy of the motion information by using not only a spatial / temporal merge candidate but also a combination merge candidate.SOLUTION: The method comprises: constructing a merge candidate list of a current block; deriving motion information of the current block from the merge candidate list; and performing inter prediction of the current block using the motion information. The merge candidate list includes at least one of a spatial merge candidate, a temporal merge candidate, and a combination merge candidate, and the combination merge candidate is derived by combining n merge candidates belonging to the merge candidate list.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an inter prediction method and apparatus. [Background technology]

[0002] In recent years, the demand for high-resolution, high-quality images, such as high definition (HD) images and ultra high definition (UHD) images, has been increasing in various application fields, and as a result, highly efficient image compression techniques have been studied.

[0003] There are various image compression techniques, such as an inter-prediction technique that predicts pixel values ​​contained in a current picture from pictures before or after the current picture, an intra-prediction technique that predicts pixel values ​​contained in a current picture using pixel information in the current picture, and an entropy coding technique that assigns short codes to values ​​that occur frequently and long codes to values ​​that occur less frequently. Using such image compression techniques, image data can be effectively compressed and transmitted or stored. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide an inter prediction method and apparatus.

[0005] The present invention aims to provide a method and apparatus for constructing a merge candidate list.

[0006] An object of the present invention is to provide a method and apparatus for motion compensation on a sub-block basis.

[0007] The present invention aims to provide a method and apparatus for determining affine candidates.

[0008] An object of the present invention is to provide an inter prediction method and device in accordance with the 360 ​​video projection format. [Means for solving the problem]

[0009] The video encoding / decoding method and apparatus according to the present invention may also be adapted to construct a merge candidate list for a current block, derive motion information of the current block from the merge candidate list, and perform inter prediction of the current block using the motion information.

[0010] In the video encoding / decoding method and apparatus according to the present invention, the merging candidate list may include at least one of spatial merging candidates, temporal merging candidates, or combined merging candidates.

[0011] In the video encoding / decoding method and apparatus according to the present invention, the combined merging candidate may be derived by combining n merging candidates belonging to a merging candidate list.

[0012] In the video encoding / decoding method and apparatus according to the present invention, the n merging candidates may be merging candidates corresponding to indexes from 0 to (n-1) in the merging candidate list.

[0013] In the video encoding / decoding method and apparatus according to the present invention, an index of the combined merging candidate may be greater than an index of the temporal merging candidate.

[0014] In the video encoding / decoding method and apparatus according to the present invention, the n merging candidates may include a first merging candidate and a second merging candidate, and the motion information of the combined merging candidate may be derived taking into account a prediction direction of the first merging candidate and a prediction direction of the second merging candidate.

[0015] In the video encoding / decoding method and apparatus according to the present invention, the motion information of the combined merging candidate may be derived by performing a weighted average of the motion information of the first merging candidate and the motion information of the second merging candidate.

[0016] In the video encoding / decoding method and apparatus according to the present invention, the weight of the weighted average may be any one of [1:1], [1:2], [1:3] and [2:3].

[0017] The video encoding / decoding method and apparatus according to the present invention may generate a candidate list for predicting motion information of a current block, derive a control point vector of the current block based on the candidate list and a candidate index, derive a motion vector of the current block based on the control point vector of the current block, and perform inter prediction on the current block using the motion vector.

[0018] In the video encoding / decoding device according to the present invention, the candidate list may include a plurality of affine candidates.

[0019] In the video encoding / decoding apparatus according to the present invention, the affine candidates may include at least one of spatial candidates, temporal candidates, or constructed candidates.

[0020] In the video encoding / decoding apparatus according to the present invention, the motion vector of the current block may be derived in units of sub-blocks of the current block.

[0021] In the video encoding / decoding apparatus according to the present invention, the spatial candidates may be determined by considering whether a boundary of the current block borders a coding tree block boundary (CTU boundary).

[0022] In the video encoding / decoding apparatus according to the present invention, the constructed candidates may be determined based on a combination of at least two of the control point vectors corresponding to each corner of the current block.

[0023] The video encoding / decoding method and apparatus according to the present invention may use data of a correlated region to obtain all or part of the pixels of the reference region when the reference region for inter prediction includes a boundary between reference pictures or a boundary between discontinuous surfaces. Effect of the Invention

[0024] According to the present invention, the accuracy of the motion information can be improved by using not only spatial / temporal merging candidates but also combined merging candidates.

[0025] According to the present invention, it is possible to improve the performance of image encoding / decoding by using inter prediction based on an affine model.

[0026] According to the present invention, it is possible to improve prediction accuracy by performing inter prediction on a sub-block basis.

[0027] According to the present invention, by determining efficient affine candidates, it is possible to improve the efficiency of encoding / decoding of inter prediction.

[0028] According to the present invention, the coding efficiency of inter prediction can be improved by setting a reference region taking correlation into consideration. [Brief description of the drawings]

[0029] [Figure 1] 1 is a block diagram of an image encoding device according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram of an image decoding device according to an embodiment of the present invention. [Diagram 3] FIG. 1 is an exemplary diagram showing the shape of a wood-based block. [Figure 4] 5A to 5C are exemplary diagrams showing various block shapes that can be obtained in the block division unit of the present invention. [Diagram 5] 4 is a diagram illustrating a block division process according to an embodiment of the present invention; [Figure 6] FIG. 1 is a diagram showing a method for performing inter prediction based on a block-based merge mode, which is an embodiment to which the present invention is applied. [Figure 7] FIG. 2 is a diagram showing an embodiment to which the present invention is applied, illustrating an inter prediction method based on an affine model. [Figure 8] FIG. 2 is a diagram showing an embodiment of the present invention, illustrating a method for deriving affine candidates from control point vectors of spatial / temporal neighboring blocks. [Figure 9] FIG. 2 illustrates an embodiment of the present invention, which illustrates a method for deriving constructed candidates based on a combination of motion vectors of spatial / temporal neighboring blocks. [Figure 10] FIG. 2 illustrates an embodiment of the present invention, showing a method for deriving motion information for sub-block based temporal candidates. [Figure 11] FIG. 1 is a diagram showing an embodiment to which the present invention is applied, illustrating an inter prediction method in an ERP projection format. [Figure 12] FIG. 1 is a diagram showing an embodiment to which the present invention is applied, illustrating an inter prediction method in a CMP projection format. [Figure 13] FIG. 1 is a diagram showing an embodiment to which the present invention is applied, illustrating an inter prediction method in a CMP projection format. [Figure 14] FIG. 1 is a diagram showing an embodiment to which the present invention is applied, illustrating an inter prediction method in a CMP projection format. [Figure 15] FIG. 1 is a diagram showing an embodiment to which the present invention is applied, illustrating an inter prediction method in a CMP projection format. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] The video encoding / decoding method and apparatus according to the present invention may also be adapted to construct a merge candidate list for a current block, derive motion information of the current block from the merge candidate list, and perform inter prediction of the current block using the motion information.

[0031] In the video encoding / decoding method and apparatus according to the present invention, the merging candidate list may include at least one of spatial merging candidates, temporal merging candidates, or combined merging candidates.

[0032] In the video encoding / decoding method and apparatus according to the present invention, the combined merging candidate may be derived by combining n merging candidates belonging to a merging candidate list.

[0033] In the video encoding / decoding method and apparatus according to the present invention, the n merging candidates may be merging candidates corresponding to indexes from 0 to (n-1) in the merging candidate list.

[0034] In the video encoding / decoding method and apparatus according to the present invention, an index of the combined merging candidate may be greater than an index of the temporal merging candidate.

[0035] In the video encoding / decoding method and apparatus according to the present invention, the n merging candidates may include a first merging candidate and a second merging candidate, and the motion information of the combined merging candidate may be derived taking into account a prediction direction of the first merging candidate and a prediction direction of the second merging candidate.

[0036] In the video encoding / decoding method and apparatus according to the present invention, the motion information of the combined merging candidate may be derived by performing a weighted average of the motion information of the first merging candidate and the motion information of the second merging candidate.

[0037] In the video encoding / decoding method and apparatus according to the present invention, the weight of the weighted average may be any one of [1:1], [1:2], [1:3] and [2:3].

[0038] The video encoding / decoding method and apparatus according to the present invention may generate a candidate list for predicting motion information of a current block, derive a control point vector of the current block based on the candidate list and a candidate index, derive a motion vector of the current block based on the control point vector of the current block, and perform inter prediction on the current block using the motion vector.

[0039] In the video encoding / decoding device according to the present invention, the candidate list may include a plurality of affine candidates.

[0040] In the video encoding / decoding apparatus according to the present invention, the affine candidates may include at least one of spatial candidates, temporal candidates, or constructed candidates.

[0041] In the video encoding / decoding apparatus according to the present invention, the motion vector of the current block may be derived in units of sub-blocks of the current block.

[0042] In the video encoding / decoding apparatus according to the present invention, the spatial candidates may be determined taking into consideration whether a boundary of the current block borders a CTU boundary.

[0043] In the video encoding / decoding apparatus according to the present invention, the constructed candidates may be determined based on a combination of at least two of the control point vectors corresponding to each corner of the current block.

[0044] The video encoding / decoding method and apparatus according to the present invention may use data of a correlated region to obtain all or part of the pixels of the reference region when the reference region for inter prediction includes a boundary between reference pictures or a boundary between discontinuous surfaces.

[0045] Although the present invention can be modified in various ways and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to the specific embodiments, and it should be understood that the present invention includes all modifications, equivalents, and alternatives included in the spirit and technical scope of the present invention.

[0046] Terms such as "first", "second", "A", "B" and the like can be used to describe various components, but the components are not limited by the terms. The terms are used only to distinguish one component from another. For example, a first component may be named a second component, and similarly, a second component may also be named a first component, without departing from the scope of the present invention. The term "and / or" includes a combination of multiple related listed items or any one of multiple related listed items.

[0047] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that the component may be directly coupled or connected to the other component, but there may be other components between them. In contrast, when a component is referred to as being "directly coupled" or "directly connected" to another component, it should be understood that there are no other components between them.

[0048] In addition, the terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural terms unless otherwise clearly indicated by the context. Terms such as "include" or "have" in this application are merely used to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0049] Unless otherwise specified, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Terms as commonly used and defined in dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as idealized or overly formal unless expressly defined in this application.

[0050] The image encoding device and the decoding device may be a user terminal such as a personal computer (PC), a notebook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a PlayStation Portable (PSP), a wireless communication terminal, a smartphone, a TV, a virtual reality device (VR), an augmented reality device (AR), a mixed reality device (MR), a head mounted display (HMD), or smart glasses, or may be a server terminal such as an application server and a service server. The image encoding device and the decoding device may include various devices having a communication device such as a communication modem for communicating with various devices or wired and wireless communication networks, a memory for storing various programs and data for encoding or decoding images and performing intra or inter prediction for encoding or decoding, and a processor for starting a program and performing calculations and control. In addition, the image coded in a bit stream by the image coding device may be transmitted in real time or non-real time to an image decoding device via a wired or wireless communication network (Network) such as the Internet, a short-range wireless communication network, a wireless LAN network, a WiBro network, or a mobile communication network, or via various communication interfaces such as a cable or a Universal Serial Bus (USB), and may be decoded in the image decoding device to restore the image and play it back.

[0051] Furthermore, an image encoded into a bitstream by an image encoding device may be transferred from the encoding device to a decoding device via a computer-readable recording medium.

[0052] The image coding device and the image decoding device described above may be separate devices, or may be fabricated as a single image coding / decoding device, if necessary. In this case, a part of the configuration of the image coding device may be substantially the same technical element as a part of the configuration of the image decoding device, and may include at least the same structure or may be realized to perform at least the same function.

[0053] Therefore, in the following detailed description of the technical elements and their operating principles, duplicated descriptions of corresponding technical elements will be omitted.

[0054] Since the image decoding apparatus corresponds to a computing device that applies the image coding method performed in the image coding apparatus for decoding, the following description will focus on the image coding apparatus.

[0055] The computing device may include a memory for storing a program or software module for implementing the image encoding method and / or the image decoding method, and a processor connected to the memory for executing the program. Note that the image encoding device may also be called an encoder, and the image decoding device may also be called a decoder.

[0056] Usually, an image is composed of a series of still images, and these still images are divided into GOP (Group of Pictures) units, and each still image may be called a picture. In this case, a picture may represent one of a frame and a field in a progressive signal or an interlaced signal, and an image may be represented as a "frame" when encoding / decoding is performed in units of frames, and as a "field" when encoding / decoding is performed in units of fields. In the present invention, a description is given assuming a progressive signal, but it is also applicable to an interlaced signal. As a higher-level concept, units such as GOP and sequence may exist, and each picture may be divided into predetermined regions such as slices, tiles, and blocks. In addition, one GOP may include units such as I-pictures, P-pictures, and B-pictures. An I-picture refers to a picture that is encoded / decoded by itself without using a reference picture, and a P-picture and a B-picture refer to a picture that is encoded / decoded by performing processes such as motion estimation and motion compensation using a reference picture. In general, a P-picture may use an I-picture and a P-picture as reference pictures, and a B-picture may use an I-picture and a P-picture as reference pictures, but the definitions are not limited thereto, and may be changed according to the encoding / decoding settings.

[0057] Here, a picture referred to in encoding / decoding is called a reference picture, and a block or pixel referred to is called a reference block or a reference pixel. The reference data may be not only pixel values ​​in the spatial domain, but also coefficient values ​​in the frequency domain, and various encoding / decoding information generated and determined during encoding / decoding. For example, this corresponds to information related to intra prediction or information related to motion in the prediction unit, information related to transformation in the transformation unit / inverse transformation unit, information related to quantization in the quantization unit / inverse quantization unit, information related to encoding / decoding in the encoding unit / decoding unit (context information), information related to filters in the in-loop filter unit, and the like.

[0058] The minimum unit of an image is a pixel, and the number of bits used to represent one pixel is called bit depth. Generally, the bit depth is 8 bits, and a higher bit depth can be supported depending on the encoding setting. The bit depth may be capable of supporting at least one bit depth depending on the color space. Also, depending on the color format of the image, it may be composed of at least one color space. Depending on the color format, it may be composed of one or more pictures having a certain size or one or more pictures having different sizes. For example, when YCbCr=4:2:0, it may be composed of one luminance component (Y in this example) and two chrominance components (Cb / Cr in this example), and in this case, the composition ratio of the chrominance component to the luminance component may be horizontal:vertical=1:2. As another example, when it is 4:4:4, it may have the same composition ratio of horizontal and vertical. When composed of one or more color spaces as in the above example, the picture may be divided into each color space.

[0059] In the present invention, a description is given based on a part of color spaces (Y in this example) of a part of color formats (YCbCr in this example), and the same or similar application (specific color space-dependent setting) may be made to other color spaces (Cb, Cr in this example) according to the color format. However, it is also possible to have partial differences (specific color space-independent setting) for each color space. That is, the dependent setting for each color space may mean having a setting proportional to the component ratio of each component (for example, determined according to 4:2:0, 4:2:2, 4:4:4, etc.) or a dependent setting, and the independent setting for each color space may mean having a setting only for the color space, regardless of or independently of the component ratio of each component. In the present invention, depending on the encoder / decoder, an independent setting may be made for some configurations, or a dependent setting may be made.

[0060] Configuration information or syntax elements required in the image encoding process may be determined at the unit level of video, sequence, picture, slice, tile, block, etc., and may be recorded in a bitstream in units of VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), slice header, tile header, block header, etc. and transmitted to a decoder. The decoder may parse the configuration information transmitted from the encoder in the same level unit and use it in the image decoding process. Also, related information in the form of SEI (Supplement Enhancement Information) or metadata may be transmitted to the bitstream and parsed for use. Each parameter set has a unique ID value, and a lower parameter set may have an ID value of a higher parameter set to be referenced. For example, a lower parameter set may refer to information of a higher parameter set having a matching ID value among one or more higher parameter sets. Among the various examples of units mentioned above, when any one unit contains one or more other units, the corresponding unit may be referred to as a higher-level unit, and the contained units may be referred to as lower-level units.

[0061] In the case of the setting information generated in the unit, it may include contents related to independent setting for each unit, or may include contents related to dependent setting for the previous, subsequent or higher unit. Here, dependent setting may be understood as representing the setting information of the unit by flag information that follows the setting of the previous, subsequent or higher unit (for example, a 1-bit flag, if 1, follows, and if 0, does not follow). The setting information in the present invention will be described with emphasis on an example of independent setting, but examples in which contents related to a dependent relationship are added to or replaced with the setting information of the previous, subsequent or higher unit of the current unit may also be included.

[0062] Fig. 1 is a block diagram of an image encoding device according to an embodiment of the present invention, and Fig. 2 is a block diagram of an image decoding device according to an embodiment of the present invention.

[0063] Referring to FIG. 1, the image encoding device may include a prediction unit, a subtraction unit, a transformation unit, a quantization unit, an inverse quantization unit, an inverse transformation unit, an addition unit, an in-loop filter unit, and a memory and / or an encoding unit. Some of the above components are not necessarily provided, and some or all of them may be selectively provided as necessary, and some further components not shown may be provided.

[0064] Referring to FIG. 2, the image decoding device may include a decoding unit, a prediction unit, an inverse quantization unit, an inverse transform unit, an adder unit, an in-loop filter unit, and / or a memory, and some of the above components are not necessarily provided, and some or all of the components may be selectively provided as necessary, and some further components not shown may be provided.

[0065] The image coding device and the image decoding device may be separate devices, or may be fabricated as one image coding / decoding device, if necessary. In this case, a part of the configuration of the image coding device may be substantially the same technical element as a part of the configuration of the image decoding device, and may include at least the same structure or perform at least the same function. Therefore, in the following detailed description of the technical elements and their operating principles, duplicated descriptions of corresponding technical elements will be omitted. The image decoding device can correspond to a computing device that applies the image coding method performed in the image coding device to decoding, so the following description will focus on the image coding device. The image coding device is sometimes called an encoder, and the image decoding device is sometimes called a decoder.

[0066] The prediction unit may include an intra prediction unit that performs intra prediction and an inter prediction unit that performs inter prediction. Intra prediction determines an intra prediction mode by configuring pixels of a block adjacent to a current block as reference pixels, and generates a predicted block using the intra prediction mode. Inter prediction determines motion information of a current block using one or more reference images, and generates a predicted block by performing motion compensation using the motion information. It is determined which of an intra prediction method or an inter prediction method is to be used for a current block (coding unit or prediction unit), and specific information (e.g., intra prediction mode, motion vector, reference image, etc.) associated with each prediction method is determined. At this time, the processing unit in which prediction is performed and the processing unit in which the prediction method and specific contents are determined may be determined according to the setting of encoding / decoding. For example, the prediction method, prediction mode, etc. are determined in a prediction unit (or a coding unit), and prediction is performed in a prediction block unit (or a coding unit, a transform unit).

[0067] The subtraction unit subtracts the predicted block from the current block to generate a residual block. That is, the subtraction unit calculates the difference between the pixel value of each pixel of the current block to be encoded and the predicted pixel value of each pixel of the predicted block generated by the prediction unit to generate the residual block, which is a block-shaped residual signal.

[0068] The transform unit can transform a signal belonging to the spatial domain into a signal belonging to the frequency domain, and a signal obtained through the transform process is called a transformed coefficient. For example, a residual block having a residual signal delivered from the subtraction unit is transformed to obtain a transform block having a transform coefficient, and the input signal is determined according to the coding setting and is not limited to a residual signal.

[0069] The transform unit may transform the residual block using a transform method such as a Hadamard transform, a discrete sine transform (DST Based-Transform), or a discrete cosine transform (DCT Based-Transform), but is not limited thereto, and may use various transform methods that are improvements or modifications of these.

[0070] The transformation may be horizontal / vertical, for example, the spatial domain pixel values ​​may be transformed to the frequency domain using a one-dimensional transformation in the horizontal direction and a one-dimensional transformation in the vertical direction using basis vectors in the transformation, for a total of two-dimensional transformation.

[0071] In addition, the conversion unit may pass the information necessary to generate the conversion block to the encoding unit to encode it, and may record the associated information in a bitstream and transmit it to a decoder, and the decoding unit of the decoder may parse the information and use it in the inverse conversion process.

[0072] The quantization unit may quantize an input signal, and a signal obtained through the quantization process is called a quantized coefficient. For example, a residual block having a residual transform coefficient delivered from the transform unit is quantized to obtain a quantized block having a quantized coefficient. The input signal is determined according to a coding setting, and is not limited to the residual transform coefficient.

[0073] The quantization unit may quantize the transformed residual block using a quantization method such as Dead Zone Uniform Threshold Quantization, Quantization Weighted Matrix, etc., but is not limited thereto, and may use various quantization methods that are improvements or modifications of these.

[0074] The quantization unit may pass the information necessary to generate the quantization block to the encoding unit to encode it, or may record the associated information in a bitstream and transmit it to a decoder, and the decoding unit of the decoder may parse the information and use it in the inverse quantization process.

[0075] In the above example, it is assumed that the residual block is transformed and quantized by the transform unit and the quantization unit, but the residual signal may be transformed to generate a residual block having a transform coefficient, and the quantization process of the residual block may not be performed, or the residual signal of the residual block may not be transformed into a transform coefficient, and only the quantization process may be performed, or neither the transform nor the quantization process may be performed, which can be determined according to the settings of the encoder.

[0076] The encoding unit may scan the quantization coefficients, transform coefficients, or residual signals of the generated residual block according to at least one scan order (e.g., zigzag scan, vertical scan, horizontal scan, etc.) to generate a quantization coefficient sequence, a transform coefficient sequence, or a signal sequence, and encode the quantization coefficient sequence, transform coefficient sequence, or signal sequence using at least one entropy coding method. In this case, information on the scan order may be determined according to the encoding setting (e.g., encoding mode, prediction mode, etc.), may be determined implicitly, or related information may be explicitly generated. For example, one of a plurality of scan orders may be selected according to the intra prediction mode. In this case, the scan pattern may be set to one of various patterns such as zig-zag, diagonal, and raster.

[0077] Also, encoded data including the encoding information transferred from each component may be generated and output as a bit stream, which may be realized by a multiplexer (MUX). In this case, the encoding method may be Exponential Golomb, Context Adaptive Variable Length Coding (CAVLC), Context Adaptive Binary Arithmetic Coding (CABAC), etc., but is not limited thereto, and various encoding methods improved or modified therefrom may be used.

[0078] When performing entropy coding (assumed to be CABAC in this example) on the residual block data and syntax elements (Syntax) such as information generated in the coding / decoding process, the entropy coding device may include a binarizer, a context modeler, and a binary arithmetic coder. In this case, the binary arithmetic coder may include a regular coding engine and a bypass coding engine. In this case, the regular coding engine may be a process performed in association with the context modeler, and the bypass coding engine may be a process performed independently of the context modeler.

[0079] Since the syntax element input to the entropy coding device may not be binary, if the syntax element is not binary, a binarization unit may binarize the syntax element and output a bin string consisting of 0 or 1. In this case, a bin indicates a bit consisting of 0 or 1, and may be coded by a binary arithmetic coding unit. In this case, either a regular coding unit or a bypass coding unit may be selected based on the occurrence probability of 0 and 1, which may be determined according to coding / decoding settings. If the syntax element is data with the same frequency of 0 and 1, the bypass coding unit may be used, and if not, the regular coding unit may be used and may be referred to when performing the next process by the regular coding unit by context modeling (or context information update).

[0080] In this case, the context is information on the occurrence probability of a bin, and the context modeling is a process of estimating the probability of a bin required for binary arithmetic coding by inputting a bin resulting from binarization. To estimate the probability, syntax element information of the bin, an index which is the position of the bin in the bin string, the probability of the bin included in the neighboring blocks of the block, etc. may be used, and at least one context table may be used for this purpose. For example, information for some flags may use multiple context tables depending on the combination of whether or not the flags of the neighboring blocks are used.

[0081] Various methods may be used when binarizing the syntax elements. For example, binarization may be divided into fixed length binarization and variable length binarization. In the case of variable length binarization, unary binarization, truncated unary binarization, truncated rice binarization, K-th order Exp-Golomb binarization, truncated binary binarization, etc. may be used. Also, signed or unsigned binarization may be performed according to the range of values ​​possessed by the syntax elements. The binarization process for the syntax elements generated in the present invention may be performed by not only the binarization methods mentioned in the above examples but also other binarization methods.

[0082] The inverse quantization unit and the inverse transform unit may be realized by performing the processes in the transform unit and the quantization unit inversely. For example, the inverse quantization unit may inverse quantize the quantized transform coefficients generated by the quantization unit, and the inverse transform unit may inverse transform the inverse quantized transform coefficients to generate a reconstructed residual block.

[0083] The adder adds the predicted block and the reconstructed residual block to reconstruct the current block. The reconstructed block may be stored in a memory and used as reference data (for the prediction unit, the filter unit, etc.).

[0084] The in-loop filter unit may include at least one post-processing filtering process, such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF). The deblocking filter may remove block artifacts occurring at boundaries between blocks from the restored image. The ALF may perform filtering based on a value obtained by comparing the restored image with an input image. In particular, filtering may be performed based on a value obtained by comparing the restored image with an input image after the blocks are filtered by the deblocking filter. Alternatively, filtering may be performed based on a value obtained by comparing the restored image with an input image after the blocks are filtered by the SAO.

[0085] The memory may store the restored block or picture. The restored block or picture stored in the memory may be provided to a prediction unit that performs intra prediction or inter prediction. In detail, a queue type storage space for a compressed bit stream in an encoder may be prepared and processed as a coded picture buffer (CPB), and a space for storing a decoded image in picture units may be prepared and processed as a decoded picture buffer (DPB). In the case of a CPB, a decoding unit may be stored according to a decoding order, a decoding operation may be emulated in the encoder, and a compressed bit stream may be stored in the emulation process. The bit stream output from the CPB is restored through a decoding process, the restored image is stored in the DPB, and the picture stored in the DPB is subsequently referred to in the image encoding and decoding processes.

[0086] The decoding unit may be realized by performing the process in the encoding unit in reverse, for example, by receiving a quantization coefficient sequence, a transform coefficient sequence, or a signal sequence from a bitstream and decoding the received sequence, or by parsing decoded data including decoding information and passing the decoded data to each component.

[0087] The image encoding device and image decoding device according to an embodiment of the present invention may be provided with a block division unit. The basic coding unit refers to a basic (or starting) unit for prediction, transformation, quantization, etc. in the image encoding / decoding process. In this case, the coding unit may be composed of one luminance coding block and two chrominance coding blocks according to a color format (YCbCr in this example), and the size of each block may be determined according to the color format. In the example described below, a block (luminance component in this example) will be used as a reference. In this case, the block is assumed to be a unit that can be obtained after each unit is determined, and a similar setting is assumed to be applicable to other types of blocks.

[0088] The block division unit may be set in relation to each component of the image encoding device and the decoding device, and the size and shape of the block may be determined by this process. In this case, the set block may be defined differently depending on the component, such as a prediction block in the case of a prediction unit, a transformation block in the case of a transformation unit, and a quantization block in the case of a quantization unit. Without being limited thereto, a block unit associated with another component may be further defined. The size and shape of the block may be defined by the horizontal and vertical lengths of the block.

[0089] In the block division unit, the blocks are represented by M×N, and the maximum and minimum values ​​of each block are obtained within the range. For example, the shape of the block can be a square, and if the maximum value of the block is 256×256 and the minimum value is 8×8, a block of size 2m×2m (in this example, m is an integer from 3 to 8, for example, 8×8, 16×16, 32×32, 64×64, 128×128, 256×256) or a block of size 2m×2m (in this example, m is an integer from 4 to 128) or a block of size m×m (in this example, m is an integer from 8 to 256) may be obtained. Alternatively, the shape of the block can be a square or a rectangular parallelepiped, and if it has the same range as the above example, a block of size 2m×2n (in this example, m and n are integers from 3 to 8. Assuming that the ratio of width to height is a maximum of 2:1, examples include 8×8, 8×16, 16×8, 16×16, 16×32, 32×16, 32×32, 32×64, 64×32, 64×64, 64×128, 128×64, 128×128, 128×256, 256×128, and 256×256. There may be no restriction on the ratio of width to height or there may be a maximum value for the ratio of width to height according to the encoding / decoding settings) may be obtained. Alternatively, a block of size 2m×2n (in this example, m and n are integers from 4 to 128) may be obtained. Alternatively, blocks of size m×n (in this example, m and n are integers between 8 and 256) may be obtained.

[0090] In the block division unit, the input block may obtain one or more blocks. For example, the input block may be output as it is, or may be divided into two or more sub-blocks and obtained. In this case, the number of sub-blocks may be 2, 3, 4, etc., and may be determined according to the division method (division type). The shape of the sub-block may be obtained based on the block before division (input block in this example), and may have various shapes according to the setting of block division.

[0091] Although the present invention will be described with emphasis on tree-based partitioning schemes, the present invention is in no way limited thereto.

[0092] The obtainable blocks may be determined according to the encoding / decoding settings (e.g., block type, partitioning method, partitioning settings, etc.). For example, the encoding block may be a block of size 2m×2n, the prediction block may be a block of size 2m×2n or m×n, and the transformation block may be a block of size 2m×2n. Based on the settings, information such as the size and range of the block (e.g., information related to exponents, multiples, etc.) may be generated.

[0093] The range (in this example, determined by a maximum value and a minimum value) may be determined according to the type of block. In addition, for some blocks, the range information of the block may be explicitly generated, and for some blocks, the range information of the block may be implicitly determined. For example, for coding and transformation blocks, the related information may be explicitly generated, and for prediction blocks, the related information may be implicitly processed.

[0094] If it is explicit, at least one piece of range information may be generated. For example, in the case of a coding block, the range information may be generated as information on a maximum value and a minimum value. Alternatively, the range information may be generated based on the difference between the maximum value and a previously set minimum value (e.g., 8) (e.g., generated based on the setting, such as information on the difference value of the exponent between the maximum value and the minimum value). Note that information on a plurality of ranges for the horizontal and vertical lengths of a rectangular block may be generated.

[0095] If it is implicit, the range information may be obtained based on the encoding / decoding settings (e.g., block type, partitioning method, partitioning settings, etc.). For example, in the case of a prediction block, maximum and minimum value information may be obtained as a candidate group (in this example, M×N and m / 2×n / 2) that can be obtained from a higher-order coding block (e.g., maximum size M×N and minimum size m×n of the coding block) by partitioning settings of the prediction block (e.g., quadtree partitioning + partitioning depth 0).

[0096] The size and shape of the initial (or starting) block of the block division unit may be determined from the upper unit. In the case of a coding block, the basic coding block may be the initial block, in the case of a prediction block, the coding block may be the initial block, and in the case of a transformation block, the coding block or the prediction block may be the initial block, which may be determined according to the setting of the coding / decoding. For example, when the coding mode is intra, the prediction block may be the upper unit of the transformation block, and when the coding mode is inter, the prediction block may be an independent unit with respect to the transformation block. The initial block is a starting unit of division and may be divided into blocks of a small size. When the optimal size and shape according to the division of each block is determined, the block may be determined as the initial block of the lower unit. For example, in the former case, it may be a coding block, and in the latter case (lower unit), it may be a prediction block or a transformation block. When the initial block of the lower unit is determined as in the above example, a division process may be performed to find a block of the optimal size and shape like the upper unit.

[0097] In short, the block division unit may divide a basic coding unit (or a maximum coding unit) into at least one coding unit (or a lower coding unit). Also, the coding unit may divide into at least one prediction unit, or may divide into at least one transform unit. The coding unit may divide into at least one coding block, and the coding block may divide into at least one prediction block, or may divide into at least one transform block. The prediction unit may divide into at least one prediction block, and the transform unit may divide into at least one transform block.

[0098] As in the above example, when a block having an optimal size and shape is found by the mode determination process, mode information (e.g., partition information, etc.) related thereto may be generated. The mode information may be recorded in a bitstream together with information generated in a component to which the block belongs (e.g., information related to prediction, information related to transformation, etc.) and transmitted to a decoder, or may be parsed in the decoder in the same level unit and used in an image decoding process.

[0099] In the examples described below, the division method is explained assuming that the initial blocks are square, but the same or similar application may be possible when the initial blocks are rectangular.

[0100] FIG. 3 is an exemplary diagram showing the shape of a wood-based block.

[0101] In the example, a indicates one 2N×2N that has not been split, b indicates two 2N×N through some split flags (horizontal split of binary tree in this example), c indicates two N×2N through some split flags (vertical split of binary tree in this example), and d indicates four N×N through some split flags (quad split of quad tree in this example). The shape of the block to be acquired may be determined according to the type of tree used for splitting. For example, when performing quad tree splitting, the available candidate blocks may be a and d. When performing binary tree splitting, the available candidate blocks may be a, b, and c. In the case of a quad tree, one split flag may be supported, and a may be acquired when the flag is "0", and d may be acquired when the flag is "1". In the case of a binary tree, multiple split flags may be supported, and one of these may be a flag indicating the presence or absence of splitting, and one of these may be a flag indicating the presence or absence of horizontal / vertical splitting. In the case of a quadtree, a basic tree-based partitioning scheme may be used, and a further tree partitioning scheme (in this example, a binary tree) may be included in the tree-based partitioning scheme. If a flag allowing further tree partitioning is implicitly or explicitly activated, multiple tree partitions may be performed. The tree-based partitioning may be a scheme that allows recursive partitioning. That is, a partitioned block may be set as an initial block again to perform tree-based partitioning, which may be determined according to partition settings such as a partition range and a partition allowance depth. This may be an example of a hierarchical partitioning scheme.

[0102] FIG. 4 is an illustrative diagram showing various block shapes that can be obtained in the block division unit of the present invention.

[0103] Referring to FIG. 4, blocks a to s may be obtained depending on the division setting and division method, and the shape of further blocks (not shown) may also be obtained.

[0104] According to example 1, asymmetrical division may be allowed for tree-based division. For example, in the case of a binary tree, blocks such as b and c may be obtainable, or blocks such as b to g may be obtainable. If a flag allowing asymmetrical division is explicitly or implicitly deactivated according to the encoding / decoding settings, the obtainable candidate block may be b or c, and if the flag allowing asymmetrical division is activated, the obtainable candidate block may be b, d, e (horizontal division in this example) or c, f, g (vertical division in this example).

[0105] In the above examples, the ratio of left:right or top:bottom lengths of the asymmetric split is assumed to be 1:3 or 3:1, but this is not limited thereto, and it is also possible to have candidates with other ratios (e.g., 1:2, 1:4, 2:3, 2:5, 3:5, etc.) according to the encoding settings.

[0106] The following shows various examples of split information that arise in a binary tree split (in this example, 1:1, 1:3, 3:1 candidate sets).

[0107] For example, a flag indicating the division shape may be generated in addition to a flag indicating the presence or absence of division and a flag indicating the division direction. In this case, the division shape may mean symmetric or asymmetric division. Among these, when asymmetric division is determined as the division shape, a flag indicating a division ratio may be generated, and an index may be assigned according to a candidate group that has already been set. If the candidate group can accommodate a division ratio of 1:3 or 3:1, the division ratio may be selected by a 1-bit flag.

[0108] Alternatively, a flag indicating the division ratio may be generated in addition to the flag indicating the presence or absence of division and the flag indicating the division direction. In this example, the candidate group for the division ratio may be configured to include candidates having a symmetric ratio of 1:1.

[0109] In the present invention (when a flag allowing asymmetric splitting is activated), it is assumed that the binary tree splitting has a configuration like the former example, and unless otherwise specified, the binary tree means a symmetric binary tree.

[0110] According to a second example, further tree division may be allowed for the tree-based division. For example, division into a ternary tree, a quad type tree, an octa tree, etc. is possible, and n divided blocks (3, 4, 8 in this example, n is an integer) may be obtained. In the case of a ternary tree, the corresponding blocks (in this example, when divided into a plurality of blocks) may be h to m, in the case of a quad type tree, the corresponding blocks may be n to p, and in the case of an octa tree, the corresponding block may be q. The compatibility of the tree-based division may be implicitly determined according to the encoding / decoding settings, or related information may be explicitly generated. It may be used alone according to the encoding / decoding settings, or may be used in combination with a binary tree, a quad type tree division, etc.

[0111] For example, in the case of a binary tree, blocks such as b and c can be obtained, and when a binary tree and a ternary tree are used together (in this example, it is assumed that the range of use of the binary tree and the range of use of the ternary tree partially overlap), blocks such as b, c, i, l can be obtained. If a flag that allows further division in addition to the existing tree is explicitly or implicitly deactivated according to the encoding / decoding settings, the obtainable candidate block may be b or c, and if it is activated, the obtainable candidate block may be b, i or b, h, i, j (in this example, horizontal division), or c, l or c, k, l, m (in this example, vertical division).

[0112] In the above examples, the ratio of the lengths of the left:middle:right or top:middle:bottom of the ternary tree division is assumed to be 2:1:1, 1:2:1 or 1:1:2, but this is not limited to this and other ratios are also possible depending on the encoding settings.

[0113] The following shows an example for the partition information that occurs in a ternary tree partition (in this example, the candidates are 1:2:1).

[0114] For example, a flag indicating the type of division may be generated in addition to a flag indicating the presence or absence of division and a flag indicating the direction of division. In this case, the type of division may indicate a binary tree or ternary tree division.

[0115] In the present invention, adaptive encoding / decoding settings may be applied depending on the division method.

[0116] According to one example, a partitioning scheme may be determined according to the type of block. For example, coding blocks and transform blocks may use quadtree partitioning, and prediction blocks may use quadtree and binary tree (or ternary tree, etc.) partitioning schemes.

[0117] According to one example, a division method may be determined according to the size of the block. For example, a quad-tree division may be used for a part of the range between the maximum and minimum values ​​of the block (e.g., a×b to c×d, where the latter is a larger size), and a binary tree (or a ternary tree, etc.) division may be used for a part of the range (e.g., e×f to g×h). In this case, range information according to the division method may be explicitly generated or implicitly determined, and overlapping ranges may also be adopted.

[0118] According to one example, a division method may be determined according to the shape of the block (or the block before division). For example, if the shape of the block is a square, quadtree and binary tree (or ternary tree, etc.) division may be possible. Alternatively, if the shape of the block is a rectangular parallelepiped, division based on a binary tree (or ternary tree, etc.) may be possible.

[0119] According to an example, the partitioning setting may be determined according to the type of block. For example, in the tree-based partitioning, the coding block and the prediction block may use quadtree partitioning, and the transformation block may use binary tree partitioning. Alternatively, the partitioning allowable depth may be set to m for the coding block, to n for the prediction block, and to o for the transformation block, where m, n, and o may be the same or different.

[0120] According to an example, the partitioning settings may be determined according to the size of the block. For example, quad-tree partitioning may be possible for some ranges of the block (e.g., a×b to c×d), binary-tree partitioning may be possible for some ranges (e.g., e×f to g×h. In this example, it is assumed that c×d is greater than g×h), and ternary-tree partitioning may be possible for some ranges (e.g., i×j to k×l. In this example, it is assumed that g×h is greater than or equal to k×l). In this case, the ranges may include all ranges between the maximum and minimum values ​​of the block, and the ranges may have non-overlapping settings or overlapping settings. For example, the minimum value of the ranges may be equal to the maximum value of the ranges, or the minimum value of the ranges may be smaller than the maximum value of the ranges. In the case of overlapping ranges, the partitioning scheme with the higher maximum value may have priority, or information regarding which partitioning scheme to use may be explicitly generated. That is, in a division method having a priority, whether or not to perform a division method having a lower priority may be determined according to the division result, or the type of division method to be used may be determined according to the selection information of the division method.

[0121] According to one example, the partitioning settings may be determined according to the shape of the block. For example, if the shape of the block is a square, a quadtree partitioning is possible. Alternatively, if the shape of the block is a rectangular parallelepiped, a binary tree or ternary tree partitioning is possible.

[0122] According to an example, the partitioning setting may be determined according to the encoding / decoding information (e.g., slice type, color component, encoding mode, etc.). For example, when the slice type is I, the quadtree (or binary tree, ternary tree) partitioning may be possible in a part of the range (e.g., a×b to c×d), when the slice type is P, in a part of the range (e.g., e×f to g×h), and when the slice type is B, in a part of the range (e.g., i×j to k×l). Also, when the slice type is I, the partitioning allowable depth of the quadtree (or binary tree, ternary tree partitioning) partitioning may be set to m, when the slice type is P, the partitioning allowable depth may be set to n, and when the slice type is B, the partitioning allowable depth may be set to o, where m, n, and o may be the same or different. When some slice types are used, they may have the same settings as other slices (e.g., P and B slices).

[0123] According to another example, when the color component is a luminance component, the allowable depth of quadtree (or binary tree, ternary tree) division may be set to m, and when the color component is a chrominance component, it may be set to n, where m and n may be the same or different. Note that the range of quadtree (or binary tree, ternary tree) division when the color component is a luminance component (e.g., a×b to c×d) may be the same as the range of quadtree (or binary tree, ternary tree) division when the color component is a chrominance component (e.g., e×f to g×h).

[0124] According to another example, when the coding mode is intra, the allowable depth of quadtree (or binary tree, ternary tree) division may be m, and when the coding mode is inter, it may be n (in this example, it is assumed that n is greater than m), where m and n may or may not be the same. Also, the range of quadtree (or binary tree, ternary tree) division when the coding mode is intra and the range of quadtree (or binary tree, ternary tree) division when the coding mode is inter may or may not be the same.

[0125] In the above example, information regarding whether or not it is possible to configure an adaptive division candidate group according to the encoding / decoding information may be explicitly generated, or may be implicitly determined.

[0126] The above example has been used to describe a case where the partitioning method and partitioning settings are determined according to the encoding / decoding settings. The above example shows some cases associated with each element, and modifications to other cases are also possible. The partitioning method and partitioning settings may also be determined according to a combination of multiple elements. For example, the partitioning method and partitioning settings may be determined according to the type, size, shape, encoding / decoding information, etc. of the block.

[0127] In addition, the elements involved in the division method, settings, etc. in the above examples may be implicitly determined, or information may be explicitly generated to determine whether adaptive cases such as the above examples are acceptable.

[0128] Among the division settings, the division depth means the number of times the initial block is spatially divided based on the initial block (in this example, the division depth of the initial block is 0), and as the division depth increases, the block can be divided into smaller blocks. The depth-related settings may be different depending on the division method. For example, among methods of performing division based on trees, the division depth of a binary tree and the division depth of a ternary tree may use a common depth, and the division depth of a quad tree and the division depth of a binary tree may use different depths, and individual depths may be used according to the type of tree.

[0129] In the above example, when using individual split depths depending on the type of tree, the split depth may be set to 0 at the split start position of the tree (in this example, the block before splitting). The split depth may be calculated centered on the position where splitting starts, without being based on the split range of each tree (in this example, the maximum value).

[0130] 5 illustrates a block division process according to an embodiment of the present invention, specifically, starting from a basic coding block, examples of block sizes and shapes that can be obtained according to one or more division methods are illustrated.

[0131] In the figure, the thick solid lines represent basic coding blocks, the thick dotted lines represent quadtree partition boundaries, the double solid lines represent symmetric binary tree partition boundaries, the solid lines represent ternary tree partition boundaries, and the thin dotted lines represent asymmetric binary tree partition boundaries. All lines except the thick solid lines represent boundaries partitioned according to each partitioning method. The partitioning settings (e.g., partitioning type, partitioning information, configuration order of partitioning information, etc.) described below are not limited to the present example, and various modified examples are possible.

[0132] For ease of explanation, it is assumed that the upper left, upper right, lower left, and lower right blocks (N×N, 64×64) have individual block division settings based on a basic coding block (2N×2N, 128×128). First, in an initial block, four sub-blocks are obtained by one division operation (division depth 0→1, i.e., the division depth increases by 1), and according to the division settings for the quadtree, the maximum coding block is 128×128, the minimum coding block is 8×8, and the maximum division depth is 4, which is a setting commonly applied to each block.

[0133] (No. 1. Top left block. A0~A6) This example is compatible with single-tree division (in this example, a quadtree), and the size and shape of the obtainable block may be determined by one block division setting such as the maximum coding block, the minimum coding block, and the division depth. In this example, when only one block can be obtained as a result of division (division into two horizontally and vertically), the division information required for one division operation (based on the block 4M×4N before division; division depth increases by 1) is a flag indicating whether or not division is performed (in this example, 0 indicates no division; 1 indicates division), and the obtainable candidates are 4M×4N and 2M×2N.

[0134] (No. 2. Top right block. A7~A11)

[0135] This example is a case where multiple tree division (in this example, quad tree, binary tree) is supported, and the size and shape of the block that can be obtained are determined by multiple block division settings. In this example, in the case of a binary tree, it is assumed that the maximum coding block is 64 x 64, the minimum coding block has one length of 4, and the maximum division depth is 4.

[0136] In this example, two or more blocks can be obtained as a result of the split (in this example, two or four), so the split information required for one splitting operation (the quadtree split depth increases by 1) is a flag indicating whether or not a split has occurred, a flag indicating the type of split, a flag indicating the split shape, and a flag indicating the split direction, and the possible candidates are 4M×4N, 4M×2N, 2M×4N, 4M×N / 4M×3N, 4M×3N / 4M×N, M×4N / 3M×4N, 3M×4N / M×4N.

[0137] If the quadtree and binary tree division ranges overlap (i.e., the range where both quadtree division and binary tree division are possible at the current stage) and the current block (state before division) is a block obtained by quadtree division (a block obtained by quadtree division from a parent block <if the division depth is 1 less than the current one>), the division information is distinguished and configured as follows: In other words, if a corresponding block according to each division setting can be obtained by multiple division methods, it is possible to generate division information by classifying it according to the following process.

[0138] (1) When quadtree division and binary tree division overlap

[0139] [Table 1]

[0140] In the above table, a means a flag indicating whether or not quad-tree division is performed, and if it is 1, quad-tree division (QT) is performed. If the flag is 0, a flag b indicating whether or not binary tree division is performed is checked. If b is 0, no further division is performed in the block (No Split), and if it is 1, binary tree division is performed.

[0141] c is a flag indicating the division direction, and if it is 0, it means horizontal division (hor) and if it is 1, it means vertical division (ver), and d is a flag indicating the division shape, and if it is 0, it means symmetric division (SBT; Symmetric Binary Tree) and if it is 1, it means asymmetric division (ABT; Asymmetric Binary Tree). Only when d is 1, information on the detailed division ratio in asymmetric division (1 / 4 or 3 / 4) is confirmed, and if it is 0, in the left / right or top / bottom blocks, the left and top blocks have a ratio of 1 / 4 and the right and bottom blocks have a ratio of 3 / 4, and if it is 1, they have the opposite ratios.

[0142] (2) When only binary tree partitioning is possible In the above table, the division information can be expressed by flags b to e excluding a.

[0143] In Figure 5, in the case of block A7, quadtree partitioning was possible for blocks A7 to A11 before partitioning (i.e., quadtree partitioning was possible, but binary tree partitioning rather than quadtree partitioning was performed), so this corresponds to the case in which partition information is generated in (1).

[0144] In contrast, in the case of A8 to A11, if the quad-tree division has not been performed on the blocks A8 to A11 before division and the binary tree division has been performed (i.e., no further division of the blocks will be performed on the blocks A8 to A11),<A8~A11> Since quadtree partitioning is not possible in (2), this corresponds to the case where partition information is generated in (2).

[0145] (No. 3. Lower left block. A12~A15) This example is a case where multiple tree division methods (in this example, quad tree, binary tree, and ternary tree) are supported, and the size and shape of blocks that can be obtained are determined by multiple block division settings. In this example, in the case of binary tree / ternary tree, it is assumed that the maximum coding block is 64 x 64, the minimum coding block has one length of 4, and the maximum division depth is 4.

[0146] In this example, two or more blocks can be obtained as a result of splitting (in this example, 2, 3, or 4), so the splitting information required for one splitting operation is a flag indicating whether or not splitting has occurred, a flag indicating the type of splitting, and a flag indicating the splitting direction, and the obtainable candidates are 4M×4N, 4M×2N, 2M×4N, 4M×N / 4M×2N / 4M×N, M×4N / 2M×4N / M×4N.

[0147] If the division ranges of the quadtree and binary / ternary tree overlap and the current block is a block obtained by quadtree division, the division information is configured to distinguish between the following cases.

[0148] (1) When quadtree division and binary / ternary tree division overlap

[0149] [Table 2]

[0150] In the above table, a is a flag indicating whether or not quadtree division is performed, and if it is 1, quadtree division is performed. If the flag is 0, a flag b indicating whether or not binary or ternary tree division is performed is checked. If b is 0, no further division is performed in the block, and if it is 1, binary or ternary tree division is performed.

[0151] c is a flag indicating the split direction, where 0 means horizontal split and 1 means vertical split, and d is a flag indicating the split type, where 0 means binary tree split (BT) and 1 means ternary tree split (TT).

[0152] (2) When only binary / ternary tree division is possible In the above table, the division information can be expressed by flags b to d, excluding a.

[0153] In FIG. 5, in the case of blocks A12 and A15, quadtree partitioning is possible for blocks A12 to A15 before partitioning, so this corresponds to the case where partition information is generated in (1).

[0154] In contrast, A13 and A14 are cases where the blocks A13 and A14 before division have not been divided into a quadtree but have been divided into a ternary tree, and therefore correspond to the case where division information is generated in (2).

[0155] (No. 4. Lower left block. A16~A20) This example is a case where multiple tree division methods (in this example, quad tree, binary tree, and ternary tree) are supported, and the size and shape of blocks that can be obtained are determined by multiple block division settings. In this example, in the case of binary tree / ternary tree, it is assumed that the maximum coding block is 64 x 64, the minimum coding block has one length of 4, and the maximum division depth is 4.

[0156] In this example, the number of blocks that can be obtained as a result of splitting is two or more (in this example, 2, 3, or 4), so the splitting information required for one splitting operation is a flag indicating whether or not splitting has occurred, a flag indicating the type of split, a flag indicating the split shape, and a flag indicating the split direction, and the obtainable candidates are 4M×4N, 4M×2N, 2M×4N, 4M×N / 4M×3N, 4M×3N / 4M×N, M×4N / 3M×4N, 3M×4N / M×4N, 4M×N / 4M×2N / 4M×N, M×4N / 2M×4N / M×4N.

[0157] If the division ranges of the quadtree and binary / ternary tree overlap and the current block is a block obtained by quadtree division, the division information is configured to distinguish between the following cases.

[0158] (1) When quadtree division and binary / ternary tree division overlap

[0159] [Table 3]

[0160] In the above table, a means a flag indicating whether or not quadtree division is performed, and if it is 1, quadtree division is performed. If the flag is 0, b, which is a flag indicating whether or not binary tree division is performed, is checked. If b is 0, no further division is performed in the block, and if it is 1, binary or ternary tree division is performed.

[0161] c is a flag indicating the division direction, and if it is 0, it means horizontal division and if it is 1, it means vertical division, and d is a flag indicating the division type, and if it is 0, it means ternary tree division and if it is 1, it means binary tree division. When d is 1, e, which is a flag for the division shape, is checked, and if e is 0, symmetric division is performed and if it is 1, asymmetric division is performed. When e is 1, information regarding the fine division ratio in asymmetric division is checked, which is the same as the previous example.

[0162] (2) When only binary / ternary tree division is possible In the above table, the division information can be expressed by flags b to f, excluding a.

[0163] In FIG. 5, block A20 corresponds to the case where division information is generated in (1) since quadtree division is possible for blocks A16 to A19 before division.

[0164] In contrast, in the case of A16 to A19, quad-tree division has not been performed on the pre-division blocks A16 to A19, and binary-tree division has been performed, so this corresponds to the case in which division information is generated in (2).

[0165] Hereinafter, an inter prediction method based on a merge mode will be described. In the merge mode, motion information may be derived in units of one block. Here, the block may refer to a coding block determined by the above-mentioned block division. Also, the block may refer to a coding block of a leaf node that is not further divided into smaller coding blocks. In the merge mode, motion information may be derived in units of sub-blocks belonging to one block. For this purpose, one block may be further divided into a plurality of sub-blocks. For example, affine motion modeling may be performed based on the sub-block-based merge mode. The current block may selectively use either the block-based merge mode or the sub-block-based merge mode, and a predetermined flag may be used for this purpose. Here, the flag may be information indicating whether the sub-block-based merge mode is applied. The flag may be coded and signaled in the coding device based on a predetermined coding parameter, or may be derived from the decoding device. The coding parameter may refer to a size / shape of a block, a component type, a partition type, a partition depth, etc. For example, if the size of the current block is greater than or equal to N×M, the flag may be signaled, otherwise the flag may be derived as a predefined value, where N and M may be integers of 8, 16 or more. N and M may be the same or different from each other. The block-based merge mode will be described with reference to FIG. 6, and the sub-block-based merge mode will be described with reference to FIGS. 7 to 10.

[0166] FIG. 6 is a diagram showing a method of performing inter prediction based on a block-based merge mode, which is an embodiment to which the present invention is applied.

[0167] Referring to FIG. 6, a merge candidate list for the current block is constructed (S600).

[0168] The merge candidate list may include at least one of spatial or temporal merge candidates for the current block.

[0169] The motion information of the spatial merging candidate may be derived from the motion information of the spatial neighboring blocks of the current block. Here, the spatial neighboring blocks refer to blocks that belong to the same picture as the current block and are neighboring to the current block. The spatial neighboring blocks may include blocks neighboring at least one of the left side, top edge, top right edge, bottom left edge, or top left edge of the current block. The top left edge neighboring block may be used only if at least one of the left side, top edge, top right edge, and bottom left edge neighboring blocks is not available (usable).

[0170] The motion information of the temporal merge candidate may be derived from the motion information of the temporal neighboring block of the current block. The temporal neighboring block is defined as a block belonging to a picture different from the current block and located at the same position as the current block. Here, the block at the same position may mean at least one of a block (BR) adjacent to the lower right corner of the current block, a block (CTR) including a position of a central sample of the current block, or a block (TL) including a position of a top left sample of the current block. Alternatively, the block at the same position may mean a block including a position shifted by a predetermined disparity vector from the position of the top left sample of the current block. Here, the disparity vector may be determined based on any one of the motion vectors of the spatial neighboring blocks described above. Alternatively, the disparity vector may be determined based on a combination of at least two of the motion vectors of the spatial neighboring blocks described above. The combination may mean a calculation of a maximum value, a minimum value, a median value, a weighted average value, or the like. For example, the disparity vector may be set to the motion vector of the left neighboring block or the top neighboring block. Alternatively, the disparity vector may be set to the midpoint or average value between the motion vector of the left adjacent block and the motion vector of the bottom left adjacent block.

[0171] The motion vector and reference picture index of the temporal merge candidate may be derived to the motion vector and reference picture index of the temporal neighboring block mentioned above, respectively. Alternatively, the motion vector of the temporal merge candidate may be derived as the motion vector of the temporal neighboring block, and the reference picture index of the temporal merge candidate may be set to a default (e.g., 0) already promised to the decoding device, regardless of the temporal neighboring block.

[0172] The merge candidate list may further include a combined merge candidate, which may be derived by combining n merge candidates belonging to the already generated merge candidate list.

[0173] Here, n may be an integer of 2, 3, 4 or more. The number (n) of merging candidates to be combined may be a fixed value already agreed to by the encoding / decoding device, or may be coded and signaled in the encoding device. The signaling may be performed in units of at least one of a sequence, a picture, a slice, a tile, a subtile (brick), or a predetermined block. The number (n) of merging candidates to be combined may be variably determined based on the number of remaining merging candidates. Here, the number of remaining merging candidates means a difference between a maximum number of merging candidates that can be included in a merging candidate list and a current number of merging candidates belonging to the merging candidate list. The maximum number may be a number already agreed to by the encoding / decoding device, or may be coded and signaled in the encoding device. The current number means the number of merging candidates configured before adding a combined merging candidate. For example, when the number of remaining merging candidates is 1, two merging candidates may be used, and when the number of remaining merging candidates is greater than 1, three or more merging candidates may be used.

[0174] The positions of the n merging candidates may be positions that have already been determined in the merging candidate list. For example, an index (0 to (k-1)) may be assigned to each merging candidate belonging to the merging candidate list. Here, k may represent the total number of merging candidates belonging to the merging candidate list. In this case, the positions of the n merging candidates may correspond to index 0 to index (n-1) in the merging candidate list. Alternatively, the n merging candidates may be determined in consideration of the prediction direction of each merging candidate belonging to the merging candidate list. For example, only bidirectional merging candidates or only unidirectional merging candidates may be selectively used among the merging candidates belonging to the merging candidate list.

[0175] The combined merge candidates may be derived using both spatial and temporal merge candidates, or may be derived using only spatial or only temporal merge candidates. For example, the combined merge candidates may be restricted to be derived using only spatial merge candidates. In this case, the number of merge candidates to be combined may be restricted to be equal to or less than the number of spatial merge candidates that belong to the merge candidate list already generated.

[0176] The combined merge candidate may be added after the spatial / temporal merge candidate in the merge candidate list. That is, the index of the combined merge candidate may be greater than the index of the spatial / temporal merge candidate. Alternatively, the combined merge candidate may be added between the spatial merge candidate and the temporal merge candidate in the merge candidate list. That is, the index of the combined merge candidate may be greater than the index of the spatial merge candidate and less than the index of the temporal merge candidate. Alternatively, the position of the combined merge candidate may be variably determined in consideration of the prediction direction of the combined merge candidate. Depending on whether the prediction direction of the combined merge candidate is bidirectional prediction, the position of the combined merge candidate may be rearranged in the merge candidate list. For example, if the combined merge candidate is bidirectional prediction, an index smaller than the spatial or temporal merge candidate may be assigned, and if not, an index larger than the spatial or temporal merge candidate may be assigned.

[0177] For ease of explanation, we will describe below how to derive a combined merge candidate based on two merge candidates.

[0178] The motion information of the combined merge candidate may be derived by weighted averaging the motion information of the first merge candidate and the second merge candidate. Here, the weight of the weighted average may be, but is not limited to, [1:1], [1:2], [1:3], [2:3], etc. The weight may be predefined in the encoding / decoding device, or may be derived from the decoding device. In this case, the weight may be derived considering at least one of the distance between the current picture and the reference picture of the merge candidate or the prediction direction of the merge candidate. Alternatively, the motion information of the combined merge candidate may be derived by obtaining motion information in the L0 direction from the first merge candidate and motion information in the L1 direction from the second merge candidate, respectively, and combining them. The motion information of the combined merge candidate may be derived based on at least one of the above-mentioned derivation methods, and this may be performed considering the prediction direction of the merge candidate to be combined, as described later.

[0179] In this specification, the motion information may include at least one of a prediction direction flag, a reference picture index, or a motion vector. The motion information may be defined for L0 prediction and L1 prediction, respectively. Here, L0 prediction means prediction referring to a reference picture list L0, and L1 prediction means prediction referring to a reference picture list L1.

[0180] 1. If the first and second merging candidates are both unidirectional (CASE 1) When the first merge candidate is L0 prediction and the second merge candidate is L1 prediction, the reference picture index in the L0 direction of the combined merge candidate is derived as the reference picture index of the first merge candidate. The prediction direction flag in the L0 direction of the combined merge candidate is derived as 1. The motion vector in the L0 direction of the combined merge candidate is derived as the motion vector of the first merge candidate. The reference picture index in the L1 direction of the combined merge candidate is derived as the reference picture index of the second merge candidate. The prediction direction flag in the L1 direction of the combined merge candidate is derived as 1. The motion vector in the L1 direction of the combined merge candidate is derived as the motion vector of the second merge candidate.

[0181] (CASE 2) When the first merge candidate is L1 prediction and the second merge candidate is L0 prediction, the reference picture index in the L0 direction of the combined merge candidate is derived as the reference picture index of the second merge candidate. The flag of the prediction direction in the L0 direction of the combined merge candidate is derived as 1. The motion vector in the L0 direction of the combined merge candidate is derived as the motion vector of the second merge candidate. The reference picture index in the L1 direction of the combined merge candidate is derived as the reference picture index of the first merge candidate. The flag of the prediction direction in the L1 direction of the combined merge candidate is derived as 1. The motion vector in the L1 direction of the combined merge candidate is derived as the motion vector of the first merge candidate.

[0182] (CASE3) When the first merge candidate and the second merge candidate are L0 prediction, the reference picture index in the L0 direction of the combined merge candidate is derived as the reference picture index of either the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate is set as the reference picture index in the L0 direction of the combined merge candidate. The flag of the prediction direction in the L0 direction of the combined merge candidate is derived as 1. The motion vector in the L0 direction of the combined merge candidate is derived as the weighted average of the motion vector of the first merge candidate and the second merge candidate. The reference picture index in the L1 direction of the combined merge candidate is derived as -1, the flag of the prediction direction in the L1 direction is derived as 0, and the motion information in the L1 direction is derived as 0.

[0183] (CASE 4) When the first merge candidate and the second merge candidate are L1 prediction, the reference picture index in the L0 direction of the combined merge candidate is derived as -1, the flag of the prediction direction in the L0 direction is derived as 0, and the motion information in the L0 direction is derived as 0. The reference picture index in the L1 direction of the combined merge candidate is derived as the reference picture index of either the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate is set as the reference picture index in the L1 direction of the combined merge candidate. The flag of the prediction direction in the L1 direction of the combined merge candidate is derived as 1. The motion vector in the L1 direction of the combined merge candidate is derived as a weighted average of the motion vector of the first merge candidate and the second merge candidate.

[0184] 2. When the first and second merging candidates are both bidirectional predictions (CASE 5) The reference picture index in the L0 direction of the combined merge candidate is derived as the reference picture index of either the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate is set as the reference picture index of the combined merge candidate in the L0 direction. The flag of the prediction direction in the L0 direction of the combined merge candidate is derived as 1. The motion vector in the L0 direction of the combined merge candidate is derived as a weighted average of the motion vector of the first merge candidate and the second merge candidate. The reference picture index in the L1 direction of the combined merge candidate is derived as the reference picture index of either the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate is set as the reference picture index of the combined merge candidate in the L1 direction. The flag of the prediction direction in the L1 direction of the combined merge candidate is derived as 1. The L1 motion vector of the combined merge candidate is derived as a weighted average of the motion vector of the first merging candidate and the motion vector of the second merging candidate.

[0185] 3. The first merging candidate is bi-predictive and the second merging candidate is uni-predictive (CASE6) When the second merge candidate is an L0 prediction, the reference picture index in the L0 direction of the combined merge candidate is derived as the reference picture index of either the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate is set as the reference picture index in the L0 direction of the combined merge candidate. The flag of the prediction direction in the L0 direction of the combined merge candidate is derived as 1. The motion vector in the L0 direction of the combined merge candidate is derived as a weighted average of the motion vector of the first merge candidate and the second merge candidate. The reference picture index in the L1 direction of the combined merge candidate is derived as the reference picture index of the first merge candidate. The flag of the prediction direction in the L1 direction of the combined merge candidate is derived as 1. The motion vector in the L1 direction of the combined merge candidate is derived as the motion vector of the first merge candidate.

[0186] (CASE 7) When the second merge candidate is an L1 prediction, the reference picture index in the L0 direction of the combined merge candidate is derived as the reference picture index of the first merge candidate. The flag of the prediction direction in the L0 direction of the combined merge candidate is derived as 1. The motion vector in the L0 direction of the combined merge candidate is derived as the motion vector of the first merge candidate. The reference picture index in the L1 direction of the combined merge candidate is derived as the reference picture index of either the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate is set as the reference picture index in the L1 direction of the combined merge candidate. The flag of the prediction direction in the L1 direction of the combined merge candidate is derived as 1. The motion vector in the L1 direction of the combined merge candidate is derived as a weighted average of the motion vector of the first merge candidate and the second merge candidate.

[0187] Referring to FIG. 6, motion information of the current block is derived from the merge candidate list (S610).

[0188] Specifically, a merge index of the current block may be signaled. The merge index may identify one of a plurality of merge candidates belonging to the merge candidate list. That is, a merge candidate having an index equal to the merge index may be extracted, and motion information of the current block may be derived using motion information of the extracted merge candidate.

[0189] Referring to FIG. 6, inter prediction is performed on the current block using the derived motion information (S620).

[0190] Specifically, a reference picture may be selected from a reference picture list based on a reference picture index of the current block. A reference block in the reference picture may be identified based on a motion vector of the current block. Prediction pixels of the current block may be generated using reconstructed pixels of the identified reference block. The current block may be reconstructed by adding prediction pixels of the current block and residual pixels. Here, the residual pixels may be derived by performing at least one of inverse quantization and inverse transformation on residual coefficients signaled by a bitstream.

[0191] FIG. 7 is a diagram showing an inter prediction method based on an affine model, which is an embodiment to which the present invention is applied.

[0192] Referring to FIG. 7, a candidate list for predicting motion information of a current block is generated (S700).

[0193] The candidate list may include one or more candidates based on an affine model (hereinafter referred to as affine candidates). An affine candidate means a candidate with a control point vector. The control point vector means a motion vector of a control point for an affine model, and may be defined with respect to a corner position of a block (e.g., at least one position of the upper left corner, the upper right corner, the lower left corner, or the lower right corner).

[0194] The affine candidates may include at least one of spatial candidates, temporal candidates, and constructed candidates. Here, the spatial candidates may be derived from vectors of neighboring blocks that are spatially adjacent to the current block, and the temporal candidates may be derived from vectors of neighboring blocks that are temporally adjacent to the current block. Here, the neighboring blocks refer to blocks coded with an affine model. The vectors may refer to motion vectors or control point vectors.

[0195] The method of deriving spatial / temporal candidates based on the vectors of spatial / temporal neighboring blocks will be detailed with reference to FIG.

[0196] Alternatively, the constructed candidates may be derived based on a combination between motion vectors of spatial / temporal neighboring blocks of the current block, which will be described in detail with reference to FIG.

[0197] The plurality of affine candidates may be arranged in the candidate list based on a predetermined priority. For example, the plurality of affine candidates may be arranged in the candidate list in the order of spatial candidates, temporal candidates, and constructed candidates. Alternatively, the plurality of affine candidates may be arranged in the candidate list in the order of temporal candidates, spatial candidates, and constructed candidates. However, without being limited thereto, the temporal candidates may be arranged following the constructed candidates. Alternatively, some of the constructed candidates may be arranged before the spatial candidates, and the rest may be arranged after the spatial candidates.

[0198] The candidate list may further include a temporal candidate based on a sub-block, and the motion information of the temporal candidate may be derived at a collocated block corresponding to the current block, which will be described in detail with reference to FIG.

[0199] Based on the candidate list and the candidate index, a control point vector of the current block is derived (S710).

[0200] The candidate index means an index coded for deriving a control point vector of a current block. The candidate index may identify one of a plurality of affine candidates belonging to a candidate list. The control point vector of the current block may be derived using the control point vector of the affine candidate identified by the candidate index.

[0201] For example, assume that the type of the affine model of the current block is four parameters (i.e., it is determined that the current block uses two control point vectors). In this case, if the affine candidate identified by the candidate index has three control point vectors, only two control point vectors (e.g., the control point vector Idx=0,1) may be selected from the three control point vectors and set as the control point vector of the current block. Alternatively, the three control point vectors of the identified affine candidate may be set as the control point vector of the current block. In this case, the type of the affine model of the current block may be updated to six parameters.

[0202] Conversely, assume that the type of the affine model of the current block is six parameters (i.e., when it is determined that the current block uses three control point vectors). At this time, if the affine candidate specified by the candidate index has two control point vectors, one additional control point vector may be generated, and the two control point vectors of the affine candidate and the additional control point vector may be set as the control point vectors of the current block. The additional control point vector may be derived based on at least one of the two control point vectors of the affine candidate, the size or position information of the current / adjacent block.

[0203] Alternatively, the two control point vectors of the specified affine candidate may be set as the control point vectors of the current block. In this case, the type of the affine model of the current block may be updated to four parameters.

[0204] Based on the control point vectors of the current block, the motion vector of the current block is derived (S720).

[0205] The motion vector may be derived in units of sub-blocks of the current block. For this purpose, the current block may be divided into a plurality of N×M sub-blocks. Here, the N×M sub-block may be in a rectangular (N>M or N<M) or square (N = M) shape. The values of N and M may be 4, 8, 16, 32 or more.

[0206] For example, the size / shape of the sub-block may be a fixed size / shape already defined in the decoding device. For example, the size / shape of the sub-block may be a square such as 4×4, 8×8, 16×16, etc., or a rectangle such as 2×4, 2×8, 4×8, 4×16, etc. Alternatively, the sub-block may be defined as a block whose sum of width and height is an integer of 8, 12, 16 or more. Alternatively, the sub-block may be defined as a block whose product of width and height is an integer of 16, 32, 64 or more.

[0207] Alternatively, the size / shape of the sub-blocks may be variably derived based on the aforementioned block attributes. For example, if the size of the current block is greater than or equal to a predetermined threshold size, the current block may be divided into a first sub-block unit (e.g., 8×8, 16×16), otherwise, the current block may be divided into a second sub-block unit (e.g., 4×4).

[0208] Alternatively, information regarding the size / shape of the sub-blocks may be coded and signaled in the coding device, the information indicating at least one of the size or shape of the sub-blocks, which may be signaled at at least one of the following levels: sequence, picture, tile group, tile, CTU.

[0209] The control point representing the current block may include the top left corner position and the top right corner position, but is not limited thereto, and the control point may include three positions, such as the top left corner position, the top right corner position, and the bottom left corner position, or may further include more positions.

[0210] When two control points are used, a motion vector for each sub-block of the current block may be derived using at least one of a first control point vector corresponding to the first control point, a second control point vector corresponding to the second control point, the position (x,y) of the sub-block, or the size (width or height) of the current block.

[0211] When three control points are used, a motion vector for each subblock of the current block may be derived using at least one of a first control point vector corresponding to the first control point, a second control point vector corresponding to the second control point, a third control point vector corresponding to the third control point, the position (x, y) of the subblock, or the size (width or height) of the current block. In this case, a difference vector between the second control point vector and the first control point vector may be used, or a difference vector between the third control point vector and the first control point vector may be used. The difference vector may be calculated for each of the horizontal direction (x-axis direction) and the vertical direction (y-axis direction).

[0212] The step of deriving the motion vector may further include a step of applying a predetermined offset to the motion vector derived based on the control point vector. The offset means a vector for improving an already derived motion vector. The offset may be determined based on information on at least one of a size or a direction of the offset. The size (Absolute) may be an integer of 1, 2, 3 or more. The direction may include at least one direction of the left side, the right side, the top end or the bottom end. The information on the size and / or the direction of the offset may be coded and signaled in the coding device. Alternatively, the size of the offset may be a fixed value already set in the decoding device.

[0213] Using the derived motion vector, inter prediction is performed on the current block (S730).

[0214] Specifically, the reference block may be identified using a motion vector of the current block. The reference block may be identified for each sub-block of the current block. The reference block for each sub-block may belong to one reference picture. That is, the sub-blocks belonging to the current block may share one reference picture. Alternatively, a reference picture index may be set independently for each sub-block of the current block.

[0215] On the other hand, the derived motion vector may include at least one of an L0 motion vector or an L1 motion vector. When the motion vector includes an L0 and an L1 motion vector, the decoding device may set one of the L0 and L1 motion vectors to 0 to perform unidirectional prediction.

[0216] The setting may be selectively performed in consideration of at least one of a block size and a prediction mode, where the block refers to a current block or a sub-block of the current block, and the prediction mode refers to a skip mode, a merge mode, an AMVP mode, a prediction mode based on an affine model, a current picture reference mode, etc.

[0217] For example, if the size of the current block (or a subblock of the current block) is smaller than or equal to a certain threshold size, unidirectional prediction may be performed by setting the L1 motion vector to 0, otherwise bidirectional prediction may be performed using the L0 and L1 motion vectors, where the threshold size may be defined as the size of a block whose width and / or height are 4, 8, or 16.

[0218] Alternatively, if the current block is coded in the first prediction mode, the L1 motion vector may be set to 0 to perform unidirectional prediction; otherwise, bidirectional prediction may be performed using the L0 and L1 motion vectors.

[0219] The identified reference block may be set as a predicted block of the current block, and the current block may be reconstructed by adding a residual block to the predicted block.

[0220] FIG. 8 is a diagram showing a method for deriving affine candidates from control point vectors of spatial / temporal neighboring blocks according to an embodiment of the present invention.

[0221] The width and height of the current block 800 are cbW and cbH, respectively, and the position of the current block is (xCb, yCb). The width and height of the spatially adjacent blocks 810 to 850 are nbW and nbH, respectively, and the positions of the spatially adjacent blocks are (xNb, yNb). The spatially adjacent blocks may include at least one of the left block 810, the lower left block 840, the upper right block 830, the upper end block 820, or the upper left block 850 of the current block. Alternatively, the spatially adjacent blocks may further include at least one of the block adjacent to the right side of the upper left block 850 or the block adjacent to the lower end.

[0222] A spatial candidate may have n control point vectors (cpMVs), where the value of n may be an integer of 1, 2, 3, or more. The value of n may be determined based on at least one of information on whether the block is decoded on a sub-block basis, information on whether the block is coded with an affine model, or information on the type of affine model (4 parameters or 6 parameters).

[0223] For example, according to the information, if the block is a sub-block decoded block or an affine model coded block, the block may have two control point vectors, whereas if not, the block may not be predicted based on the affine model.

[0224] Alternatively, according to the information, if the block is an affine model coded block and the type of the affine model is six parameters, the block may have three control point vectors, whereas if not, the block may not be predicted based on the affine model.

[0225] The above information may be coded and signaled in the coding device. Alternatively, all or part of the information may be derived from the decoding device based on attributes of the block. Here, the block may mean the current block or a spatial / temporal neighboring block of the current block. The attributes may mean a size, a shape, a position, a partition type, an inter mode, parameters related to residual coefficients, etc. The inter mode may mean a mode already defined in the decoding device, such as a merge mode, a skip mode, an AMVP mode, an affine model, an intra / inter combined mode, a current picture reference mode, etc. Alternatively, the n value may be derived from the decoding device based on the above-mentioned attributes of the block.

[0226] In this embodiment, the n control point vectors may be represented as a first control point vector (cpMV[0]), a second control point vector (cpMV[1]), a third control point vector (cpMV[2]), and an nth control point vector (cpMV[n-1]).

[0227] According to an example, the first control point vector (cpMV[0]), the second control point vector (cpMV[1]), the third control point vector (cpMV[2]) and the fourth control point vector (cpMV[3]) may be vectors corresponding to the positions of the top left sample, the top right sample, the bottom left sample and the bottom right sample of the block, respectively. Here, it is assumed that the spatial candidate has three control point vectors, and the three control point vectors may be any control point vectors selected from the first to n-th control point vectors. However, the present invention is not limited thereto, and the spatial candidate may have two control point vectors, and the two control point vectors may be any control point vectors selected from the first to n-th control point vectors.

[0228] In the following, a method for deriving control point vectors for spatial candidates will be described.

[0229] 1. If the current block boundary does not touch the CTU boundary The first control point vector may be derived based on at least one of the first control point vector of a spatially neighboring block, a predetermined difference value, position information (xCb, yCb) of the current block, or position information (xNb, yNb) of the spatially neighboring block.

[0230] The number of the difference values ​​may be one, two, three or more. The number of the difference values ​​may be variably determined in consideration of the above-mentioned block attributes, or may be a fixed value already assigned to the decoding device. The difference value may be defined as a difference value between any one of a plurality of control point vectors and another one. For example, the difference value may include at least one of a first difference value between a second control point vector and a first control point vector, a second difference value between a third control point vector and a first control point vector, a third difference value between a fourth control point vector and a third control point vector, or a fourth difference value between a fourth control point vector and a second control point vector.

[0231] For example, the first control point vector may be derived as shown in Equation 1 below.

[0232]

number

[0233] In Equation 1, the variables mvScaleHor and mvScaleVer may represent the first control point vector of the spatially neighboring block, or may represent a value derived by applying a shift operation corresponding to k to the first control point vector. Here, k may be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9 or more. The variables dHorX and dVerX correspond to the x and y components of the first difference value between the second control point vector and the first control point vector, respectively. The variables dHorY and dVerY correspond to the x and y components of the second difference value between the third control point vector and the first control point vector, respectively. The above variables may be derived as shown in Equation 2 below.

[0234]

number

[0235] The second control point vector may be derived based on at least one of the first control point vector of the spatially adjacent block, a predetermined difference value, position information (xCb, yCb) of the current block, block size (width or height), or position information (xNb, yNb) of the spatially adjacent block. Here, the block size means the size of the current block and / or the spatially adjacent block. The difference value is the same as described in the first control point vector, and therefore a detailed description thereof will be omitted here. However, the range and / or number of the difference value used in the derivation process of the second control point vector may be different from that of the first control point vector.

[0236] For example, the second control point vector may be derived as shown in Equation 3 below.

[0237]

number

[0238] In Equation 3, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 1, and detailed description thereof will be omitted here.

[0239] The third control point vector may be derived based on at least one of the first control point vector of the spatially adjacent block, a predetermined difference value, position information (xCb, yCb) of the current block, block size (width or height), or position information (xNb, yNb) of the spatially adjacent block. Here, the block size means the size of the current block and / or the spatially adjacent block. The difference value is the same as described in the first control point vector, and therefore a detailed description thereof will be omitted here. However, the range and / or number of the difference value used in the derivation process of the third control point vector may be different from that of the first control point vector or the second control point vector.

[0240] For example, the third control point vector may be derived as shown in Equation 4 below.

[0241]

number

[0242] In Equation 4, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 1, and detailed description thereof will be omitted here. Meanwhile, the n-th control point vector of the spatial candidate may be derived through the above-mentioned process.

[0243] 2. If the current block boundary is adjacent to the CTU boundary The first control point vector may be derived based on at least one of a motion vector (MV) of a spatially neighboring block, a predetermined difference value, position information (xCb, yCb) of the current block, or position information (xNb, yNb) of the spatially neighboring block.

[0244] The motion vector may be a motion vector of a sub-block located at the bottom end of the spatially adjacent blocks. The sub-block may be the leftmost, center or rightmost sub-block among a plurality of sub-blocks located at the bottom end of the spatially adjacent blocks. Alternatively, the motion vector may represent an average value, a maximum value or a minimum value of the motion vectors of the sub-blocks.

[0245] The number of the difference values ​​may be one, two, three or more. The number of the difference values ​​may be variably determined in consideration of the above-mentioned block attributes, or may be a fixed value already set to the decoding device. The difference value may be defined as a difference value between any one of a plurality of motion vectors stored in units of sub-blocks in the spatially adjacent blocks and another one of the motion vectors. For example, the difference value may mean a difference value between the motion vector of the lower right sub-block and the motion vector of the lower left sub-block of the spatially adjacent blocks.

[0246] For example, the first control point vector may be derived as shown in Equation 5 below.

[0247]

number

[0248] In Equation 5, the variables mvScaleHor and mvScaleVer may represent the motion vector (MV) of the spatially neighboring block or a value derived by applying a shift operation corresponding to k to the motion vector, where k may be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9, or more.

[0249] The variables dHorX and dVerX correspond to the x and y components of a predetermined difference value, respectively. Here, the difference value means a difference value between the motion vector of the bottom right sub-block and the motion vector of the bottom left sub-block in the spatially adjacent blocks. The variables dHorY and dVerY may be derived based on the variables dHorX and dVerX. The above variables may be derived as shown in the following Equation 6.

[0250]

number

[0251] The second control point vector may be derived based on at least one of the motion vector (MV) of the spatially adjacent block, a predetermined difference value, position information (xCb, yCb) of the current block, block size (width or height), or position information (xNb, yNb) of the spatially adjacent block. Here, the block size means the size of the current block and / or the spatially adjacent block. The motion vector and the difference value are the same as those described in the first control point vector, and therefore a detailed description thereof will be omitted here. However, the position of the motion vector, the range and / or the number of the difference values ​​used in the derivation process of the second control point vector may be different from those of the first control point vector.

[0252] For example, the second control point vector may be derived as shown in Equation 7 below.

[0253]

number

[0254] In Equation 7, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 5, and detailed description thereof will be omitted here.

[0255] The third control point vector may be derived based on at least one of the motion vector (MV) of the spatially adjacent block, a predetermined difference value, position information (xCb, yCb) of the current block, block size (width or height) or position information (xNb, yNb) of the spatially adjacent block. Here, the block size means the size of the current block and / or the spatially adjacent block. The motion vector and the difference value are the same as those described in the first control point vector, and therefore a detailed description thereof will be omitted here. However, the position, range and / or number of the difference value of the motion vector used in the derivation process of the third control point vector may be different from those of the first control point vector or the second control point vector.

[0256] For example, the third control point vector may be derived as shown in Equation 8 below.

[0257]

number

[0258] In Equation 8, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 5, and detailed description thereof will be omitted here. Meanwhile, the n-th control point vector of the spatial candidate may be derived through the above-mentioned process.

[0259] The above-described process of deriving affine candidates may be performed for each of the predefined spatial neighboring blocks, which may include at least one of a left block, a lower left block, a upper right block, an upper block, or an upper left block of the current block.

[0260] Alternatively, the affine candidate derivation process may be performed for each group of the spatially adjacent blocks, where the spatially adjacent blocks may be classified into a first group including a left block and a bottom left block, and a second group including a top right block, a top block, and a top left block.

[0261] For example, one affine candidate may be derived from spatially adjacent blocks belonging to the first group. The derivation may be performed based on a predetermined priority order until a usable affine candidate is found. The priority order may be from the left block to the bottom left block, or vice versa. It may be determined according to the priority order whether the spatially adjacent blocks in the first group are blocks decoded by prediction based on an affine model, and the block decoded by prediction based on an affine model first may be selected as the affine candidate.

[0262] Similarly, one affine candidate may be derived from spatially adjacent blocks belonging to the second group. The derivation may be performed based on a predetermined priority order until a usable affine candidate is found. The priority order may be the top right block→top block→top left block, or may be the reverse order. According to the priority order, it may be determined whether the spatially adjacent blocks in the second group are blocks decoded by prediction based on an affine model, and the block decoded by prediction based on an affine model first may be selected as the affine candidate.

[0263] The above-described embodiment may be similarly / similarly applied to a temporally neighboring block. Here, the temporally neighboring block may be a block that belongs to a different picture from the current block but is located at the same position as the current block. The located block may be a block including the position of the top left sample of the current block, the center position, or the position of a sample adjacent to the bottom right sample of the current block.

[0264] Alternatively, the temporally adjacent block may refer to a block shifted from the block at the same position by a distance corresponding to a predetermined disparity vector, where the disparity vector may be determined based on the motion vector of any one of the spatially adjacent blocks of the current block.

[0265] FIG. 9 is a diagram illustrating a method for deriving constructed candidates based on a combination of motion vectors of spatial / temporal neighboring blocks according to one embodiment of the present invention.

[0266] The constructed candidates of the present invention may be derived based on at least two combinations of control point vectors (hereinafter referred to as control point vectors (cpMVCorner[n])) corresponding to each corner of the current block, where n may be 0, 1, 2, or 3.

[0267] The control point vector (cpMVCorner[n]) may be derived based on the motion vectors of spatially adjacent blocks and / or temporally adjacent blocks. Here, the spatially adjacent blocks may include at least one of the first adjacent block A, B, or C adjacent to the top left sample of the current block, the second adjacent block D or E adjacent to the top right sample of the current block, or the third adjacent block F or G adjacent to the bottom left sample of the current block. The temporally adjacent block refers to a block that belongs to a different picture from the current block and is located at the same position as the current block (hereinafter, referred to as a fourth adjacent block (Col)). Here, the fourth adjacent block may refer to block H, I, or J including the position of the top left sample, top right sample, or bottom left sample of the current block, or may refer to a neighboring block at the position of the bottom right sample of the current block.

[0268] The first neighboring block may refer to the neighboring block at the top left corner A, top corner B, or left side C of the current block. It may be determined whether the motion vectors of the neighboring blocks A, B, and C are available according to a predetermined priority, and the control point vector may be determined using the motion vectors of the available neighboring blocks. The availability determination may be performed until a neighboring block having an available motion vector is found. Here, the priority order may be A→B→C. However, it is not limited thereto, and may be A→C→B, C→A→B, or B→A→C.

[0269] The second neighboring block may refer to the neighboring block at the top end D or the top right end E of the current block. Similarly, it may be determined whether the motion vectors of the neighboring blocks D and E are available according to a predetermined priority, and the control point vector may be determined using the motion vector of the available neighboring block. The availability determination may be performed until a neighboring block with an available motion vector is found. Here, the priority order may be D→E or E→D.

[0270] The third neighboring block may refer to the neighboring block on the left side F or the bottom left corner G of the current block. Similarly, it may be determined whether the motion vector of the neighboring block is available according to a predetermined priority, and the control point vector may be determined using the motion vector of the available neighboring block. The availability determination may be performed until a neighboring block with an available motion vector is found. Here, the priority order may be G→F or F→G.

[0271] For example, the first control point vector (cpMVCorner[0]) may be set to the motion vector of the first adjacent block, the second control point vector (cpMVCorner[1]) may be set to the motion vector of the second adjacent block, the third control point vector (cpMVCorner[2]) may be set to the motion vector of the third adjacent block, and the fourth control point vector (cpMVCorner[3]) may be set to the motion vector of the fourth adjacent block.

[0272] Alternatively, the first control point vector may be derived using at least one motion vector of the first adjacent block or the fourth adjacent block, where the fourth adjacent block may be block H including the position of the top left sample. The second control point vector may be derived using at least one motion vector of the second adjacent block or the fourth adjacent block, where the fourth adjacent block may be block I including the position of the top right sample. The third control point vector may be derived using at least one motion vector of the third adjacent block or the fourth adjacent block, where the fourth adjacent block may be block J including the position of the bottom left sample.

[0273] Alternatively, any one of the first to fourth control point vectors may be derived based on the other one. For example, the second control point vector may be derived by applying a predetermined offset vector to the first control point vector. The offset vector may be a difference vector between the third control point vector and the first control point vector, and may be derived by applying a predetermined scaling factor to the difference vector. The scaling factor may be determined based on at least one of the width and height of the current block and / or the neighboring block.

[0274] A combination of at least two of the first to fourth control point vectors may determine K constructed candidates (ConstK) according to the present invention. The value of K may be an integer of 1, 2, 3, 4, 5, 6, 7 or more. The value of K may be derived based on information signaled in the encoding device, or may be a value already promised to the decoding device. The information may include information indicating the maximum number of constructed candidates to be included in a candidate list.

[0275] Specifically, the first constructed candidate (Const1) may be derived by combining the first to third control point vectors. For example, the first constructed candidate (Const1) may have a control point vector as shown in Table 4 below. Meanwhile, only when the reference picture information of the first neighboring block is the same as the reference picture information of the second and third neighboring blocks, the control point vector may be restricted to be constructed as shown in Table 4. Here, the reference picture information may mean a reference picture index indicating the position of the reference picture in the reference picture list, or may mean a POC (picture order count) value indicating an output order.

[0276] [Table 4]

[0277] The second constructed candidate (Const2) may be derived by combining the first, second and fourth control point vectors. For example, the second constructed candidate (Const2) may have a control point vector as shown in Table 5 below. Meanwhile, the control point vector may be restricted to be constructed as shown in Table 5 only when the reference picture information of the first neighboring block is the same as the reference picture information of the second and fourth neighboring blocks. Here, the reference picture information is as described above.

[0278] [Table 5]

[0279] The third constructed candidate (Const3) may be derived by combining the first, third and fourth control point vectors. For example, the third constructed candidate (Const3) may have a control point vector as shown in Table 6 below. Meanwhile, only when the reference picture information of the first neighboring block is the same as the reference picture information of the third and fourth neighboring blocks, the control point vector may be restricted to be constructed as shown in Table 6. Here, the reference picture information is as described above.

[0280] [Table 6]

[0281] The fourth constructed candidate (Const4) may be derived by combining the second, third and fourth control point vectors. For example, the fourth constructed candidate (Const4) may have a control point vector as shown in Table 7 below. Meanwhile, the fourth constructed candidate (Const4) may be restricted to be constructed as shown in Table 7 only when the reference picture information of the second neighboring block is the same as the reference picture information of the third and fourth neighboring blocks. Here, the reference picture information is as described above.

[0282] [Table 7]

[0283] The fifth constructed candidate (Const5) may be derived by combining the first and second control point vectors. For example, the fifth constructed candidate (Const5) may have a control point vector as shown in Table 8 below. Meanwhile, the control point vector may be restricted to be constructed as shown in Table 8 only when the reference picture information of the first neighboring block is the same as the reference picture information of the second neighboring block. Here, the reference picture information is as described above.

[0284] [Table 8]

[0285] The sixth constructed candidate (Const6) may be derived by combining the first and third control point vectors. For example, the sixth constructed candidate (Const6) may have a control point vector as shown in Table 9 below. Meanwhile, only when the reference picture information of the first neighboring block is the same as the reference picture information of the third neighboring block, the control point vector may be restricted to be constructed as shown in Table 9. Here, the reference picture information is as described above.

[0286] [Table 9]

[0287] In Table 9, cpMvCorner[1] may be a second control point vector derived based on the first and third control point vectors. The second control point vector may be derived based on at least one of the first control point vector, a predetermined difference value, or a size of the current / neighboring block. For example, the second control point vector may be derived as shown in the following Equation 9.

[0288]

number

[0289] In the above-mentioned K constructed candidates (ConstK), the value of K does not limit the position or priority of the constructed candidates arranged in the candidate list.

[0290] Also, all of the first to sixth configured candidates may be included in the candidate list, or only some of them may be included in the candidate list.

[0291] For example, if it is determined that the current block uses three control point vectors, only the constructed candidates generated by a combination of three of the first to fourth control point vectors may be used, if it is determined that the current block uses two control point vectors, the constructed candidates generated by a combination of at least two of the first to fourth control point vectors may be used, or the constructed candidates generated by only a combination of two of the first to fourth control point vectors may be used.

[0292] Alternatively, only some of the constructed candidates may be included in the candidate list in consideration of the maximum number of affine candidates included in the candidate list. Here, the maximum number may be determined based on maximum number information signaled in the encoding device, or may be variably determined in consideration of the above-mentioned attributes of the current block. In this case, the K value of the constructed candidate (ConstK) may represent a priority order for insertion into the candidate list.

[0293] On the other hand, if the difference value between two control point vectors belonging to the constructed candidate is smaller than a predetermined threshold, the constructed candidate may not be inserted into the candidate list. The difference value between the two control point vectors may be classified into a horizontal difference value and a vertical difference value. Here, the horizontal difference value means the difference value between the first control point vector 910 and the second control point vector 920, and the vertical difference value means the difference value between the first control point vector 910 and the third control point vector 930. The threshold may mean 0 or a vector of a size already promised to the encoding / decoding device.

[0294] FIG. 10 is a diagram illustrating a method for deriving motion information of sub-block based temporal candidates in one embodiment of the present invention.

[0295] The motion information of a temporal candidate according to the present invention may be derived from the motion information of a collocated block. The motion information may be derived on a sub-block basis.

[0296] Here, the collocated block is a block belonging to a picture different from the current block 1010 (i.e., the collocated picture), and may be the block 1040 at the same position as the current block or the block 1030 at a position shifted by an amount corresponding to the temporal vector from the position of the current block. The temporal vector may be determined based on the motion vectors of the spatially adjacent blocks of the current block. The spatially adjacent blocks may mean the blocks adjacent to at least one of the left side, the lower left end, the upper end, the upper right end, or the upper left end of the current block. The temporal vector may be determined using only the adjacent blocks at the positions already agreed upon in the encoding / decoding apparatus. For example, the already agreed upon positions may be the left side 1011 or the upper end 1012, or may be the left side 1011 and the upper end 1012. When there are a plurality of adjacent blocks on the left side, among the plurality of adjacent blocks, the block located at any one of the lowermost end, the uppermost end, or the center may be used. When there are a plurality of adjacent blocks at the upper end, among the plurality of adjacent blocks, the block located at any one of the leftmost side, the rightmost side, or the center may be used.

[0297] When deriving the temporal candidates based on the sub-blocks, the current block and / or the collocated block may be divided into a plurality of sub-blocks.

[0298] Here, the sub-block may have a certain size / shape already agreed upon in the encoding / decoding apparatus. For example, the sub-block may be represented by an N×M block, and the values of N and M may be integers of 4, 8, or more. The sub-block may be square (N = M) or rectangular (N>M, N<M). It may also mean the size. Alternatively, the encoding apparatus may encode and signal the information regarding the size / shape of the sub-block, and the decoding apparatus may determine the size / shape of the sub-block based on the signaled information.

[0299] Alternatively, the current block and / or the co-located block may be divided into multiple sub-blocks based on a predetermined number, where the number may be a fixed number already promised to the encoding / decoding device, or may be variably determined taking into account the block size / shape, or the number may be determined based on number information signaled in the encoding device.

[0300] A method of deriving motion information of a temporal candidate in sub-block units will be described below. A motion vector of a temporal candidate may be set to a motion vector stored for each sub-block of a co-located block. A reference picture index of a temporal candidate may be set to a value (e.g., 0) that is predefined in an encoding / decoding apparatus. Alternatively, the reference picture index of the temporal candidate may be set to a reference picture index of a co-located block or a reference picture index of an upper leftmost sub-block among sub-blocks. Alternatively, like a motion vector, a reference picture index of a temporal candidate may also be set to a reference picture index stored for each sub-block.

[0301] However, if there is an unavailable subblock among the subblocks belonging to the co-located block, the motion vector of the unavailable subblock may be replaced with the motion vector of an available subblock in the co-located block. Here, the available subblock means a block adjacent to any one of the left, right, top, or bottom of the unavailable subblock. Alternatively, the available subblock may be a block at a position already promised to the encoding / decoding device. For example, the already promised position may be the position of the bottom right subblock in the co-located block, or may be the position of a subblock including a center position of the co-located block. The temporal candidate based on the subblock may be added to the candidate list only if the available subblock exists. Alternatively, the temporal candidate based on the subblock may be added to the candidate list only if the subblock at the already promised position is available.

[0302] FIG. 11 is a diagram showing an inter prediction method in the ERP projection format, which is an embodiment to which the present invention is applied.

[0303] Referring to FIG. 11, in the case of the ERP projection format, the reference region specified by the motion vector may straddle the boundary of the reference picture.

[0304] In this case, the border of the reference picture may be expanded and the expanded area may be filled horizontally with data adjacent to the border (see A). Alternatively, the border of the reference picture may be expanded and data adjacent to the border on the opposite side of the expanded area (data that is continuous or correlated with the border) may be filled (see B). Alternatively, data in the non-existent area may be filled with data adjacent to the opposite border (see C).

[0305] Although A and B require a process of extending the reference picture data, the calculation process can be simplified, and C does not require additional memory due to the extension, but may require additional calculation processes.

[0306] In the cases of A and B, the size of the current picture and the size of the (extended) reference picture may not be the same. A predetermined adjustment process that takes into account such size differences may be included in the process of obtaining predicted pixels.

[0307] When comparing only the horizontal length, the current picture may have a range of 0 to (pic_width-1), and the reference picture (extended as on the right side) may have a range of -L_offset to (pic_width-1+R_offset) or a range of 0 to (pic_width-1+L_offset+R_offset). In the former case, a predicted pixel may be obtained from the extended reference picture at a position obtained by adding the coordinates of the current block and the motion vector. In the latter case, a predicted pixel may be obtained from a re-adjusted position after an adjustment process of the obtained position is performed. Here, L_offest and R_offset may mean lengths extended in the left and right directions. In the present invention, as described above, a case will be described assuming that the coordinates are calculated based on the range of the current picture or the existing reference picture.

[0308] In case C, the coordinates of the current block and the motion vector may be added to calculate the position in the reference picture. If the calculated pixel position is within the range of the reference picture (<0,0>~<pic_width-1,pic_height-1> ) or not. If the calculated pixel position is not outside the range of the reference picture, a predicted pixel may be obtained from the calculated pixel position. On the other hand, if the calculated pixel position is outside the range of the reference picture, a position corresponding to the calculated position may be calculated from the opposite position (the right side when it is the left side, and the left side when it is the right side) inside the boundary <inside the picture>, and a predicted pixel may be obtained from the calculated corresponding position.

[0309] Inter prediction may be performed on the boundary of a reference picture according to at least one of the above-mentioned embodiments. The inter prediction method on the boundary of a reference picture may be one already specified in the encoding / decoding device, and may be determined based on selection information specifying one of the above-mentioned embodiments and information related to the selected method (such as data processing method, extension size information, etc.). This is also applicable to the embodiments described below.

[0310] 12 to 15 are diagrams showing an inter prediction method in the CMP projection format, which is an embodiment to which the present invention is applied.

[0311] 12, in the case of the CMP projection format, continuity in 3D space may exist between the upper and lower 3×1 faces due to region-wise packing, but continuity in 3D space may not exist between the upper 3×1 face and the lower 3×1 face.

[0312] Similarly, the reference region identified by the motion vector may straddle the border of the reference picture or may straddle the border of some surface. In case A, it points inside the reference picture but lies on the border of some surface (which has low correlation due to lack of mutual continuity in 3D space), and in case B, the reference region lies on the border of the reference picture.

[0313] In particular, in the case of A, although the inside of the reference picture is indicated, the reference region includes a discontinuous surface boundary, so there is a risk of the accuracy of inter prediction being degraded.

[0314] In cases A and B, the coordinates of the current block and the motion vector may be added to calculate the position of the reference area in the reference picture. It may be determined whether at least a part of the reference area is outside the range of the reference picture or whether it contains a discontinuous boundary in the reference picture.

[0315] If the reference region is outside the range of the reference picture, a position where continuity or correlation exists in the three-dimensional space may be determined, and the predicted pixel may be obtained from the determined position.

[0316] On the other hand, if the reference area is not outside the range of the reference picture, it may be determined whether at least one pixel in the reference area indicated by the motion vector of the current block belongs to two or more surfaces, where the determination process may be performed according to pixel positions such as the top left, top right, bottom left, and bottom right of the current block.

[0317] If at least one pixel in the reference area does not belong to two or more surfaces, the predicted pixel may be obtained from the reference area. However, if this is not the case, it may be determined whether the surface has continuity with the surface to which the current block belongs. In the case of reference area A', the top edge area has continuity with the surface to which the current block belongs, and the bottom edge area does not have continuity with the surface to which the current block belongs.

[0318] If the aforementioned continuity exists, the predicted pixels may be obtained from the reference region, but if not, the predicted pixels may be obtained from a region that has continuity or correlation in the three-dimensional space.

[0319] The above-described inter prediction order is merely an example, and various modifications can be made by changing the order, changing some of the configuration, and so on.

[0320] In the cases of C and D, the reference picture may be extended, and the extended region may be padded horizontally / vertically with data adjacent to the boundary, or may be packed with data from a correlated region. The case of D is as described in ERP, and a detailed description thereof will be omitted here. In the case of C, like A, the reference region is located inside the reference picture, but includes a surface boundary with low correlation. Therefore, if a separate extended region is not generated inside the reference picture, the accuracy of inter prediction may be reduced, which will be described in detail with reference to FIG. 13.

[0321] In the cases of C and D, the size of the current picture and the size of the reference picture may not be the same due to the extension of the reference picture. In this case, the reference picture may have a range of (-L_offset, -U_offset) to (pic_width-1+R_offset, pic_height-1+D_offset). Here, L_, R_, U_, and D_offset may mean the length extended in the left, right, top, and bottom directions from the reference picture. The offsets may have the same value or different values. The length of extension according to the reference picture may be explicitly signaled or may be implicitly determined.

[0322] For example, the extension length may correspond to one of the lengths that can be handled by the block division unit, such as 2, 4, 8, etc. Alternatively, the extension length may be determined based on the smallest coding block, the largest coding block, the smallest predicted block, the largest predicted block, etc. Alternatively, the extension length may be determined based on units such as tiles, bricks, etc.

[0323] For example, the prediction block may be extended by an amount corresponding to the length of the maximum prediction block, in which case, the prediction block of all blocks may be obtained within the extended range. If the area indicated by the motion vector of the current block is outside the range of the extended reference picture despite the extension of the reference picture, the prediction pixels of the part corresponding to the outside range may be obtained by padding in the horizontal, vertical, or diagonal direction.

[0324] Referring to FIG. 13, cases A to D correspond to cases where the reference picture is extended in units of one or more surfaces. Cases A and B are examples where the reference picture (or sub-reference picture) is extended by grouping surfaces where continuity exists. Cases C and D are examples where the reference picture is extended individually in units of surfaces. Data processing of the extended area is the same as or similar to the above-mentioned embodiment. In the above-mentioned embodiment, there was one range related to the reference picture, but in this example, the reference picture is divided into two or more sheets and each area (called a sub-reference picture) is extended, so that overlapping ranges may occur.

[0325] In the examples of A and B, the extended sub-reference picture located at the top has a range of (-L_offset,-U_offset) to (pic_width-1+R_offset,pic_height / 2-1+D_offset), and the extended sub-reference picture located at the bottom has a range of (-L_offset,pic_height / 2-U_offset) to (pic_width-1+R_offset,pic_height-1+D_offset). It is assumed that the extended left, right, top, and bottom lengths are the same in each sub-reference picture. However, the extended lengths in the left, right, top, and bottom directions may be set to be the same or different in the extended units. The extended lengths may be set individually according to the extended units.

[0326] In this case, the coordinates of the range extended downward from the upper sub-reference picture are the same as the coordinates within the lower sub-reference picture, but in the case of the upper sub-reference picture, the area contains extended correlated data, and in the case of the lower sub-reference picture, it contains the original data before extension.

[0327] In short, the reference picture may have a range of (-L_offset,-U_offset) to (pic_width-1+R_offset,pic_height-1+D_offset). Generally, it is appropriate to assign one pixel value to each position, but since the reference picture is divided into two or more regions and expanded in consideration of the characteristics of the omnidirectional image, two or more pixel values ​​may be assigned in some ranges. In this case, it may be required to determine what pixel value is to be obtained as a predicted pixel value from a position to which two or more pixel values ​​are assigned.

[0328] Referring to FIG. 14, in case E, it may be determined which sub-reference picture includes the larger region, and a prediction value may be obtained using data of the sub-reference picture including the larger region.

[0329] In the case of F, if the sub-reference picture including the current block already contains a set number of pixels or more, or contains a predetermined width, the data of the sub-reference picture may be used. However, if this is not the case, processing using data of a sub-reference picture of an area including a larger area, such as E, may be possible.

[0330] In the case of G, the current block is included equally in both sub-reference pictures. In this case, the sub-reference picture including the current block may be given priority and the data of the sub-reference picture may be used. This example may be included as a detailed condition of F and E.

[0331] In short, when the area indicated by the motion vector of the current block is a boundary between sub-reference pictures, it may be determined which sub-reference picture data to use depending on the included area, whether the sub-reference picture is the same as the current block, etc. It may be determined which sub-reference picture data to use for inter prediction, including all or part of the above conditions, according to the encoding / decoding settings.

[0332] In the above-mentioned example, the reference picture is divided into two and each region is expanded, but in this example, a case will be described in which even more sub-reference pictures are generated on a surface-by-surface basis.

[0333] In Fig. 15, it is assumed that each surface unit is a sub-reference picture. Similarly, it is necessary to determine what pixel value to obtain as a predicted pixel value from a position where two or more pixel values ​​are assigned.

[0334] Referring to FIG. 15, in the case of H, data of a sub-reference picture of a region that is more included may be used, or data of a sub-reference picture that includes the current block may be used.

[0335] For example, if a sub-reference picture including the current block is included in a number of pixels already set or in a range already defined, the data of the sub-reference picture may be used. Alternatively, if data of a sub-reference picture including a larger area is used, the data of the sub-reference picture may be used. However, if the included area is the same, the data of the sub-reference picture including the current block may be limited to be used.

[0336] In the case of I, data of a sub-reference picture with a larger included area may be used. In this example, the sub-reference picture can be handled on a surface-by-surface basis, but since the surface setting has a characteristic that the upper end 3×1 area and the lower end 3×1 area are not continuous, a priority can be given to the sub-reference picture on the side having continuous characteristics over the sub-reference picture on the side having discontinuous characteristics.

[0337] In the case of J, data of a sub-reference picture with a larger included area may be used. However, this is the case when the included areas are uniform, and in this example, like I, the sub-reference picture containing the current block (the right side at the top of the 3×2) is not a target to be straddled. And, this is the case when the sub-reference picture (the left side at the top, center of the 3×2) belongs to the same 3×1 as the sub-reference picture containing the current block. In this case, priority can be given to the sub-reference picture closer to the sub-reference picture containing the current block.

[0338] In short, when the area indicated by the motion vector of the current block is a boundary between sub-reference pictures, it may be determined which sub-reference picture data to use depending on the encompassing area, the usage priority of the sub-reference picture, etc.

[0339] Meanwhile, a method for performing inter prediction based on affine motion modeling has been described with reference to Figures 7 to 10. This can be used in a similar manner to encoding / decoding 360 video, and detailed description thereof will be omitted here.

[0340] The methods of the present invention may be embodied in the form of program instructions stored on computer-readable media that can be executed by various computer means. The computer-readable media may include, alone or in combination with program instructions, data files, data structures, and the like. The program instructions stored on the computer-readable media may be those specially designed and constructed for the purposes of the present invention, or they may be of the type well known and available to those skilled in the art of computer software.

[0341] Examples of computer readable media include hardware devices specially adapted to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions may include not only machine code, such as produced by a compiler, but also high-level language code that may be executed by a computer using an interpreter, etc. The aforementioned hardware devices may be configured to operate as at least one software module to perform the operations of the present invention, and vice versa.

[0342] Furthermore, the above-described methods or devices may be realized in such a manner that all or part of their configurations or functions are combined or separated.

[0343] Although the preferred embodiments of the present invention have been described above, it will be understood that those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention as defined in the appended claims. [Industrial Applicability]

[0344] The present invention can be used in encoding / decoding video signals.

Claims

1. 1. A method for decoding a video signal, comprising the steps of: constructing an affine candidate list for the current block, the affine candidate list including at least one of spatial candidates, temporal candidates or constructed candidates, and a maximum number of affine candidates to be included in the affine candidate list is determined based on information regarding a maximum number parsed from the bitstream; parsing affine candidate indices from the bitstream; determining a control point vector of the current block based on the affine candidate list and the affine candidate index; determining a motion vector of the current block based on the control point vector of the current block; performing an affine mode based prediction of the current block based on the motion vector of the current block.

2. The method for decoding a video signal comprises:

2. The method of claim 1, further comprising the step of parsing information regarding a type of the affine mode from the bitstream, the information indicating whether a four-parameter based affine mode or a six-parameter based affine mode is applied.

3. When the four-parameter based affine mode is applied, one affine candidate in the affine candidate list includes two control point vectors; determining a control point vector of the current block based on the affine candidate list and the affine candidate index, 3. A method for decoding a video signal as claimed in claim 2, further comprising the step of determining two control point vectors indicated by said affine candidate indexes.

4. When the six-parameter based affine mode is applied, one affine candidate in the affine candidate list includes three control point vectors; determining a control point vector of the current block based on the affine candidate list and the affine candidate index, 3. A method for decoding a video signal as claimed in claim 2, further comprising the step of determining three control point vectors indicated by said affine candidate indexes.

5. The method of claim 2, wherein the constructed candidates are determined based on a combination of at least two control point vectors.

6. When the four-parameter based affine mode is applied, the constructed candidate is determined based on a combination of two control point vectors; or The method of claim 5, wherein when the six-parameter based affine mode is applied, the constructed candidates are determined based on a combination of three control point vectors.

7. the two control point vectors include a first control point vector and a second control point vector; determining a motion vector of the current block based on the control point vector of the current block, 4. The method of claim 3, further comprising determining a motion vector for the current block based on the first control point vector, the second control point vector, and a size of the current block.

8. the three control point vectors include a first control point vector, a second control point vector, and a third control point vector; determining a motion vector of the current block based on the control point vector of the current block, 5. The method of claim 4, further comprising determining a motion vector for the current block based on the first control point vector, the second control point vector, the third control point vector, and a size of the current block.

9. 1. A video signal decoding device, comprising: A device for decoding a video signal, comprising a processor configured to carry out the method for decoding a video signal according to any one of claims 1 to 8.

10. 1. A method for encoding a video signal, comprising the steps of: constructing an affine candidate list for the current block, the affine candidate list including at least one of spatial candidates, temporal candidates or constructed candidates, and a maximum number of affine candidates to be included in the affine candidate list is determined based on information regarding a maximum number parsed from the bitstream; determining a control point vector of the current block based on affine candidates included in the affine candidate list, the affine candidates being indicated by affine candidate indexes; determining a motion vector of the current block based on the control point vector of the current block; performing an affine mode based prediction of the current block based on the motion vector of the current block; encoding the affine candidate indexes for the current block into a bitstream.

11. The method for encoding a video signal comprises:

11. The method of claim 10, further comprising the step of: encoding information regarding the type of affine mode into the bitstream, the information indicating whether a four-parameter based affine mode or a six-parameter based affine mode is applied.

12. The method for encoding a video signal according to claim 11, wherein when the four-parameter based affine mode is applied, one affine candidate in an affine candidate list includes two control point vectors.

13. The method for encoding a video signal according to claim 11, wherein when the six-parameter based affine mode is applied, one affine candidate in an affine candidate list includes three control point vectors.

14. The method of claim 11, wherein the constructed candidates are determined based on a combination of at least two control point vectors.

15. When the four-parameter based affine mode is applied, the constructed candidate is determined based on a combination of two control point vectors; or The method of claim 14, wherein, when the six-parameter based affine mode is applied, the constructed candidates are determined based on a combination of three control point vectors.

16. the two control point vectors include a first control point vector and a second control point vector; determining a motion vector of the current block based on the control point vector of the current block, 13. The method of claim 12, further comprising determining a motion vector for the current block based on the first control point vector, the second control point vector, and a size of the current block.

17. the three control point vectors include a first control point vector, a second control point vector, and a third control point vector; determining a motion vector of the current block based on the control point vector of the current block, 14. The method of claim 13, further comprising determining a motion vector for the current block based on the first control point vector, the second control point vector, the third control point vector, and a size of the current block.

18. An apparatus for encoding a video signal, comprising: Apparatus for encoding a video signal, comprising a processor configured to carry out the method for encoding a video signal according to any one of claims 10 to 17.

19. A non-transitory computer readable storage medium comprising a bitstream, said bitstream being generated by a method for encoding a video signal according to any one of claims 10 to 17.

Images