Interpretation method and apparatus based on merge mode

Abstract

This invention provides an inter-prediction method and apparatus that improves the accuracy of motion information by using not only spatial / temporal merge candidates but also combined merge candidates. [Solution] The method includes the steps of: constructing a merge candidate list for the current block; deriving motion information for the current block from the merge candidate list; and performing an inter-block prediction for the current block using the motion information. The merge candidate list includes at least one of spatial merge candidates, temporal merge candidates, or combined merge candidates, and the combined merge candidate is derived by combining n merge candidates belonging to the merge candidate list.

Description

Technical Field

[0001] The present invention relates to an inter prediction method and apparatus.

Background Art

[0002] In recent years, demands for high-resolution and high-quality images such as high-definition (HD) images and ultra-high-definition (UHD) images have been increasing in various application fields. Along with this, highly efficient image compression technologies have been studied.

[0003] As image compression technologies, there are various technologies such as an inter prediction technology that predicts pixel values included in a current picture from pictures before or after the current picture, an intra prediction technology that predicts pixel values included in the current picture using pixel information within the current picture, and an entropy coding technology that assigns short codes to values with high occurrence frequencies and long codes to values with low occurrence frequencies. Using such image compression technologies, image data can be effectively compressed for transmission or storage.

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to provide an inter prediction method and apparatus.

[0005] An object of the present invention is to provide a method and apparatus for constructing a merge candidate list.

[0006] An object of the present invention is to provide a motion compensation method and apparatus in sub-block units.

[0007] An object of the present invention is to provide a method and apparatus for determining an affine candidate.

[0008] The present invention aims to provide an interpretation method and apparatus in accordance with the 360 ​​video projection format. [Means for solving the problem]

[0009] The video encoding / decoding method and apparatus according to the present invention may configure a merge candidate list for the current block, derive motion information for the current block from the merge candidate list, and perform inter-block prediction using the motion information.

[0010] In the video encoding / decoding method and apparatus according to the present invention, the merge candidate list may include at least one of spatial merge candidates, temporal merge candidates, or combined merge candidates.

[0011] In the video encoding / decoding method and apparatus according to the present invention, the combined merge candidate may be derived by combining n merge candidates belonging to the merge candidate list.

[0012] In the video encoding / decoding method and apparatus according to the present invention, the n merge candidates may be merge candidates corresponding to indices 0 to (n-1) in the merge candidate list.

[0013] In the video encoding / decoding method and apparatus according to the present invention, the index of the combination merge candidate may be greater than the index of the temporal merge candidate.

[0014] In the video encoding / decoding method and apparatus according to the present invention, the n merge candidates include a first merge candidate and a second merge candidate, and the motion information of the combined merge candidate may be derived considering the predicted direction of the first merge candidate and the predicted direction of the second merge candidate.

[0015] In the video encoding / decoding method and apparatus according to the present invention, the motion information of the combination merge candidate may be derived by taking a weighted average of the motion information of the first merge candidate and the motion information of the second merge candidate.

[0016] In the video encoding / decoding method and apparatus according to the present invention, the weights of the weighted average may be any one of [1:1], [1:2], [1:3], or [2:3].

[0017] The video encoding / decoding method and apparatus according to the present invention may generate a candidate list for predicting motion information of the current block, derive a control point vector of the current block based on the candidate list and candidate index, derive a motion vector of the current block based on the control point vector of the current block, and perform interpretation on the current block using the motion vector.

[0018] In the video encoding / decoding device according to the present invention, the candidate list may include a plurality of affine candidates.

[0019] In the video encoding / decoding device according to the present invention, the affine candidate may include at least one of spatial candidates, temporal candidates, or configured candidates.

[0020] In the video encoding / decoding device according to the present invention, the motion vector of the current block may be derived in units of subblocks of the current block.

[0021] In the video encoding / decoding device according to the present invention, the spatial candidate may be determined by considering whether the boundary of the current block touches the boundary of the encoding tree block (CTU boundary).

[0022] In the video encoding / decoding device according to the present invention, the configured candidates may be determined based on at least two combinations of control point vectors corresponding to each corner of the current block.

[0023] According to the video encoding / decoding method and apparatus of the present invention, when the reference region for inter prediction includes the boundary of the reference picture or the boundary between discontinuous surfaces, all or some pixels of the reference region may be obtained by using data of relevant regions.

Advantages of the Invention

[0024] According to the present invention, the accuracy of motion information can be improved by using not only spatial / temporal merge candidates but also combination merge candidates.

[0025] According to the present invention, the performance of image encoding / decoding can be improved by inter prediction based on an affine model.

[0026] According to the present invention, the accuracy of prediction can be improved by inter prediction in sub-block units.

[0027] According to the present invention, the efficiency of inter prediction encoding / decoding can be improved by determining efficient affine candidates.

[0028] According to the present invention, the encoding efficiency of inter prediction can be improved by setting a reference region considering correlation.

Brief Description of the Drawings

[0029] [Figure 1] It is a block diagram of an image encoding apparatus according to an embodiment of the present invention. [Figure 2] It is a block diagram of an image decoding apparatus according to an embodiment of the present invention. [Figure 3] It is an exemplary diagram showing the shape of a tree-based block. [Figure 4] It is an exemplary diagram showing various block shapes that can be obtained in the block division unit of the present invention. [Figure 5] It is a diagram showing the block division process according to an embodiment of the present invention. [Figure 6] This figure shows a method for performing inter-prediction based on block-level merge modes, which is one embodiment to which the present invention is applied. [Figure 7] This figure shows an embodiment to which the present invention is applied, which is an inter-prediction method based on an affine model. [Figure 8] This figure shows a method for deriving affine candidates from control point vectors of spatially / temporal adjacent blocks, in one embodiment to which the present invention is applied. [Figure 9] This figure shows a method for deriving a configured candidate based on a combination of motion vectors of spatially / temporal adjacent blocks, in one embodiment to which the present invention is applied. [Figure 10] This figure shows a method for deriving temporal candidate motion information based on subblocks, in one embodiment to which the present invention is applied. [Figure 11] This figure shows an embodiment to which the present invention is applied, which is a method for predicting interoperability in an ERP projection format. [Figure 12] This figure shows an embodiment to which the present invention is applied, which is a method for predicting interoperability in the CMP projection format. [Figure 13] This figure shows an embodiment to which the present invention is applied, which is a method for predicting interoperability in the CMP projection format. [Figure 14] This figure shows an embodiment to which the present invention is applied, which is a method for predicting interoperability in the CMP projection format. [Figure 15] This figure shows an embodiment to which the present invention is applied, which is a method for predicting interoperability in the CMP projection format. [Modes for carrying out the invention]

[0030] The video encoding / decoding method and apparatus according to the present invention may configure a merge candidate list for the current block, derive motion information for the current block from the merge candidate list, and perform inter-block prediction using the motion information.

[0031] In the video encoding / decoding method and apparatus according to the present invention, the merge candidate list may include at least one of spatial merge candidates, temporal merge candidates, or combined merge candidates.

[0032] In the video encoding / decoding method and apparatus according to the present invention, the combined merge candidate may be derived by combining n merge candidates belonging to the merge candidate list.

[0033] In the video encoding / decoding method and apparatus according to the present invention, the n merge candidates may be merge candidates corresponding to indices 0 to (n-1) in the merge candidate list.

[0034] In the video encoding / decoding method and apparatus according to the present invention, the index of the combination merge candidate may be greater than the index of the temporal merge candidate.

[0035] In the video encoding / decoding method and apparatus according to the present invention, the n merge candidates include a first merge candidate and a second merge candidate, and the motion information of the combined merge candidate may be derived considering the predicted direction of the first merge candidate and the predicted direction of the second merge candidate.

[0036] In the video encoding / decoding method and apparatus according to the present invention, the motion information of the combination merge candidate may be derived by taking a weighted average of the motion information of the first merge candidate and the motion information of the second merge candidate.

[0037] In the video encoding / decoding method and apparatus according to the present invention, the weights of the weighted average may be any one of [1:1], [1:2], [1:3], or [2:3].

[0038] The video encoding / decoding method and apparatus according to the present invention may generate a candidate list for predicting motion information of the current block, derive a control point vector of the current block based on the candidate list and candidate index, derive a motion vector of the current block based on the control point vector of the current block, and perform interpretation on the current block using the motion vector.

[0039] In the video encoding / decoding device according to the present invention, the candidate list may include a plurality of affine candidates.

[0040] In the video encoding / decoding device according to the present invention, the affine candidate may include at least one of spatial candidates, temporal candidates, or configured candidates.

[0041] In the video encoding / decoding device according to the present invention, the motion vector of the current block may be derived in units of subblocks of the current block.

[0042] In the video encoding / decoding device according to the present invention, the spatial candidate may be determined by considering whether the boundary of the current block is adjacent to the boundary of the coding tree block (CTU boundary).

[0043] In the video encoding / decoding device according to the present invention, the configured candidates may be determined based on at least two combinations of control point vectors corresponding to each corner of the current block.

[0044] The video encoding / decoding method and apparatus according to the present invention may, when the reference region for interpretation includes the boundary of a reference picture or the boundary between discontinuous surfaces, acquire all or part of the pixels of the reference region using data from correlated regions.

[0045] The present invention can be modified in various ways and may have various embodiments, but specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, or substitutes that fall within the spirit and technical scope of the present invention.

[0046] Terms such as "first," "second," "A," and "B" can be used to describe various components, but the components are not limited in any way by such terms. The terms are used solely for the purpose of distinguishing one component from another. For example, within the scope of the rights of the present invention, the first component may be named the second component, and similarly, the second component may also be named the first component. The terms "and / or" include a combination of multiple related descriptions or any one of multiple related descriptions.

[0047] When one component is described as being "linked" or "connected" to another component, it should be understood that this may mean it is directly linked or connected to the other component, or that another component may exist in between. Conversely, when one component is described as being "directly linked" or "directly connected" to another component, it should be understood that there is no other component in between.

[0048] Furthermore, the terms used in this application are used solely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Terms such as “includes” or “has” in this application merely specify the existence of features, figures, stages, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preemptively exclude the possibility of the existence or addition of one or more other features, figures, stages, operations, components, parts, or combinations thereof.

[0049] Unless otherwise specified, all terms used herein, including technical and scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this invention pertains. Commonly used, dictionary-defined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined herein.

[0050] The image encoding and decoding device may be a user terminal such as a personal computer (PC), notebook computer, personal digital assistant (PDA), portable multimedia player (PMP), PlayStation Portable (PSP), wireless communication terminal (Wireless Communication Terminal), smartphone (Smart Phone), TV, virtual reality device (VR), augmented reality device (AR), mixed reality device (MR), head-mounted display (HMD), or smart glasses, or it may be a server terminal such as an application server or service server. It may also be equipped with various devices such as communication modems for communicating with various devices or wired / wireless networks, memory for storing various programs and data for encoding or decoding images and performing intra or inter prediction for encoding or decoding, and a processor for launching, calculating, and controlling programs. Furthermore, the image encoded as a bitstream by the image encoding device may be transmitted in real time or non-real time to an image decoding device via a wired wireless network (Network) such as the Internet, a short-range wireless communication network, a wireless LAN network, a WiBro network, or a mobile communication network, or via various communication interfaces such as a cable or a Universal Serial Bus (USB), where it may be decoded, restored to an image, and reproduced.

[0051] Alternatively, the image encoded as a bitstream by the image encoding device may be transferred from the encoding device to the decoding device via a computer-readable recording medium.

[0052] The aforementioned image encoding device and image decoding device may be separate devices, but they may also be manufactured as a single image encoding / decoding device if necessary. In that case, some components of the image encoding device may be substantially the same technical elements as some components of the image decoding device, and may include at least the same structure or perform at least the same function.

[0053] Therefore, in providing detailed explanations of the technical elements and their operating principles below, redundant explanations of corresponding technical elements will be omitted.

[0054] Furthermore, since the image decoding device corresponds to a computing device that applies the image encoding method performed in the image encoding device to decoding, the following explanation will focus on the image encoding device.

[0055] A computing device may include a memory for storing programs or software modules that implement an image encoding method and / or an image decoding method, and a processor connected to the memory for executing the program. Note that an image encoding device is sometimes referred to as an encoder, and an image decoding device is sometimes referred to as a decoder.

[0056] Typically, an image consists of a series of still images, which are divided into GOP (Group of Pictures) units, and each still image is sometimes referred to as a picture. In this case, a picture may represent one of the frames or fields in a progressive signal or an interlaced signal. An image can be represented as a "frame" when encoding / decoding is performed in frame units, and as a "field" when it is performed in field units. In this invention, the explanation assumes a progressive signal, but it is also applicable to interlaced signals. Higher-level concepts such as GOP and Sequence may exist, and each picture may be divided into predetermined areas such as slices, tiles, and blocks. Furthermore, one GOP may contain units such as I-picture, P-picture, and B-picture. An I-picture refers to a picture that is encoded / decoded automatically without using a reference picture, while P-pictures and B-pictures refer to pictures that are encoded / decoded using a reference picture through processes such as motion estimation and motion compensation. Generally, in the case of a P-picture, an I-picture and a P-picture may be used as reference pictures, and in the case of a B-picture, an I-picture and a P-picture may be used as reference pictures, but this is not limited to this, and the above definitions can also be changed depending on the encoding / decoding settings.

[0057] Here, the picture referenced during encoding / decoding is referred to as the Reference Picture, and the referenced block or pixel is referred to as the Reference Block or Reference Pixel. Furthermore, the referenced data may not only be pixel values ​​in the spatial domain, but also coefficient values ​​in the frequency domain, and various encoding / decoding information generated and determined during encoding / decoding. For example, this includes information related to intra-prediction or motion in the prediction unit, information related to transformation in the transformation / inverse transformation unit, information related to quantization in the quantization / inverse quantization unit, information related to encoding / decoding (context information) in the encoding / decoding unit, and information related to filtering in the in-loop filtering unit.

[0058] The smallest unit of an image is a pixel, and the number of bits used to represent one pixel is called the bit depth. Generally, the bit depth is 8 bits, but depending on the encoding settings, higher bit depths are possible. Depending on the color space, at least one bit depth may be supported. Also, depending on the color format of the image, it may consist of at least one color space. Depending on the color format, it may consist of one or more pictures of a fixed size or one or more pictures of different sizes. For example, if YCbCr = 4:2:0, it may consist of one luminance component (Y in this example) and two chrominance components (Cb / Cr in this example), in which case the ratio of chrominance component to luminance component may be horizontal:vertical = 1:2. As another example, if it is 4:4:4, the ratio of horizontal to vertical may be the same. As in the above example, if it consists of one or more color spaces, the picture may be divided into each color space.

[0059] In this invention, the explanation is based on a certain color format (in this example, YCbCr) and a certain color space (in this example, Y), and the same or similar application (settings dependent on a specific color space) may be made to other color spaces (in this example, Cb, Cr) corresponding to the color format. However, it is also possible to have partial differences for each color space (settings independent of a specific color space). That is, settings dependent on each color space may mean settings that are proportional to or dependent on the composition ratio of each component (for example, determined according to 4:2:0, 4:2:2, 4:4:4, etc.), while settings independent of each color space may mean that settings are made only for that color space, regardless of or independently of the composition ratio of each component. In this invention, depending on the encoder / decoder, some configurations may have independent settings, or they may have dependent settings.

[0060] The configuration information or syntax elements required in the image encoding process may be defined at the unit level, such as video, sequence, picture, slice, tile, or block. This information may be recorded in the bitstream in units such as VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), slice header, tile header, or block header, and transmitted to the decoder. The decoder may then parse the information at the same level to reconstruct the configuration information transmitted from the encoder and use it in the image decoding process. Alternatively, related information may be transmitted to the bitstream in the form of SEI (Supplement Enhancement Information) or metadata, and then parsed and used. Each parameter set has a unique ID value, and a sub-parameter set may have the ID value of the higher-level parameter set it should reference. For example, a sub-parameter set may reference information from one or more higher-level parameter sets that have a matching ID value. Among the various unit examples mentioned above, a unit in which any one unit contains one or more other units is sometimes referred to as a higher-level unit, and the units containing it are sometimes referred to as lower-level units.

[0061] In the case of setting information generated in the aforementioned unit, it may include content relating to settings independent of each unit, or it may include content relating to settings dependent on preceding, succeeding, or higher-level units. Here, dependent settings can be understood as representing the setting information of the unit with flag information (for example, a 1-bit flag where 1 means follow, and 0 means do not) indicating that it follows the settings of preceding, succeeding, or higher-level units. In describing the setting information in the present invention, we will focus on examples relating to independent settings, but examples in which content relating to dependent relationships to the setting information of preceding, succeeding, or higher-level units of the current unit is added or replaced may also be included.

[0062] Figure 1 is a block diagram of an image encoding device according to one embodiment of the present invention. Figure 2 is a block diagram of an image decoding device according to one embodiment of the present invention.

[0063] Referring to Figure 1, the image coding device may include a prediction unit, a subtraction unit, a transformation unit, a quantization unit, an inverse quantization unit, an inverse transformation unit, an addition unit, an in-loop filter unit, and a memory and / or coding unit. Some of the above components are not necessarily provided, and some or all of them may be selectively provided as needed, and further components not shown may also be provided.

[0064] Referring to Figure 2, the image decoding device may include a decoding unit, a prediction unit, an inverse quantization unit, an inverse transform unit, an addition unit, an in-loop filter unit and / or memory, and some of the above components are not necessarily provided, and some or all of them may be selectively provided as needed, and further components not shown may be provided.

[0065] The image encoding device and the image decoding device may be separate devices, but they may be manufactured as a single image encoding / decoding device if necessary. In that case, some components of the image encoding device may be substantially the same technical elements as some components of the image decoding device, and may include at least the same structure or be implemented to perform at least the same function. Therefore, in the detailed explanation of the technical elements and their operating principles below, redundant explanations of corresponding technical elements will be omitted. Since the image decoding device can correspond to a computing device that applies the image encoding method performed in the image encoding device to decoding, the following explanation will focus on the image encoding device. The image encoding device is sometimes called an encoder, and the image decoding device is sometimes called a decoder.

[0066] The prediction unit may include an intra-prediction unit that performs intra-prediction and an inter-prediction unit that performs inter-prediction. Intra-prediction determines an intra-prediction mode by configuring pixels of blocks adjacent to the current block as reference pixels, and generates a prediction block using the intra-prediction mode. Inter-prediction determines motion information of the current block using one or more reference images, and generates a prediction block by performing motion compensation using the motion information. It is determined which of the intra-prediction method or the inter-prediction method to use for the current block (encoded unit or prediction unit), and specific information associated with each prediction method (e.g., intra-prediction mode, motion vector, reference image, etc.) is determined. In this case, the processing unit in which prediction is performed, the prediction method, and the processing unit in which specific content is defined may be determined according to the encoding / decoding settings. For example, the prediction method, prediction mode, etc. are determined at the prediction unit (or encoding unit), and prediction is performed at the prediction block unit (or encoding unit, transformation unit).

[0067] The subtraction unit generates a residual block by subtracting the predicted block from the current block. In other words, the subtraction unit calculates the difference between the pixel value of each pixel in the current block to be encoded and the predicted pixel value of each pixel in the predicted block generated by the prediction unit, thereby generating a residual block, which is a residual signal in the shape of a block.

[0068] The conversion unit can convert signals belonging to the spatial domain into signals belonging to the frequency domain, and the signals obtained through this conversion process are called transformed coefficients. For example, a residual block containing a residual signal received from the subtraction unit is converted to obtain a transformed block containing transformed coefficients, but the input signal is determined according to the encoding settings and is not limited to residual signals.

[0069] The transformation unit may transform the residual block using transformation methods such as the Hadamard Transform, Discrete Sine Transform (DST-Based Transform), or Discrete Cosine Transform (DCT-Based Transform), but is not limited to these, and may use various transformation methods that are improvements or modifications of these methods.

[0070] The transformation may be performed in the horizontal or vertical direction. For example, the pixel values ​​in the spatial domain may be converted to the frequency domain by performing a one-dimensional transformation in the horizontal direction and a one-dimensional transformation in the vertical direction using the basis vectors in the transformation, resulting in a total of two dimensions of transformation.

[0071] Alternatively, the conversion unit may pass the information necessary for generating the conversion block to the encoding unit for encoding, record the associated information in a bitstream and transmit it to the decoder, and the decoder's decoding unit may parse this information and use it in the inverse conversion process.

[0072] The quantization unit may quantize the input signal, and the signal obtained through the quantization process is called the quantized coefficient. For example, a residual block having residual conversion coefficients passed from the conversion unit is quantized to obtain a quantized block having quantized coefficients, but the input signal is determined according to the coding settings, and this is not limited to residual conversion coefficients.

[0073] The quantization unit may quantize the transformed residual block using quantization methods such as Dead Zone Uniform Threshold Quantization or Quantization Weighted Matrix, but is not limited to these, and may use various quantization methods that are improvements or modifications thereof.

[0074] The quantization unit may pass the information necessary for generating quantization blocks to the encoding unit for encoding, record the associated information in a bitstream and transmit it to the decoder, and the decoder's decoding unit may parse this information and use it in the dequantization process.

[0075] In the above example, it was assumed that the residual block is transformed and quantized by a transformation unit and a quantization unit. However, it is also possible to transform the residual signal to generate a residual block with transformation coefficients and not perform the quantization process of the residual block. Furthermore, it is possible to perform only the quantization process without converting the residual signal of the residual block to transformation coefficients, or even to perform neither transformation nor quantization. This can be determined according to the settings of the encoder.

[0076] The coding unit may scan the quantization coefficients, transformation coefficients, or residual signals of the generated residual block according to at least one scan sequence (e.g., zigzag scan, vertical scan, horizontal scan, etc.) to generate a sequence of quantization coefficients, a sequence of transformation coefficients, or a sequence of signals, and encode them using at least one entropy coding method. In this case, information regarding the scan sequence may be determined according to the coding settings (e.g., coding mode, prediction mode, etc.), implicitly determined, or the relevant information may be explicitly generated. For example, one of several scan sequences may be selected depending on the intra-prediction mode. In this case, the scan pattern may be set to one of various patterns such as zigzag, diagonal, or raster.

[0077] Furthermore, encoded data containing encoded information passed from each component may be generated and output as a bitstream, which may be implemented by a multiplexer (MUX). In this case, encoding methods such as Exponential Golomb, Context Adaptive Variable Length Coding (CAVLC), and Context Adaptive Binary Arithmetic Coding (CABAC) may be used, but are not limited to these, and various improved and modified encoding methods may be used.

[0078] When performing entropy coding (in this example, assuming CABAC) on syntactic elements such as residual block data and information generated during the coding / decoding process, the entropy coding device may include a binarizer, a context modeler, and a binary arithmetic coder. In this case, the binary arithmetic coder may include a regular coding engine and a bypass coding engine. In this case, the regular coding engine may be a process performed in relation to the context modeler, while the bypass coding engine may be a process performed independently of the context modeler.

[0079] Since the syntactic elements input to the entropy encoding device may not be binary, if the syntactic elements are not binary, the binarization unit may binarize the syntactic elements and output a bin string consisting of 0s or 1s. In this case, the bins represent bits consisting of 0s or 1s and may be encoded by a binary arithmetic encoding unit. In this case, either a regular encoding unit or a bypass encoding unit may be selected based on the probability of occurrence of 0s and 1s, and this may be determined according to the encoding / decoding settings. If the syntactic elements are data with equal frequencies of 0s and 1s, the bypass encoding unit may be used; otherwise, the regular encoding unit may be used, and this may be referenced when performing the next process by the regular encoding unit through context modeling (or context information updating).

[0080] In this context, the context is information about the probability of a bin occurring, and context modeling is the process of estimating the probability of a bin required for binary arithmetic coding, using the bins, which are the result of binarization, as input. For probability estimation, syntactic element information of the bin, the index which is the position of the bin in the bin string, the probability of bins included in the surrounding blocks of a block, etc., may be used, and at least one context table may be used for this purpose. For example, for information on some flags, multiple context tables may be used depending on the combination of whether the flags are used or not in the surrounding blocks.

[0081] Various methods may be used when performing binarization on the syntactic elements. For example, it may be divided into fixed-length binarization and variable-length binarization, and in the case of variable-length binarization, unary binarization (truncated unary binarization), truncated rice binarization, K-order Exp-Golomb binarization, truncated binary binarization, etc. may be used. In addition, signed binarization or unsigned binarization may be performed depending on the range of values ​​that the syntactic elements have. The binarization process for syntactic elements generated in the present invention may be carried out not only by the binarization methods mentioned in the above examples, but also by other further binarization methods.

[0082] The inverse quantization unit and the inverse transformation unit may be implemented by reversing the processes in the transformation unit and the quantization unit. For example, the inverse quantization unit may inverse quantize the quantized transformation coefficients generated in the quantization unit, and the inverse transformation unit may inverse transform the inverse quantized transformation coefficients to generate the restored residual block.

[0083] The addition unit adds the predicted block and the restored residual block to restore the current block. The restored block may be stored in memory and used as reference data (for the prediction unit and filter unit, etc.).

[0084] The loop filtering section may include at least one post-processing filtering step, such as a deblocking filter, a sample adaptive offset (SAO), or an adaptive loop filter (ALF). The deblocking filter may remove block distortion at the boundaries between blocks from the reconstructed image. The ALF may perform filtering based on a comparison between the reconstructed image and the input image. More specifically, after the blocks are filtered by the deblocking filter, filtering may be performed based on a comparison between the reconstructed image and the input image. Alternatively, after the blocks are filtered by the SAO, filtering may be performed based on a comparison between the reconstructed image and the input image.

[0085] Memory may store restored blocks or pictures. Restored blocks or pictures stored in memory may be provided to a prediction unit that performs intra-prediction or inter-prediction. More specifically, a storage space for a queue type of compressed bitstream in the encoder may be prepared and processed as a Coded Picture Buffer (CPB), and a space for storing decoded images in picture units may be prepared and processed as a Decoded Picture Buffer (DPB). In the case of a CPB, the decoded units are stored according to the decoding order, the decoding operation is emulated within the encoder, and the compressed bitstream is stored during the emulation process. The bitstream output from the CPB is restored through the decoding process, the restored images are stored in the DPB, and the pictures stored in the DPB are referenced thereafter in the image encoding and decoding processes.

[0086] The decoding unit may be implemented by reversing the process in the encoding unit. For example, it may receive a sequence of quantization coefficients, a sequence of transformation coefficients, or a sequence of signals from a bitstream and decode them, or it may parse the decoded data containing the decoded information and pass it to each component.

[0087] A block division unit may be provided in the image coding device and image decoding device according to one embodiment of the present invention. The basic coding unit means the basic (or starting) unit related to prediction, transformation, quantization, etc. in the image coding / decoding process. In this case, the coding unit may consist of one luminance coding block and two chrominance coding blocks according to the color format (YCbCr in this example), and the size of each block may be determined according to the color format. In the examples described later, the block (luminance component in this example) will be used as the basis for explanation. In this case, it will be assumed that the block is a unit that can be obtained after each unit has been determined, and that similar settings can be applied to other types of blocks.

[0088] The block division section may be set in relation to each component of the image encoding device and the decoding device, and the size and shape of the blocks may be determined by this process. In this case, the blocks to be set may be defined differently depending on the component, such as a prediction block in the case of the prediction unit, a transformation block in the case of the transformation unit, and a quantization block in the case of the quantization unit. However, this is not limited to this, and further block units associated with other components may be defined. The size and shape of the blocks may be defined by the width and height of the blocks.

[0089] In the block division section, blocks are represented as M×N, and the maximum and minimum values ​​of each block are obtained within the specified range. For example, a square block shape is possible, and if the maximum block size is 256×256 and the minimum block size is 8×8, then a block of size 2m×2m (in this example, m is an integer from 3 to 8; for example, 8×8, 16×16, 32×32, 64×64, 128×128, 256×256), or a block of size 2m×2m (in this example, m is an integer from 4 to 128), or a block of size m×m (in this example, m is an integer from 8 to 256) may be obtained. Alternatively, the block shape can correspond to squares and rectangular prisms, and if it has the same range as the example above, a block of size 2m × 2n may be obtained (in this example, m and n are integers from 3 to 8. Assuming the maximum width-to-height ratio is 2:1, examples include 8×8, 8×16, 16×8, 16×16, 16×32, 32×16, 32×32, 32×64, 64×32, 64×64, 64×128, 128×64, 128×128, 128×256, 256×128, 256×256. Depending on the encoding / decoding settings, there may be no restrictions on the width-to-height ratio, or there may be a maximum value for the width-to-height ratio). Alternatively, a block of size 2m × 2n may be obtained (in this example, m and n are integers from 4 to 128). Alternatively, you can obtain a block of size m x n (in this example, m and n are integers from 8 to 256).

[0090] In the block division section, the input block may be converted into one or more blocks. For example, the input block may be output as is, or it may be converted into two or more subblocks. In this case, the number of subblocks may be two, three, four, etc., and this may be determined according to the division method (division type). The shape of the subblock may be obtained based on the block before division (in this example, the input block), or it may have various shapes according to the block division settings.

[0091] While this invention focuses on a wood-based division method, it is by no means limited to this.

[0092] The obtainable blocks may be determined according to the encoding / decoding settings (e.g., block type, partitioning method, partitioning settings, etc.). For example, the encoding block may be a 2m x 2n block, the prediction block may be a 2m x 2n or m x n block, and the transformation block may be a 2m x 2n block. Based on the above settings, information such as block size and range (e.g., information related to exponents and multiples, etc.) may be generated.

[0093] Depending on the type of block, the range (defined in this example by the maximum and minimum values) may be defined. Furthermore, for some blocks, the range information may be explicitly generated, while for others, the range information may be implicitly defined. For example, for encoding and transformation blocks, the relevant information may be explicitly generated, while for prediction blocks, the relevant information may be processed implicitly.

[0094] If explicit, at least one range information may be generated. For example, in the case of a coded block, the range information may generate information about the maximum and minimum values. Alternatively, it may be generated based on the difference between the maximum value and a previously set minimum value (e.g., 8) (e.g., generated based on the aforementioned setting, such as information about the difference in the exponent between the maximum and minimum values). Note that multiple range information may be generated for the horizontal and vertical lengths of a rectangular block.

[0095] If implicit, range information may be obtained based on the encoding / decoding settings (e.g., block type, partitioning scheme, partitioning settings). For example, in the case of a prediction block, the maximum and minimum value information may be obtained from the higher-level unit, the encoded block (e.g., the encoded block's maximum size M×N and minimum size m×n), as a group of candidates obtainable through the partitioning settings of the prediction block (e.g., quadtree partitioning + partitioning depth 0) (in this example, M×N and m / 2×n / 2).

[0096] The size and shape of the initial (or starting) block of the block division section may be determined from the higher-level unit. In the case of a coding block, the basic coding block may be the initial block; in the case of a prediction block, the coding block may be the initial block; and in the case of a transformation block, either the coding block or the prediction block may be the initial block, which may be determined according to the coding / decoding settings. For example, if the coding mode is Intra, the prediction block may be a higher-level unit of the transformation block; if it is Inter, the prediction block may be a unit independent of the transformation block. The initial block is the starting unit of the division and may be divided into smaller blocks. Once the optimal size and shape for each block division is determined, that block may be determined as the initial block of the lower-level unit. For example, in the former case it may be the coding block, and in the latter case (lower-level unit) it may be the prediction block or the transformation block. As in the example above, once the initial block of the lower-level unit is determined, a division process may be performed to find a block of the optimal size and shape, similar to the higher-level unit.

[0097] In short, the block division unit may divide the basic coding unit (or the largest coding unit) into at least one coding unit (or lower coding unit). The coding unit may be divided into at least one prediction unit, or at least one transformation unit. The coding unit may be divided into at least one coding block, and the coding block may be divided into at least one prediction block, or at least one transformation block. The prediction unit may be divided into at least one prediction block, and the transformation unit may be divided into at least one transformation block.

[0098] As in the example above, when the optimal size and shape of a block is found through a mode determination process, mode information (e.g., division information) related to this block may be generated. The mode information may be recorded in a bitstream along with information generated in the components to which the block belongs (e.g., information related to prediction, information related to transformation, etc.) and transmitted to the decoder, where it may be parsed at the same level and used in the image decoding process.

[0099] In the examples described later, the division method will be explained assuming that the initial block is a square, but the same or similar methods may be applicable if the block is a rectangular prism.

[0100] Figure 3 is an illustrative diagram showing the shape of a wood-based block.

[0101] Examples show that a is a single 2N×2N block that has not been split, b is two 2N×N blocks obtained through some splitting flags (in this example, horizontal splitting of a binary tree), c is two N×2N blocks obtained through some splitting flags (in this example, vertical splitting of a binary tree), and d is four N×N blocks obtained through some splitting flags (in this example, quad partitioning of a quadtree). The shape of the obtained blocks may be determined depending on the type of tree used for splitting. For example, when performing a quad tree split, the obtainable candidate blocks may be a and d. When performing a binary tree split, the obtainable candidate blocks may be a, b, and c. In the case of a quad tree, it is possible to correspond to one splitting flag, and if the flag is "0", a may be obtained, and if it is "1", d may be obtained. In the case of a binary tree, it is possible to correspond to multiple splitting flags, one of which may be a flag indicating whether or not a split has occurred, and one of which may be a flag indicating whether or not a horizontal / vertical split has occurred. In the case of a quadtree, the partitioning scheme may be based on a basic tree, and may include a further tree partitioning scheme (a binary tree in this example). Multiple tree partitions may be performed if a flag allowing further tree partitions is implicitly or explicitly activated. The tree partitioning scheme may be recursive; that is, a partitioned block may be set back as the initial block for tree partitioning, which may be determined according to partitioning settings such as the partition range and allowed partition depth. This may be an example of a hierarchical partitioning scheme.

[0102] Figure 4 is an illustrative diagram showing various block shapes that can be obtained in the block division section of the present invention.

[0103] Referring to Figure 4, depending on the division settings and division method, blocks a to s may be obtained, and the shapes of further blocks not shown may also be obtained.

[0104] According to Example 1, tree-based partitioning may allow asymmetric partitioning. For example, in the case of a binary tree, blocks such as b and c may be obtainable, or blocks such as b to g may be obtainable. If the flag allowing asymmetric partitioning is explicitly or implicitly deactivated according to the encoding / decoding settings, the obtainable candidate block may be b or c, and if the flag allowing asymmetric partitioning is activated, the obtainable candidate block may be b, d, e (horizontal partitioning in this example), or c, f, g (vertical partitioning in this example).

[0105] In the above example, we assume that the ratio of the left:right or top:bottom lengths of the asymmetric partition is 1:3 or 3:1, but we are not limited to these. Depending on the coding settings, we can also have a group of candidates with other ratios (e.g., 1:2, 1:4, 2:3, 2:5, 3:5, etc.).

[0106] Next, we will show various examples of partition information that arises in binary tree partitioning (in this example, candidate groups of 1:1, 1:3, and 3:1).

[0107] For example, a flag indicating whether or not a division exists, a flag indicating the direction of the division, and a flag indicating the shape of the division may be generated. In this case, the shape of the division may mean a symmetrical or asymmetrical division. If an asymmetrical division is determined to be the shape of the division, a flag indicating the division ratio may be generated, and an index may be assigned according to the already set candidate group. If the candidate group can accommodate division ratios of 1:3 or 3:1, the division ratio may be selected using a 1-bit flag.

[0108] Alternatively, a flag indicating the division ratio may be generated in addition to the flag indicating whether or not division is performed and the flag indicating the division direction. In this example, the candidate group for the division ratio may include candidates having a symmetrical ratio of 1:1.

[0109] In this invention (when the flag allowing asymmetric partitioning is activated), we assume that the binary tree partition has a configuration like the former example, and unless otherwise specified, binary tree means symmetric binary tree.

[0110] According to Example 2, further tree partitioning may be permitted in a tree-based partition. For example, partitioning into a ternary tree, quad-type tree, or octa-tree is possible, which may yield n partitioned blocks (in this example, 3, 4, and 8, where n is an integer). In the case of a ternary tree, the corresponding blocks (in this example, if partitioned into multiple blocks) may be h to m; in the case of a quad-tree, the corresponding blocks may be n to p; and in the case of an octa-tree, the corresponding block may be q. Whether the aforementioned tree-based partitions are compatible may be implicitly determined according to the encoding / decoding settings, or the related information may be explicitly generated. Note that these may be used alone according to the encoding / decoding settings, or they may be used in combination with binary tree, quad-tree partitioning, etc.

[0111] For example, in the case of a binary tree, blocks such as b and c may be obtainable, and when a binary tree and a ternary tree are used together (in this example, we assume that the scope of use of the binary tree and the ternary tree partially overlap), blocks such as b, c, i, l may also be obtainable. If a flag that allows further partitioning in addition to the existing tree is explicitly or implicitly deactivated according to the encoding / decoding settings, the obtainable candidate block may be b or c, and if activated, the obtainable candidate block may be b, i or b, h, i, j (in this example, a horizontal partition), or c, l or c, k, l, m (in this example, a vertical partition).

[0112] In the above example, the explanation assumes that the ratio of the lengths of the left:middle:right or top:middle:bottom of the ternary tree is 2:1:1, 1:2:1, or 1:1:2. However, it is not limited to these ratios, and other ratios are possible depending on the encoding settings.

[0113] Next, we will show an example of the partition information that arises in a ternary tree partition (in this example, a candidate with a 1:2:1 structure).

[0114] For example, a flag indicating whether a partition exists, a flag indicating the direction of the partition, and a flag indicating the type of partition may be generated. In this case, the type of partition may mean a binary tree or a ternary tree partition.

[0115] In this invention, adaptive encoding / decoding settings may be applied depending on the partitioning method.

[0116] For example, the partitioning method may be determined according to the type of block. For instance, a quadtree partitioning method may be used for the encoding block and the transformation block, while a quadtree and binary (or ternary, etc.) partitioning method may be used for the prediction block.

[0117] For example, the partitioning method may be determined according to the size of the block. For instance, a quadtree partition may be used for a portion of the range between the maximum and minimum values ​​of the block (e.g., a×b to c×d, where the latter is even larger), while a binary tree (or ternary tree, etc.) partition may be used for a portion of the range (e.g., e×f to g×h). In this case, range information corresponding to the partitioning method may be explicitly generated or implicitly determined, and overlapping ranges are also acceptable.

[0118] For example, the partitioning method may be determined according to the shape of the block (or the block before partitioning). For instance, if the block is square, partitioning into a quadtree and a binary tree (or a ternary tree, etc.) may be possible. Alternatively, if the block is rectangular, partitioning based on a binary tree (or a ternary tree, etc.) may be possible.

[0119] For example, the partitioning settings may be determined according to the type of block. For instance, in a tree-based partitioning scheme, a quadtree partitioning scheme may be used for the coding block and prediction block, and a binary tree partitioning scheme for the transformation block. Alternatively, the allowable partitioning depth may be set to m for the coding block, n for the prediction block, and o for the transformation block, and m, n, and o may or may not be the same.

[0120] For example, the partitioning settings may be determined according to the size of the block. For instance, a quadtree partition may be possible for a portion of the block (e.g., a×b to c×d), a binary tree partition for a portion of the block (e.g., e×f to g×h; in this example, it is assumed that c×d is greater than g×h), and a ternary tree partition for a portion of the block (e.g., i×j to k×l; in this example, it is assumed that g×h is greater than or equal to k×l). In this case, the range may include all ranges between the maximum and minimum values ​​of the block, and the ranges may have settings that do not overlap or settings that overlap. For example, the minimum value of a portion of the range may be equal to the maximum value of a portion of the range, or the minimum value of a portion of the range may be less than the maximum value of a portion of the range. If there are overlapping ranges, the partitioning method with the higher maximum value may have priority, or information on which partitioning method to use may be explicitly generated. In other words, in a partitioning scheme with priority, the decision to perform a lower-priority partitioning scheme may be made according to the partitioning result, or the choice of partitioning scheme may be made according to the selection information of the partitioning scheme.

[0121] For example, the partitioning settings may be determined according to the shape of the block. For instance, if the block is square, a quadtree partition is possible. Alternatively, if the block is rectangular, a binary or ternary partition is possible.

[0122] For example, partition settings may be determined according to encoding / decoding information (e.g., slice type, color components, encoding mode, etc.). For example, if the slice type is I, quadtree (or binary tree, ternary tree) partitioning may be possible in a limited range (e.g., a×b~c×d), if it is P, in a limited range (e.g., e×f~g×h), and if it is B, in a limited range (e.g., i×j~k×l). Also, if the slice type is I, the allowable partition depth for quadtree (or binary tree, ternary tree) partitioning may be set to m, if it is P, to n, and if it is B, to o, where m, n, and o may be the same or different. Some slice types may have the same settings as other slices (e.g., P and B slices).

[0123] In another example, if the color component is a luminance component, the allowable quad tree (or binary tree, ternary tree) partitioning depth may be set to m, and if it is a color difference component, it may be set to n, and m and n may or may not be the same. Furthermore, the range of the quad tree (or binary tree, ternary tree) partitioning when the color component is a luminance component (e.g., a×b~c×d) and the range of the quad tree (or binary tree, ternary tree) partitioning when the color component is a color difference component (e.g., e×f~g×h) may or may not be the same.

[0124] In another example, if the encoding mode is intra, the allowable depth of the quadtree (or binary tree, ternary tree) partition may be m, and if it is inter, it may be n (in this example, n is assumed to be greater than m), and m and n may be the same or different. Also, the range of the quadtree (or binary tree, ternary tree) partition when the encoding mode is intra and the range of the quadtree (or binary tree, ternary tree) partition when the encoding mode is inter may be the same or different.

[0125] In the example above, information regarding the feasibility of constructing an adaptive group of candidate partitions according to the encoding / decoding information may be explicitly generated or implicitly determined.

[0126] The above example illustrates how the partitioning method and partitioning settings are determined according to the encoding / decoding settings. The above example shows only some cases associated with each element, and variations to other cases are also possible. Furthermore, the partitioning method and partitioning settings may be determined according to the combination of multiple elements. For example, the partitioning method and partitioning settings may be determined by the block type, size, shape, encoding / decoding information, etc.

[0127] Furthermore, in the above example, the elements involved in the division method, settings, etc., may be implicitly determined, or information may be explicitly generated to determine whether or not the adaptive cases such as the above example are permissible.

[0128] In the above partitioning settings, the partitioning depth refers to the number of times the initial block has been spatially divided relative to it (in this example, the partitioning depth of the initial block is 0), and as the partitioning depth increases, it can be further divided into smaller blocks. Note that settings related to depth may differ depending on the partitioning method. For example, in a tree-based partitioning method, a single common depth may be used for the partitioning depth of a binary tree and a ternary tree, while different depths may be used for the partitioning depth of a quadtree and a binary tree, and individual depths may be used depending on the type of tree.

[0129] In the above example, if individual split depths are used depending on the type of tree, the split depth may be set to 0 at the tree's splitting start position (in this example, the block before splitting). The split depth may also be calculated based on the position where the splitting begins, without being based on the splitting range of each tree (in this example, the maximum value).

[0130] Figure 5 shows a block partitioning process according to one embodiment of the present invention. Specifically, it shows examples of the size and shape of blocks that can be obtained according to one or more partitioning methods, starting with a basic coded block.

[0131] In the diagram, thick solid lines represent basic coding blocks, thick dotted lines represent quadtree partition boundaries, double solid lines represent symmetric binary tree partition boundaries, solid lines represent ternary tree partition boundaries, and thin dotted lines represent asymmetric binary tree partition boundaries. Except for the thick solid lines, the boundaries represent divisions according to each partitioning method. The partitioning settings described below (e.g., partitioning type, partitioning information, order of configuration of partitioning information, etc.) are not limited to the example given, and various variations are possible.

[0132] For the sake of explanation, we will assume that the basic coded block (2N×2N, 128×128) is the base, and that the upper-left, upper-right, lower-left, and lower-right blocks (N×N, 64×64) have individual block partitioning settings. First, we will assume that in the initial block, four subblocks are obtained through one partitioning operation (partitioning depth 0→1, i.e., the partitioning depth increases by 1), and that, according to the partitioning settings for the quadtree, the maximum coded block is 128×128, the minimum coded block is 8×8, and the maximum partitioning depth is 4, and that these settings are applied commonly to each block.

[0133] (Number 1. Top left block. A0~A6) This example is applicable to single-tree partitioning (in this example, a quadtree), where the size and shape of obtainable blocks may be determined by a single block partitioning setting such as the maximum encoded block, minimum encoded block, and partitioning depth. In this example, there is one block that can be obtained as a result of the partitioning (divided horizontally and vertically into two), and the partitioning information required for one partitioning operation (based on the original block of 4M×4N, with the partitioning depth increasing by 1) is a flag indicating whether or not a partition has occurred (in this example, 0 means no partition, 1 means partition has occurred), and the obtainable candidates are 4M×4N and 2M×2N.

[0134] (Number 2. Upper right block. A7~A11)

[0135] This example supports multi-tree partitioning (quadrutree and binary tree in this example), where the size and shape of obtainable blocks are determined by multiple block partitioning settings. In this example, for a binary tree, it is assumed that the maximum encoded block is 64x64, the minimum encoded block has one side length of 4, and the maximum partitioning depth is 4.

[0136] In this example, since there are two or more blocks that can be obtained as a result of the split (two or four in this example), the splitting information required for one splitting operation (increase of 1 in quadtree splitting depth) is a flag indicating whether a split exists, a flag indicating the type of split, a flag indicating the shape of the split, and a flag indicating the direction of the split. The obtainable candidates are 4M×4N, 4M×2N, 2M×4N, 4M×N / 4M×3N, 4M×3N / 4M×N, M×4N / 3M×4N, and 3M×4N / M×4N.

[0137] If the partition ranges of the quadtree and binary tree overlap (i.e., the range where both quadtree and binary tree partitioning are possible at the current stage), and the current block (the state before partitioning) is a block obtained by quadtree partitioning (a block obtained by quadtree partitioning from the parent block <when ​​the partition depth is 1 less than the current block>), then the partition information is structured by distinguishing between the following cases. That is, if blocks that can be handled according to each partition setting can be obtained by multiple partitioning methods, it is possible to generate partition information by classifying them through the following process.

[0138] (1) When quadtree partitioning and binary tree partitioning overlap

[0139] [Table 1]

[0140] In the table above, 'a' represents a flag indicating whether or not a quadtree split is performed. If it is 1, a quadtree split (QT) is performed. If the above flag is 0, check flag 'b', which indicates whether or not a binary tree split is performed. If 'b' is 0, no further splitting is performed in that block (No Split); if it is 1, a binary tree split is performed.

[0141] c is a flag indicating the division direction; 0 means horizontal division (hor), and 1 means vertical division (ver). d is a flag indicating the division shape; 0 means symmetric division (SBT; Symmetric Binary Tree), and 1 means asymmetric division (ABT; Asymmetric Binary Tree). Only when d is 1 is information regarding the subdivision ratio in asymmetric division (1 / 4 or 3 / 4) checked. When it is 0, the left and upper blocks have a 1 / 4 ratio, and the right and lower blocks have a 3 / 4 ratio in the left / right or upper / lower blocks; when it is 1, the opposite ratio is used.

[0142] (2) When only binary tree splitting is possible In the table above, the partitioning information can be represented by the flags b through e, excluding a.

[0143] In Figure 5, in the case of block A7, a quadtree partition was possible in blocks A7 to A11 before the partition (i.e., a binary tree partition was possible, but a binary tree partition was performed instead of a quadtree partition), which corresponds to the case where partition information in (1) is generated.

[0144] In contrast, in the case of A8-A11, if a quadtree split has not already been performed in the block A8-A11 before the split, and a binary tree split has been performed (i.e., if the block<A8~A11> In this case, quadtree partitioning is impossible, which corresponds to the case where partitioning information is generated in (2).

[0145] (Number 3. Bottom left block. A12~A15) This example supports multi-tree partitioning (quadrutree, binary tree, and ternary tree in this example), where the size and shape of obtainable blocks are determined by multiple block partitioning settings. In this example, for binary / ternary trees, it is assumed that the maximum encoded block is 64x64, the minimum encoded block has a length of 4 on one side, and the maximum partitioning depth is 4.

[0146] In this example, since there are two or more blocks that can be obtained as a result of the division (2, 3, or 4 in this example), the division information required for a single division operation is a flag indicating whether or not to divide, a flag indicating the type of division, and a flag indicating the direction of division. The obtainable candidates are 4M×4N, 4M×2N, 2M×4N, 4M×N / 4M×2N / 4M×N, M×4N / 2M×4N / M×4N.

[0147] If the partition ranges of a quadtree and a binary / ternary tree overlap, and the current block is a block obtained through quadtree partitioning, the partition information is structured in the following ways to distinguish between them.

[0148] (1) When quadtree partitioning and binary / ternary tree partitioning overlap

[0149] [Table 2]

[0150] In the table above, 'a' is a flag indicating whether or not a quadtree split is performed; if it is 1, a quadtree split is performed. If the flag is 0, check 'b', which is a flag indicating whether or not a binary or ternary split is performed. If 'b' is 0, no further splitting is performed in that block; if it is 1, a binary or ternary split is performed.

[0151] c is a flag indicating the partitioning direction; 0 means horizontal partitioning, and 1 means vertical partitioning. d is a flag indicating the partitioning type; 0 means binary tree partitioning (BT), and 1 means ternary tree partitioning (TT).

[0152] (2) When only binary / ternary tree splitting is possible In the table above, the partitioning information can be represented by the flags b through d, excluding a.

[0153] In Figure 5, in the case of blocks A12 and A15, since quadtree partitioning was possible in blocks A12 to A15 before partitioning, this corresponds to the case where partitioning information in (1) is generated.

[0154] In contrast, A13 and A14 are cases where the blocks A13 and A14 before the split were not split into quadtrees but into ternary trees, which corresponds to the case where the split information in (2) is generated.

[0155] (Number 4. Bottom left block. A16~A20) This example supports multi-tree partitioning (quadrutree, binary tree, and ternary tree in this example), where the size and shape of obtainable blocks are determined by multiple block partitioning settings. In this example, for binary / ternary trees, it is assumed that the maximum encoded block is 64x64, the minimum encoded block has a length of 4 on one side, and the maximum partitioning depth is 4.

[0156] In this example, the number of blocks obtainable through the division is two or more (2, 3, or 4 in this example). Therefore, the division information required for a single division operation is a flag indicating whether or not a division is performed, a flag indicating the type of division, a flag indicating the shape of the division, and a flag indicating the direction of the division. The obtainable candidates are 4M×4N, 4M×2N, 2M×4N, 4M×N / 4M×3N, 4M×3N / 4M×N, M×4N / 3M×4N, 3M×4N / M×4N, 4M×N / 4M×2N / 4M×N, and M×4N / 2M×4N / M×4N.

[0157] If the partition ranges of a quadtree and a binary / ternary tree overlap, and the current block is a block obtained through quadtree partitioning, the partition information is structured in the following ways to distinguish between them.

[0158] (1) When quadtree partitioning and binary / ternary tree partitioning overlap

[0159] [Table 3]

[0160] In the table above, 'a' represents a flag indicating whether or not a quadtree split is performed; if it is 1, a quadtree split is performed. If the flag is 0, check 'b', which is a flag indicating whether or not a binary tree split is performed. If 'b' is 0, no further splitting is performed in that block; if it is 1, a binary or ternary tree split is performed.

[0161] c is a flag indicating the partitioning direction; 0 means horizontal partitioning, and 1 means vertical partitioning. d is a flag indicating the partitioning type; 0 means ternary tree partitioning, and 1 means binary tree partitioning. When d is 1, check e, a flag for the partitioning shape; if e is 0, perform symmetric partitioning, and if e is 1, perform asymmetric partitioning. When e is 1, check information regarding the subdivision ratio in asymmetric partitioning, which is the same as in the previous example.

[0162] (2) When only binary / ternary tree splitting is possible In the table above, the partitioning information can be represented by the flags b through f, excluding a.

[0163] In Figure 5, block A20 corresponds to the case where quadtree partitioning is possible in blocks A16 to A19 before partitioning, and therefore the partitioning information in (1) is generated.

[0164] In contrast, in the case of A16-A19, a quadtree partitioning has not been performed in the blocks A16-A19 before the partitioning, but a binary tree partitioning has been performed, which corresponds to the case in (2) where partitioning information is generated.

[0165] The following describes an interpretation method based on merge mode. In merge mode, motion information may be derived in units of one block. Here, a block may mean an encoded block determined by the block partitioning described above. Alternatively, a block may mean an encoded block of a leaf node that is not further divided into smaller encoded blocks. In the merge mode, motion information may be derived in units of subblocks belonging to one block. For this purpose, one block may be further divided into multiple subblocks. For example, affine motion modeling may be performed based on the subblock unit merge mode. Currently, a block may selectively use either a block unit merge mode or a subblock unit merge mode, and a predetermined flag may be used for this purpose. Here, the flag may be information indicating whether or not the subblock unit merge mode is applied. The flag may be encoded and signaled in an encoding device based on predetermined encoding parameters, or it may be derived from a decoding device. The encoding parameters may mean the size / shape of the block, component type, partition type, partition depth, etc. For example, if the current block size is greater than or equal to N × M, the flag may be signaled; otherwise, the flag may be derived as a previously defined value. Here, N and M may be integers of 8, 16, or more. N and M may be the same or different from each other. The block-level merge mode is explained with reference to Figure 6, and the sub-block-level merge modes are explained with reference to Figures 7 to 10.

[0166] Figure 6 shows an embodiment to which the present invention is applied, which is a method for performing inter prediction based on block-level merge modes.

[0167] Referring to Figure 6, the current list of merge candidates for the block is constructed (S600).

[0168] The merge candidate list may include at least one of the spatial or temporal merge candidates for the current block.

[0169] The movement information of a spatial merge candidate may be derived from the movement information of the spatially adjacent blocks of the current block. Here, a spatially adjacent block means a block that belongs to the same picture as the current block and is adjacent to the current block. A spatially adjacent block may include blocks adjacent to at least one of the left, top, top-right, bottom-left, or top-left edges of the current block. The top-left adjacent block may be used only if at least one of the blocks adjacent to the left, top, top-right, and bottom-left edges is not available (usable).

[0170] The motion information of a temporal merge candidate may be derived from the motion information of the temporally adjacent block of the current block. A temporally adjacent block is defined as a block belonging to a different picture than the current block and located at the same position as the current block. Here, a block at the same position may mean at least one of the following: a block adjacent to the lower right corner of the current block (BR), a block containing the position of the central sample of the current block (CTR), or a block containing the position of the upper left sample of the current block (TL). Alternatively, a block at the same position may mean a block located at a position shifted by an amount corresponding to a predetermined disparity vector from the position of the upper left sample of the current block. Here, the disparity vector may be determined based on any one of the motion vectors of the spatially adjacent blocks described above. Alternatively, the disparity vector may be determined based on at least two combinations of the spatially adjacent block motion vectors described above. The combination may mean calculations such as maximum value, minimum value, median value, or weighted average value. For example, the disparity vector may be set to the motion vector of a block adjacent to the left or an adjacent block at the top. Alternatively, the parallax vector may be set to an intermediate or average value between the motion vector of the block adjacent to the left and the motion vector of the block adjacent to the lower left corner.

[0171] The motion vector and reference picture index of the temporal merge candidate may be derived from the motion vector and reference picture index of the temporal adjacent block described above, respectively. Alternatively, the motion vector of the temporal merge candidate may be derived as the motion vector of the temporal adjacent block, and the reference picture index of the temporal merge candidate may be set to a default already promised to the decoder (e.g., 0), independently of the temporal adjacent block.

[0172] The merge candidate list may further contain combination merge candidates. Combination merge candidates may be derived by combining n merge candidates that already belong to the generated merge candidate list.

[0173] Here, n may be an integer of 2, 3, 4 or more. The number of merge candidates to be combined (n) may be a fixed value already promised to the encoding / decoding device, or it may be encoded and signaled in the encoding device. The signaling may be performed in at least one unit of a sequence, picture, slice, tile, subtile (brick), or given block. The number of merge candidates to be combined (n) may be determined variably based on the number of remaining merge candidates, where the number of remaining merge candidates is the difference between the maximum number of merge candidates that can be included in the merge candidate list and the current number of merge candidates belonging to the merge candidate list. The maximum number may be a number already promised to the encoding / decoding device, or it may be encoded and signaled in the encoding device. The current number is the number of merge candidates configured up to the point before adding the combined merge candidates. For example, if the number of remaining merge candidates is 1, two merge candidates may be used, and if the number of remaining merge candidates is greater than 1, three or more merge candidates may be used.

[0174] The positions of the n merge candidates may be positions already determined in the merge candidate list. For example, each merge candidate belonging to the merge candidate list may be assigned an index (0 to (k-1)). Here, k may represent the total number of merge candidates belonging to the merge candidate list. In this case, the positions of the n merge candidates may correspond to indices 0 to (n-1) in the merge candidate list. Alternatively, the n merge candidates may be determined by considering the prediction direction of each merge candidate belonging to the merge candidate list. For example, only merge candidates with bidirectional predictions may be selectively used from among the merge candidates belonging to the merge candidate list, or only merge candidates with unidirectional predictions may be selectively used.

[0175] Combination merge candidates may be derived using both spatial and temporal merge candidates, or they may be derived using only one of either spatial or temporal merge candidates. For example, combination merge candidates may be restricted to being derived using only spatial merge candidates. In this case, the number of merge candidates to be combined may be restricted to the number of spatial merge candidates already belonging to the generated merge candidate list.

[0176] The aforementioned combination merge candidates may be added after the spatial / temporal merge candidates in the merge candidate list. That is, the index of the combination merge candidate may be greater than the index of the spatial / temporal merge candidate. Alternatively, the combination merge candidate may be added between the spatial merge candidate and the temporal merge candidate in the merge candidate list. That is, the index of the combination merge candidate may be greater than the index of the spatial merge candidate and less than the index of the temporal merge candidate. Alternatively, the position of the combination merge candidate may be determined variably, taking into account the prediction direction of the combination merge candidate. Depending on whether the prediction direction of the combination merge candidate is bidirectional, the position of the combination merge candidate in the merge candidate list may be sorted. For example, if the combination merge candidate is bidirectional, it may be assigned a smaller index than the spatial or temporal merge candidate; otherwise, it may be assigned a larger index than the spatial or temporal merge candidate.

[0177] For the sake of clarity, the following describes how to derive a combined merge candidate based on two merge candidates.

[0178] The motion information of the combined merge candidate may be derived by taking a weighted average of the motion information of the first merge candidate and the second merge candidate. Here, the weights of the weighted average are, but are not limited to, [1:1], [1:2], [1:3], [2:3], etc. The weights may be those already defined in the encoding / decoding device or may be derived from the decoding device. In this case, the weights may be derived by considering at least one of the distance between the current picture and the reference picture of the merge candidate or the predicted direction of the merge candidate. Alternatively, the motion information of the combined merge candidate may be derived by obtaining motion information in the L0 direction from the first merge candidate and motion information in the L1 direction from the second merge candidate, and combining these. The motion information of the combined merge candidate may also be derived based on at least one of the derivation methods described above, and this may be done by considering the predicted direction of the merge candidate to be combined, as will be described later.

[0179] In this specification, motion information may include at least one of a prediction direction flag, a reference picture index, or a motion vector. The motion information may be defined for L0 prediction and L1 prediction, respectively. Here, L0 prediction means a prediction that refers to reference picture list L0, and L1 prediction means a prediction that refers to reference picture list L1.

[0180] 1. When both the first merge candidate and the second merge candidate are unidirectional predictions. (CASE 1) If the first merge candidate is predicted to be L0 and the second merge candidate is predicted to be L1, the reference picture index in the L0 direction of the combined merge candidate is derived as the reference picture index of the first merge candidate. The flag for the predicted direction in the L0 direction of the combined merge candidate is derived as 1. The motion vector in the L0 direction of the combined merge candidate is derived as the motion vector of the first merge candidate. The reference picture index in the L1 direction of the combined merge candidate is derived as the reference picture index of the second merge candidate. The flag for the predicted direction in the L1 direction of the combined merge candidate is derived as 1. The motion vector in the L1 direction of the combined merge candidate is derived as the motion vector of the second merge candidate.

[0181] (CASE 2) If the first merge candidate is predicted in L1 and the second merge candidate is predicted in L0, the reference picture index in the L0 direction of the combined merge candidate is derived as the reference picture index of the second merge candidate. The flag for the predicted direction in the L0 direction of the combined merge candidate is derived as 1. The motion vector in the L0 direction of the combined merge candidate is derived as the motion vector of the second merge candidate. The reference picture index in the L1 direction of the combined merge candidate is derived as the reference picture index of the first merge candidate. The flag for the predicted direction in the L1 direction of the combined merge candidate is derived as 1. The motion vector in the L1 direction of the combined merge candidate is derived as the motion vector of the first merge candidate.

[0182] (CASE 3) If the first and second merge candidates are L0 predictions, the L0 direction reference picture index of the combined merge candidate is derived as the reference picture index of either the first or second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first and second merge candidates is set as the L0 direction reference picture index of the combined merge candidate. The L0 direction prediction direction flag of the combined merge candidate is derived as 1. The L0 direction motion vector of the combined merge candidate is derived as a weighted average of the motion vectors of the first and second merge candidates. The L1 direction reference picture index of the combined merge candidate is derived as -1, the L1 direction prediction direction flag is derived as 0, and the L1 direction motion information is derived as 0.

[0183] (CASE 4) If the first and second merge candidates are L1 predictions, the reference picture index in the L0 direction of the combined merge candidate is derived as -1, the flag for the predicted direction in the L0 direction is derived as 0, and the motion information in the L0 direction is derived as 0. The reference picture index in the L1 direction of the combined merge candidate is derived as the reference picture index of either the first or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first and second merge candidates is set as the reference picture index in the L1 direction of the combined merge candidate. The flag for the predicted direction in the L1 direction of the combined merge candidate is derived as 1. The motion vector in the L1 direction of the combined merge candidate is derived as a weighted average of the motion vector of the first merge candidate and the second merge candidate.

[0184] 2. When both the first and second merge candidates are bidirectional predictions. (CASE 5) The L0 direction reference picture index of the combined merge candidate is derived as the reference picture index of either the first or second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first and second merge candidates is set as the L0 direction reference picture index of the combined merge candidate. The L0 direction prediction direction flag of the combined merge candidate is derived as 1. The L0 direction motion vector of the combined merge candidate is derived as a weighted average of the motion vectors of the first and second merge candidates. The L1 direction reference picture index of the combined merge candidate is derived as the reference picture index of either the first or second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first and second merge candidates is set as the L1 direction reference picture index of the combined merge candidate. The L1 direction prediction direction flag of the combined merge candidate is derived as 1. The L1 direction motion vector of the combined merge candidate is derived as a weighted average of the motion vectors of the first merge candidate and the second merge candidate.

[0185] 3. When the first merge candidate is a bidirectional prediction and the second merge candidate is a unidirectional prediction. (CASE 6) If the second merge candidate is an L0 prediction, the L0 direction reference picture index of the combined merge candidate is derived as the reference picture index of either the first or second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first and second merge candidates is set as the L0 direction reference picture index of the combined merge candidate. The prediction direction flag for the L0 direction of the combined merge candidate is derived as 1. The L0 direction motion vector of the combined merge candidate is derived as a weighted average of the motion vectors of the first and second merge candidates. The L1 direction reference picture index of the combined merge candidate is derived as the reference picture index of the first merge candidate. The prediction direction flag for the L1 direction of the combined merge candidate is derived as 1. The L1 direction motion vector of the combined merge candidate is derived as the motion vector of the first merge candidate.

[0186] (CASE 7) If the second merge candidate is an L1 prediction, the L0 direction reference picture index of the combined merge candidate is derived as the reference picture index of the first merge candidate. The prediction direction flag for the L0 direction of the combined merge candidate is derived as 1. The L0 direction motion vector of the combined merge candidate is derived as the motion vector of the first merge candidate. The L1 direction reference picture index of the combined merge candidate is derived as the reference picture index of either the first or second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first and second merge candidates is set as the L1 direction reference picture index of the combined merge candidate. The prediction direction flag for the L1 direction of the combined merge candidate is derived as 1. The L1 direction motion vector of the combined merge candidate is derived as a weighted average of the motion vectors of the first and second merge candidates.

[0187] Referring to Figure 6, the movement information of the current block is derived from the merge candidate list (S610).

[0188] Specifically, the merge index of the current block may be signaled. The merge index may identify one of several merge candidates belonging to the merge candidate list. That is, a merge candidate with the same index as the merge index may be extracted, and the movement information of the current block may be derived using the movement information of the extracted merge candidate.

[0189] Referring to Figure 6, the derived motion information is used to predict the inter-block of the current block (S620).

[0190] Specifically, a reference picture may be selected from a list of reference pictures based on the reference picture index of the current block. A reference block within the reference picture may be identified based on the motion vector of the current block. Predicted pixels of the current block may be generated using the restored pixels of the identified reference block. The current block may be restored by adding the predicted pixels and residual pixels of the current block. Here, the residual pixels may be derived by performing at least one of inverse quantization or inverse transformation on residual coefficients signaled by the bitstream.

[0191] Figure 7 shows an embodiment to which the present invention is applied, illustrating an interpretation method based on an affine model.

[0192] Referring to Figure 7, a candidate list is generated for predicting the movement information of the current block (S700).

[0193] The candidate list may include one or more candidates based on affine models (hereinafter referred to as affine candidates). An affine candidate is a candidate with a control point vector. The control point vector is a motion vector of a control point for an affine model and may be defined for the position of a corner of a block (for example, at least one position of the upper left, upper right, lower left, or lower right corner).

[0194] An affine candidate may include at least one of a spatial candidate, a temporal candidate, or a constructed candidate. Here, the spatial candidate may be derived from a vector of an adjacent block that is spatially adjacent to the current block, and the temporal candidate may be derived from a vector of an adjacent block that is temporally adjacent to the current block. Here, the adjacent block means a block encoded in an affine model. The vector may mean a motion vector or a control point vector.

[0195] The method for deriving spatial / temporal candidates based on vectors of spatially / temporal adjacent blocks will be described in detail with reference to Figure 8.

[0196] On the other hand, the configured candidates may also be derived based on the combination of motion vectors between spatially / temporally adjacent blocks to the current block, which will be described in detail with reference to Figure 9.

[0197] The aforementioned affine candidates may be arranged in the candidate list based on a predetermined priority. For example, the affine candidates may be arranged in the candidate list in the order of spatial candidates, temporal candidates, and configured candidates. Alternatively, the affine candidates may be arranged in the candidate list in the order of temporal candidates, spatial candidates, and configured candidates. However, this is not limited to the above, and temporal candidates may be arranged after configured candidates. Alternatively, some of the configured candidates may be arranged before the spatial candidates, and the rest may be arranged after the spatial candidates.

[0198] The candidate list may further include temporal candidates based on subblocks. The motion information of the temporal candidates may be derived from the collocated block corresponding to the current block, which will be described in detail with reference to Figure 10.

[0199] Based on the aforementioned candidate list and candidate index, the control point vector of the current block is derived (S710).

[0200] The candidate index refers to an index encoded for deriving the control point vector of the current block. The candidate index may identify one of several affine candidates belonging to the candidate list. The control point vector of the current block may be derived using the control point vector of the affine candidate identified by the candidate index.

[0201] For example, suppose the current block's affine model type is 4 parameters (i.e., it is determined that the current block uses 2 control point vectors). In this case, if the affine candidate identified by the candidate index has 3 control point vectors, only 2 of the 3 control point vectors (for example, the control point vector Idx=0,1) may be selected and set as the control point vectors for the current block. Alternatively, all 3 control point vectors of the identified affine candidate may be set as the control point vectors for the current block. In this case, the current block's affine model type may be updated to 6 parameters.

[0202] Conversely, assume that the type of the affine model of the current block is six parameters (i.e., when it is determined that the current block uses three control point vectors). At this time, if the affine candidate specified by the candidate index has two control point vectors, one additional control point vector may be generated, and the two control point vectors of the affine candidate and the additional control point vector may be set as the control point vectors of the current block. The additional control point vector may be derived based on at least one of the two control point vectors of the affine candidate, the size or position information of the current / adjacent block.

[0203] Alternatively, the two control point vectors of the specified affine candidate may be set as the control point vectors of the current block. In this case, the type of the affine model of the current block may be updated to four parameters.

[0204] Based on the control point vectors of the current block, the motion vector of the current block is derived (S720).

[0205] The motion vector may be derived in units of sub-blocks of the current block. For this purpose, the current block may be divided into a plurality of N×M sub-blocks. Here, the N×M sub-block may have a rectangular (N>M or N<M) or square (N = M) shape. The values of N and M may be 4, 8, 16, 32 or more.

[0206] For example, the size / shape of the sub-block may be a fixed size / shape already defined in the decoding device. For example, the size / shape of the sub-block may be a square such as 4×4, 8×8, 16×16, etc., or a rectangle such as 2×4, 2×8, 4×8, 4×16, etc. Alternatively, the sub-block may be defined as a block whose sum of width and height is an integer of 8, 12, 16 or more. Alternatively, the sub-block may be defined as a block whose product of width and height is an integer of 16, 32, 64 or more.

[0207] Alternatively, the size / shape of the subblock may be derived variably based on the block attributes described above. For example, if the current block size is greater than or equal to a predetermined threshold size, the current block may be divided into a first subblock unit (e.g., 8x8, 16x16); otherwise, the current block may be divided into a second subblock unit (e.g., 4x4).

[0208] Alternatively, information regarding the size / shape of a subblock may be encoded and signaled in an encoding device. The information indicates at least one of the size or shape of the subblock, and this may be signaled at at least one level of sequence, picture, tile group, tile, or CTU.

[0209] Currently, the control point representing the block may include the position of the upper left corner and the position of the upper right corner. However, it is not limited to this, and the control point may include three locations such as the position of the upper left corner, the position of the upper right corner, and the position of the lower left corner, or it may include even more locations.

[0210] When two control points are used, the motion vector for each subblock of the current block may be derived using the first control point vector corresponding to the first control point, the second control point vector corresponding to the second control point, and at least one of the subblock's position (x,y) or the current block's size (width or height).

[0211] When three control points are used, the motion vector for each subblock of the current block may be derived using the first control point vector corresponding to the first control point, the second control point vector corresponding to the second control point, the third control point vector corresponding to the third control point, and at least one of the subblock's position (x,y) or the current block's size (width or height). In this case, the difference vector between the second control point vector and the first control point vector may be used, or the difference vector between the third control point vector and the first control point vector may be used. The difference vector may be calculated for both the horizontal direction (x-axis direction) and the vertical direction (y-axis direction).

[0212] The motion vector derivation process may further include a process of applying a predetermined offset to the motion vector derived based on the control point vector. The offset means a vector for improving the already derived motion vector. The offset may be determined based on information about at least one of the size or direction of the offset. The size (Absolute) may be an integer of 1, 2, 3, or more. The direction may include at least one of the following directions: left, right, upper, or lower. The information regarding the size and / or direction of the offset may be encoded and signaled in the encoding device. Alternatively, the size of the offset may be a fixed value already set in the decoding device.

[0213] Using the derived motion vector, inter-prediction is performed for the current block (S730).

[0214] Specifically, the referenced block may be identified using the motion vector of the current block. The referenced block may be identified separately for each subblock of the current block. The referenced block of each subblock may belong to a single referenced picture. That is, subblocks belonging to the current block may share a single referenced picture. Alternatively, a referenced picture index may be set independently for each subblock of the current block.

[0215] On the other hand, the derived motion vector may include at least one of the L0 motion vector or the L1 motion vector. If the motion vector includes both the L0 and L1 motion vectors, the decoding device may set either the L0 or L1 motion vector to 0 to perform unidirectional prediction.

[0216] The above settings may be made selectively, taking into consideration at least one of the block size or the prediction mode. Here, a block means the current block or a subblock of the current block. The prediction mode means a skip mode, a merge mode, an AMVP mode, an affine model-based prediction mode, a current picture reference mode, and the like.

[0217] For example, if the size of the current block (or a subblock of the current block) is smaller than or equal to a predetermined threshold size, the L1 motion vector may be set to 0 to perform a unidirectional prediction; otherwise, the L0 and L1 motion vectors may be used to perform a bidirectional prediction. Here, the threshold size may be defined as the size of a block where at least one of the width and height is 4, 8, or 16.

[0218] Alternatively, if the current block is encoded in the first prediction mode, the L1 motion vector may be set to 0 to perform unidirectional prediction; otherwise, the L0 and L1 motion vectors may be used to perform bidirectional prediction.

[0219] The identified reference block may be set as the predicted block of the current block. The current block may be restored by adding the residual block to the predicted block.

[0220] Figure 8 shows an embodiment to which the present invention is applied, illustrating a method for deriving affine candidates from control point vectors of spatially / temporal adjacent blocks.

[0221] The current block 800 has a width and height of cbW and cbH, respectively, and its position is (xCb, yCb). The spatially adjacent blocks 810-850 have widths and heights of nbW and nbH, respectively, and their positions are (xNb, yNb). The spatially adjacent blocks may include at least one of the blocks to the left of the current block 810, the lower left end block 840, the upper right end block 830, the upper end block 820, or the upper left end block 850. Alternatively, the spatially adjacent blocks may further include at least one of the blocks adjacent to the right of the upper left end block 850 or the blocks adjacent to the lower end.

[0222] A spatial candidate may have n control point vectors (cpMVs), where n is an integer of 1, 2, 3, or more. The value of n may be determined based on at least one of the following: information about whether or not it can be decoded in subblock units, information about whether or not it is a block encoded in an affine model, or information about the type of affine model (4-parameter or 6-parameter).

[0223] For example, if, according to the information described above, the block is decoded in subblock units or is a block encoded with an affine model, the block may have two control point vectors. In contrast, if this is not the case, the block does not need to perform predictions based on an affine model.

[0224] Alternatively, if, according to the information above, the block is encoded with an affine model and the type of the affine model is 6 parameters, the block may have three control point vectors. In contrast, if not, the block does not need to make predictions based on the affine model.

[0225] The aforementioned information may be encoded and signaled in the encoding device. Alternatively, all or part of the aforementioned information may be derived from the decoding device based on the attributes of a block. Here, a block may mean the current block, or it may mean the spatially / temporal adjacent blocks of the current block. The attributes may mean parameters such as size, shape, position, partition type, intermode, and residual coefficient. The intermode may be a mode already defined in the decoding device, such as merge mode, skip mode, AMVP mode, affine model, intra / inter combination mode, current picture reference mode, etc. Alternatively, the n value may be derived from the decoding device based on the aforementioned attributes of a block.

[0226] In this embodiment, the n control point vectors may be represented as a first control point vector (cpMV[0]), a second control point vector (cpMV[1]), a third control point vector (cpMV[2]), and an nth control point vector (cpMV[n-1]).

[0227] For example, the first control point vector (cpMV[0]), the second control point vector (cpMV[1]), the third control point vector (cpMV[2]), and the fourth control point vector (cpMV[3]) may be vectors corresponding to the positions of the top-left, top-right, bottom-left, and bottom-right samples of the block, respectively. Here, it is assumed that the spatial candidate has three control point vectors, and these three control point vectors may be any control point vectors selected from the first to the nth control point vectors. However, it is not limited to this, and the spatial candidate may have two control point vectors, and these two control point vectors may be any control point vectors selected from the first to the nth control point vectors.

[0228] The following describes a method for deriving control point vectors for spatial candidates.

[0229] 1. If the current block boundary does not touch the CTU boundary The first control point vector may be derived based on at least one of the following: the first control point vector of a spatially adjacent block, a predetermined difference value, the current block's position information (xCb, yCb), or the spatially adjacent block's position information (xNb, yNb).

[0230] The number of difference values ​​may be one, two, three, or more. The number of difference values ​​may be determined variably considering the attributes of the block described above, or it may be a fixed value already promised to the decoding device. The difference value may be defined as the difference between any one of a plurality of control point vectors and another one. For example, the difference value may include at least one of the following: a first difference between a second control point vector and a first control point vector, a second difference between a third control point vector and a first control point vector, a third difference between a fourth control point vector and a third control point vector, or a fourth difference between a fourth control point vector and a second control point vector.

[0231] For example, the first control point vector may be derived as shown in Equation 1 below.

[0232]

number

[0233] In Equation 1, the variables mvScaleHor and mvScaleVer may represent the first control point vector of a spatially adjacent block, or they may represent the value derived by applying a shift operation corresponding to k to the first control point vector, where k may be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9 or more. The variables dHorX and dVerX correspond to the x and y components, respectively, of the first difference value between the second control point vector and the first control point vector. The variables dHorY and dVerY correspond to the x and y components, respectively, of the second difference value between the third control point vector and the first control point vector. The variables described above may also be derived as shown in Equation 2 below.

[0234]

number

[0235] The second control point vector may be derived based on at least one of the following: the first control point vector of a spatially adjacent block, a predetermined difference value, the current block's position information (xCb, yCb), the block size (width or height), or the position information of a spatially adjacent block (xNb, yNb). Here, the block size refers to the size of the current block and / or the spatially adjacent block. The difference value is as described in relation to the first control point vector, so a detailed explanation of it is omitted here. However, the range and / or number of difference values ​​used in the derivation process of the second control point vector may differ from those of the first control point vector.

[0236] For example, the second control point vector may be derived as shown in equation 3 below.

[0237]

number

[0238] In equation 3, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in equation 1, and a detailed explanation of them is omitted here.

[0239] The third control point vector may be derived based on at least one of the following: the first control point vector of a spatially adjacent block, a predetermined difference value, the current block's position information (xCb, yCb), the block size (width or height), or the spatially adjacent block's position information (xNb, yNb). Here, the block size refers to the size of the current block and / or the spatially adjacent block. The difference value is as described in relation to the first control point vector, so a detailed explanation of it is omitted here. However, the range and / or number of difference values ​​used in the derivation process of the third control point vector may differ from those of the first or second control point vector.

[0240] For example, the third control point vector may be derived as shown in equation 4 below.

[0241]

number

[0242] In Equation 4, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 1, and a detailed explanation of them is omitted here. On the other hand, the nth control point vector of the spatial candidate may be derived through the process described above.

[0243] 2. When the current block boundary is adjacent to the CTU boundary. The first control point vector may be derived based on at least one of the following: the motion vector (MV) of a spatially adjacent block, a predetermined difference value, the current block's position information (xCb, yCb), or the position information of a spatially adjacent block (xNb, yNb).

[0244] The motion vector may be the motion vector of a subblock located at the bottommost end of a spatially adjacent block. The subblock may be the leftmost, centered, or rightmost of a plurality of subblocks located at the bottommost end of a spatially adjacent block. Alternatively, the motion vector may represent the average, maximum, or minimum value of the subblock's motion vector.

[0245] The number of difference values ​​may be one, two, three, or more. The number of difference values ​​may be determined variably considering the attributes of the block described above, or it may be a fixed value already promised to the decoding device. The difference value may be defined as the difference between one of several motion vectors stored in units of subblocks in a spatially adjacent block and another. For example, the difference value may mean the difference between the motion vector of the lower right subblock and the motion vector of the lower left subblock of a spatially adjacent block.

[0246] For example, the first control point vector may be derived as shown in equation 5 below.

[0247]

number

[0248] In equation 5, the variables mvScaleHor and mvScaleVer may represent the motion vector (MV) of the spatially adjacent blocks mentioned above, or the value derived by applying a shift operation corresponding to k to the motion vector. Here, k may be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9 or more.

[0249] The variables dHorX and dVerX correspond to the x and y components, respectively, of a predetermined difference value. Here, the difference value means the difference between the motion vector of the lower right subblock and the motion vector of the lower left subblock within the spatially adjacent block. The variables dHorY and dVerY may be derived based on the variables dHorX and dVerX. The variables described above may also be derived as shown in the following equation 6.

[0250]

number

[0251] The second control point vector may be derived based on at least one of the following: the motion vector (MV) of a spatially adjacent block, a predetermined difference value, the current block's position information (xCb, yCb), the block size (width or height), or the position information (xNb, yNb) of a spatially adjacent block. Here, the block size refers to the size of the current block and / or the spatially adjacent block. The motion vector and difference value are as described in the first control point vector, so a detailed explanation of them is omitted here. However, the position of the motion vector, the range of the difference value, and / or the number used in the derivation process of the second control point vector may differ from those of the first control point vector.

[0252] For example, the second control point vector may be derived as shown in equation 7 below.

[0253]

number

[0254] In equation 7, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in equation 5, and a detailed explanation of them is omitted here.

[0255] The third control point vector may be derived based on at least one of the following: the motion vector (MV) of a spatially adjacent block, a predetermined difference value, the current block's position information (xCb, yCb), the block size (width or height), or the position information (xNb, yNb) of a spatially adjacent block. Here, the block size refers to the size of the current block and / or the spatially adjacent block. The motion vector and difference value are as described in the first control point vector, so a detailed explanation of them is omitted here. However, the position of the motion vector, the range of the difference value, and / or the number used in the derivation process of the third control point vector may differ from those of the first or second control point vector.

[0256] For example, the third control point vector may be derived as shown in equation 8 below.

[0257]

number

[0258] In equation 8, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in equation 5, and a detailed explanation of them is omitted here. On the other hand, the nth control point vector of the spatial candidate may be derived by the process described above.

[0259] The aforementioned process for deriving affine candidates may be performed for each of the already defined spatially adjacent blocks. The already defined spatially adjacent blocks may include at least one of the blocks to the left of the current block, the lower left block, the upper right block, the top block, or the upper left block.

[0260] Alternatively, the process for deriving the affine candidate may be carried out separately for each group of spatially adjacent blocks. Here, the spatially adjacent blocks may be classified into a first group including the left block and the lower left end block, and a second group including the upper right end block, the top end block, and the upper left end block.

[0261] For example, one affine candidate may be derived from spatially adjacent blocks belonging to the first group. This derivation may be carried out based on a predetermined priority until a usable affine candidate is found. The priority may be in the order of leftmost blocks → bottom leftmost blocks, or in the reverse order. According to the priority, it may be determined whether the spatially adjacent block in the first group is a block that has been decoded by a prediction based on the affine model, and the block that has been decoded by a prediction based on the affine model first may be selected as the affine candidate.

[0262] Similarly, an affine candidate may be derived from spatially adjacent blocks belonging to a second group. This derivation may be carried out based on a predetermined priority until a usable affine candidate is found. The priority may be in the order of upper-right end block → top end block → upper-left end block, or in the reverse order. According to the priority, it may be determined whether the spatially adjacent block in the second group is a block that has been decoded by a prediction based on the affine model, and the first block that has been decoded by a prediction based on the affine model may be selected as an affine candidate.

[0263] The embodiments described above may also be applied similarly to temporally adjacent blocks. Here, a temporally adjacent block may be a block that belongs to a different picture than the current block but is in the same position as the current block. A block in the same position may be a block that includes the position of the top-left sample of the current block, the center position, or the position of a sample adjacent to the bottom-right sample of the current block.

[0264] Alternatively, the temporally adjacent block means a block at a position shifted by an amount corresponding to a predetermined parallax vector from the block at the same position. Here, the parallax vector may be determined based on the motion vector of any one of the spatially adjacent blocks of the current block described above.

[0265] FIG. 9 is a diagram showing a method for deriving candidates configured based on a combination of motion vectors of spatially / temporally adjacent blocks in one embodiment to which the present invention is applied.

[0266] The configured candidate of the present invention may be derived based on at least two combinations of control point vectors (hereinafter referred to as control point vectors (cpMVCorner[n])) corresponding to each corner of the current block. Here, n may be 0, 1, 2, or 3.

[0267] The control point vector (cpMVCorner[n]) may be derived based on the motion vectors of spatially adjacent blocks and / or temporally adjacent blocks. Here, the spatially adjacent blocks may include at least one of the first adjacent blocks A, B, or C adjacent to the upper left sample of the current block, the second adjacent blocks D or E adjacent to the upper right sample of the current block, or the third adjacent blocks F or G adjacent to the lower left sample of the current block. The temporally adjacent block is a block belonging to a picture different from the current block and means a block at the same position as the current block (hereinafter referred to as the fourth adjacent block (Col)). Here, the fourth adjacent block may mean a block H, I, or J including the position of the upper left sample, the upper right sample, or the lower left sample of the current block, or may mean an adjacent block at the position of the lower right sample of the current block.

[0268] The first adjacent block may mean the adjacent block at the top left A, top B, or left C of the current block. According to a predetermined priority order, it may be determined whether the motion vectors of adjacent blocks A, B, and C are available, and the control point vector may be determined using the motion vectors of the available adjacent blocks. This availability determination may be continued until an adjacent block with available motion vectors is found. Here, the priority order may be A→B→C, but is not limited to this; it may also be A→C→B, C→A→B, or B→A→C.

[0269] The second adjacent block may mean the adjacent block at the top edge D or top right edge E of the current block. Similarly, the motion vectors of adjacent blocks D and E may be determined to be available according to a predetermined priority order, and the control point vector may be determined using the motion vectors of the available adjacent blocks. The availability determination may be carried out until an adjacent block with an available motion vector is found. Here, the priority order may be D → E or E → D.

[0270] The third adjacent block may mean the adjacent block to the left F or the lower left G of the current block. Similarly, the motion vector of an adjacent block may be determined to be available according to a predetermined priority order, and the control point vector may be determined using the motion vector of an available adjacent block. The availability determination may be carried out until an adjacent block with an available motion vector is found. Here, the priority order may be G → F or F → G.

[0271] For example, the first control point vector (cpMVCorner[0]) may be set to the motion vector of the first adjacent block, the second control point vector (cpMVCorner[1]) may be set to the motion vector of the second adjacent block, and the third control point vector (cpMVCorner[2]) may be set to the motion vector of the third adjacent block. The fourth control point vector (cpMVCorner[3]) may be set to the motion vector of the fourth adjacent block.

[0272] Alternatively, the first control point vector may be derived using the motion vector of at least one of the first adjacent block or the fourth adjacent block, where the fourth adjacent block may be block H containing the position of the upper left end sample. The second control point vector may be derived using at least one motion vector of the second adjacent block or the fourth adjacent block, where the fourth adjacent block may be block I containing the position of the upper right end sample. The third control point vector may be derived using at least one motion vector of the third adjacent block or the fourth adjacent block, where the fourth adjacent block may be block J containing the position of the lower left end sample.

[0273] Alternatively, any one of the first to fourth control point vectors may be derived based on the other one. For example, the second control point vector may be derived by applying a predetermined offset vector to the first control point vector. The offset vector may be a difference vector between the third control point vector and the first control point vector, and may be derived by applying a predetermined scaling factor to the difference vector. The scaling factor may be determined based on at least one of the width or height of the current block and / or adjacent blocks.

[0274] The K configured candidates (ConstK) according to the present invention may be determined by a combination of at least two of the first to fourth control point vectors described above. The K value may be an integer of 1, 2, 3, 4, 5, 6, 7 or more. The K value may be derived based on information signaled in the encoding device, or it may be a value already promised to the decoding device. The information may include information indicating the maximum number of configured candidates to be included in the candidate list.

[0275] Specifically, the first configured candidate (Const1) may be derived by combining the first to third control point vectors. For example, the first configured candidate (Const1) may have control point vectors as shown in Table 4. On the other hand, the configuration of the control point vectors may be restricted to the case where the reference picture information of the first adjacent block is the same as the reference picture information of the second and third adjacent blocks, as shown in Table 4. Here, the reference picture information may mean the reference picture index that indicates the position of the reference picture in the reference picture list, or it may mean the POC (picture order count) value that indicates the output order.

[0276] [Table 4]

[0277] The second configured candidate (Const2) may be derived by combining the first, second, and fourth control point vectors. For example, the second configured candidate (Const2) may have control point vectors as shown in Table 5. On the other hand, the configuration of the control point vectors may be restricted to the case where the reference picture information of the first adjacent block is the same as the reference picture information of the second and fourth adjacent blocks, as shown in Table 5. Here, the reference picture information is as described above.

[0278] [Table 5]

[0279] The third configured candidate (Const3) may be derived by combining the first, third, and fourth control point vectors. For example, the third configured candidate (Const3) may have control point vectors as shown in Table 6. On the other hand, the configuration of the control point vectors may be restricted to the case where the reference picture information of the first adjacent block is the same as the reference picture information of the third and fourth adjacent blocks, as shown in Table 6. Here, the reference picture information is as described above.

[0280]

Table 6

[0281] The fourth constructed candidate (Const4) may be derived by combining the second, third, and fourth control point vectors. For example, the fourth constructed candidate (Const4) may have a control point vector as shown in Table 7 below. On the other hand, it may be restricted to be configured as shown in Table 7 only when the reference picture information of the second adjacent block is the same as the reference picture information of the third and fourth adjacent blocks. Here, the reference picture information is as described above.

[0282]

Table 7

[0283] The fifth constructed candidate (Const5) may be derived by combining the first and second control point vectors. For example, the fifth constructed candidate (Const5) may have a control point vector as shown in Table 8 below. On the other hand, it may be restricted such that the control point vector is configured as shown in Table 8 only when the reference picture information of the first adjacent block is the same as the reference picture information of the second adjacent block. Here, the reference picture information is as described above.

[0284]

Table 8

[0285] The sixth configured candidate (Const6) may be derived by combining the first and third control point vectors. For example, the sixth configured candidate (Const6) may have control point vectors as shown in Table 9. On the other hand, the configuration of the control point vectors may be restricted to the case where the reference picture information of the first adjacent block is the same as the reference picture information of the third adjacent block, as shown in Table 9. Here, the reference picture information is as described above.

[0286] [Table 9]

[0287] In Table 9, cpMvCorner[1] may be a second control point vector derived based on the first and third control point vectors. The second control point vector may be derived based on at least one of the first control point vector, a predetermined difference value, or the size of the current / adjacent block. For example, the second control point vector may be derived as shown in Equation 9 below.

[0288]

number

[0289] In the aforementioned K configured candidates (ConstK), the K value does not limit the position or priority of the configured candidates arranged in the candidate list.

[0290] Furthermore, all of the candidates configured as described in the first to sixth sections above may be included in the candidate list, or only some of them may be included in the candidate list.

[0291] For example, if it is determined that the current block uses three control point vectors, only the constructed candidates generated by three combinations of the first to fourth control point vectors may be used. If it is determined that the current block uses two control point vectors, then the constructed candidates generated by at least two combinations of the first to fourth control point vectors may be used, or the constructed candidates generated by only two combinations of the first to fourth control point vectors may be used.

[0292] Alternatively, only a portion of the configured candidates may be included in the candidate list, taking into account the maximum number of affine candidates that can be included in the candidate list. Here, the maximum number may be determined based on the maximum number information signaled in the encoding device, or it may be determined variably taking into account the attributes of the current block mentioned above. In this case, the K value of the configured candidate (ConstK) may represent the priority for insertion into the candidate list.

[0293] On the other hand, if the difference between two control point vectors belonging to the configured candidate is smaller than a predetermined threshold, the configured candidate does not need to be inserted into the candidate list. The difference between the two control point vectors may be distinguished into a horizontal difference and a vertical difference. Here, the horizontal difference means the difference between the first control point vector 910 and the second control point vector 920, and the vertical difference means the difference between the first control point vector 910 and the third control point vector 930. The threshold may mean 0 or a vector of a size already promised to the encoding / decoding device.

[0294] Figure 10 shows an embodiment to which the present invention is applied, illustrating a method for deriving temporal candidate motion information based on subblocks.

[0295] The motion information of the temporal candidate according to the present invention may be derived from the motion information of the collocated block. The motion information may also be derived at the subblock level.

[0296] Here, the collocated block is a block belonging to a picture different from the current block 1010 (i.e., the collocated picture), and may be the block 1040 at the same position as the current block or the block 1030 at a position shifted by a time vector corresponding to the position of the current block. The time vector may be determined based on the motion vectors of the spatially adjacent blocks of the current block. The spatially adjacent blocks may mean the blocks adjacent to at least one of the left side, the lower left corner, the upper side, the upper right corner or the upper left corner of the current block. The time vector may be determined using only the adjacent blocks at positions already agreed upon in the encoding / decoding device. For example, the already agreed upon positions may be the left side 1011 or the upper side 1012, or may be the left side 1011 and the upper side 1012. When there are a plurality of adjacent blocks on the left side, among the plurality of adjacent blocks, a block located at any one of the lowermost, uppermost or central positions may be used. When there are a plurality of adjacent blocks on the upper side, among the plurality of adjacent blocks, a block located at any one of the leftmost, rightmost or central positions may be used.

[0297] When deriving the time candidates based on the sub-blocks, the current block and / or the collocated block may be divided into a plurality of sub-blocks.

[0298] Here, the sub-block may have a fixed size / shape already agreed upon in the encoding / decoding device. For example, the sub-block may be represented by an N×M block, and the values of N and M may be integers of 4, 8 or more. The sub-block may be square (N = M), or may be rectangular (N>M, N<M). It may also mean the size. Alternatively, the encoding device may encode and signal the information regarding the size / shape of the sub-block, and the decoding device may determine the size / shape of the sub-block based on the signaled information.

[0299] Alternatively, the current block and / or collated block may be divided into a plurality of subblocks based on a predetermined number, where the number may be a fixed number already promised to the encoding / decoding device, or it may be determined variably considering the block size / shape, or the number may be determined based on numerical information signaled in the encoding device.

[0300] The following describes a method for deriving motion information for temporal candidates on a subblock basis. The motion vector of a temporal candidate may be set to a motion vector stored separately for each subblock of the collated block. The reference picture index of a temporal candidate may be set to a value already defined in the encoding / decoding device (e.g., 0). Alternatively, the reference picture index of the temporal candidate may be set to the reference picture index of the collated block or to the reference picture index of the upper-left subblock among the subblocks. Alternatively, similar to the motion vector, the reference picture index of a temporal candidate may also be set to a reference picture index stored separately for each subblock.

[0301] However, if there are non-available subblocks among the subblocks belonging to the collated block, the motion vector of the non-available subblock may be replaced with the motion vector of an available subblock within the collated block. Here, an available subblock means a block adjacent to the left, right, top, or bottom of a non-available subblock. Alternatively, an available subblock may be a block at a position already promised to the encoding / decoding device. For example, the already promised position may be the position of the bottom-right subblock within the collated block, or the position of a subblock that includes the center position of the collated block. Temporal candidates based on the aforementioned subblocks may be added to the candidate list only if an available subblock exists. Alternatively, temporal candidates based on the aforementioned subblocks may be added to the candidate list only if the subblock at the already promised position is available.

[0302] Figure 11 shows an embodiment to which the present invention is applied, illustrating an inter-prediction method in an ERP projection format.

[0303] Referring to Figure 11, in the case of the ERP projection format, the reference region identified by the motion vector may span the boundaries of the reference picture.

[0304] In this case, the boundary of the reference picture may be extended, and data adjacent to the boundary may be packed horizontally into the extended area (see A). Alternatively, the boundary of the reference picture may be extended, and data adjacent to the boundary on the opposite side of the extended area (data that is continuous with or correlated with the boundary) may be packed in (see B). Alternatively, data in non-existent areas may be packed using data adjacent to the opposite boundary (see C).

[0305] A and B require a process of expanding the data of the reference picture, but the calculation process can be simplified. C does not require additional memory due to expansion, but it may require further calculations.

[0306] In cases A and B, the current picture size and the (expanded) reference picture size may not be the same. A predetermined adjustment process that takes such size differences into account may be included in the process of obtaining predicted pixels.

[0307] When comparing only the horizontal length, the current picture has a range of 0 to (pic_width-1), and the reference picture (extended as shown on the right) may have a range of -L_offset to (pic_width-1+R_offset) or 0 to (pic_width-1+L_offset+R_offset). In the former case, predicted pixels can be obtained from the extended reference picture at the same position obtained by adding the coordinates and motion vector of the current block. In the latter case, after the adjustment process of the obtained position is performed, predicted pixels can be obtained from the readjusted position. Here, L_offset and R_offset may mean the length extended to the left and right, respectively. In this invention, as described above, the explanation assumes that the coordinates are calculated based on the range of the current picture or an existing reference picture.

[0308] In case C, the current block's coordinates and motion vector can be added together to calculate the position within the reference picture. The calculated pixel position is within the range of the reference picture (<0,0>~<pic_width-1,pic_height-1> It is possible to check whether or not the pixel is outside the bounds of the reference picture. If the calculated pixel position is not outside the bounds of the reference picture, a predicted pixel may be obtained from the calculated pixel position. On the other hand, if the calculated pixel position is outside the bounds of the reference picture, a position corresponding to the calculated position may be calculated from the opposite position inside the boundary (the right side if it is the left side, and the left side if it is the right side), and a predicted pixel may be obtained from the calculated corresponding position.

[0309] Interpretation at the reference picture boundary may be performed based on at least one of the embodiments described above. The interpretation method at the reference picture boundary may be one that is already promised in the encoding / decoding device, or it may be determined based on selection information that identifies one of the embodiments described above and information related to the selected method (such as data processing method and expanded size information). This is also applicable to embodiments described later.

[0310] Figures 12 to 15 illustrate an embodiment to which the present invention is applied, showing an inter-prediction method in the CMP projection format.

[0311] Referring to Figure 12, in the case of the CMP projection format, region-wise packing can create continuity in three-dimensional space between the upper and lower 3x1 surfaces. However, continuity in three-dimensional space may not exist between the upper 3x1 surface and the lower 3x1 surface.

[0312] Similarly, the reference region identified by the motion vector may span the boundaries of the reference picture or the boundaries of some surfaces. In case A, it represents the interior of the reference picture but is located at the boundaries of some surfaces (which have little correlation due to their lack of mutual continuity in 3D space), while in case B, the reference region is located at the boundaries of the reference picture.

[0313] In particular, in case A, although it shows the interior of the reference picture, the accuracy of interpretation prediction may decrease because the reference region includes the boundary of a discontinuous surface.

[0314] In cases A and B, the position of the reference region within the reference picture may be calculated by adding the coordinates and motion vector of the current block. It may also be checked whether at least a portion of the reference region is outside the scope of the reference picture, or whether it contains a discontinuous boundary within the reference picture.

[0315] If the reference region is outside the range of the reference picture, the location where continuity or correlation exists in 3D space may be determined, and the predicted pixels may be obtained from the determined location.

[0316] In contrast, if the reference region is not outside the range of the reference picture, it may be checked whether at least one pixel within the reference region, indicated by the motion vector of the current block, belongs to two or more surfaces. Here, the checking process may be performed using the pixel positions such as the top-left, top-right, bottom-left, and bottom-right of the current block.

[0317] If at least one pixel within the reference region does not belong to two or more surfaces, a predicted pixel may be obtained from the reference region. However, if this is not the case, it may be checked whether the surface has continuity with the surface to which the current block belongs. In the case of reference region A', the upper region has continuity with the surface to which the current block belongs, while the lower region does not have continuity with the surface to which the current block belongs.

[0318] If the aforementioned continuity exists, predicted pixels may be obtained from the reference region. However, if this does not exist, predicted pixels may be obtained from a region with continuity or correlation in the three-dimensional space.

[0319] The interpretation order described above is merely one example, and various modifications can be made by changing the order, altering some of the configurations, etc.

[0320] In cases C and D, the reference picture may be extended, and the extended region may be padded horizontally / vertically with data adjacent to the boundary, or packed with data from correlated regions. In case D, as described in ERP, a detailed explanation is omitted here. In case C, as in A, the reference region is located inside the reference picture, but includes a surface boundary with low correlation. Therefore, if a separate extended region is not generated inside the reference picture, the accuracy of interpretation may decrease, which will be explained in detail based on Figure 13.

[0321] In cases C and D, the size of the current picture and the size of the reference picture may not be the same due to the expansion of the reference picture. In this case, the reference picture may have a range of (-L_offset,-U_offset)~(pic_width-1+R_offset,pic_height-1+D_offset). Here, L_, R_, U_, and D_offset may represent the length expanded from the reference picture in the left, right, up, and down directions. The offsets may have the same value or they may have different values. The length expanded according to the reference picture may be explicitly signaled or implicitly determined.

[0322] For example, it may correspond to one of the lengths that can be handled in the block division, such as 2, 4, or 8. Alternatively, the extended length may be determined based on the minimum coding block, maximum coding block, minimum prediction block, maximum prediction block, etc. Alternatively, it may be determined based on units such as tiles or bricks.

[0323] For example, the reference picture may be expanded to match the length of the maximum predicted block, in which case the acquisition of predicted blocks for all blocks may be processed within the expanded range. If, despite the expansion of the reference picture, the area currently indicated by the block's motion vector is outside the range of the expanded reference picture, the predicted pixels of the portion corresponding to the outside range may be acquired by padding in the horizontal, vertical, or diagonal directions.

[0324] Referring to Figure 13, cases A to D apply when the reference picture is extended on a surface-by-surface basis of one or more surfaces. Cases A and B are examples of extending the reference picture (or sub-reference picture) by grouping together surfaces where continuity exists. Cases C and D are examples of extending the reference picture individually on a surface-by-surface basis. The data processing of the extended area is the same as or similar to that of the embodiments described above. In the embodiments described above, there was one range related to the reference picture, but in this example, the reference picture is divided into two or more surfaces and each range (referred to as a sub-reference picture) is extended, so overlapping ranges may occur.

[0325] In examples A and B, the expanded subreference picture located at the top has an extension range of (-L_offset,-U_offset)~(pic_width-1+R_offset,pic_height / 2-1+D_offset), and the expanded subreference picture located at the bottom has an extension range of (-L_offset,pic_height / 2-U_offset)~(pic_width-1+R_offset,pic_height-1+D_offset). Assume that the expanded left, right, top, and bottom lengths are the same for each subreference picture. However, the lengths to be expanded in the left, right, top, and bottom directions may be set similarly for each unit being expanded, or they may be set differently. Furthermore, the expanded lengths may be set individually depending on the unit being expanded.

[0326] In this case, the coordinates of the area extended downward from the upper subreference picture are the same as the coordinates inside the lower subreference picture. However, in the case of the upper subreference picture, that area contains the extended correlated data, while in the case of the lower subreference picture, it contains the original data before the extension.

[0327] In short, the reference picture may have a range of (-L_offset,-U_offset)~(pic_width-1+R_offset,pic_height-1+D_offset). Generally, it is reasonable to assign one pixel value to each position, but because we have an expanded reference image divided into two or more regions considering the characteristics of an omnidirectional image, two or more pixel values ​​may be assigned to some ranges. In this case, it may be necessary to decide which pixel values ​​to obtain as predicted pixel values ​​from positions where two or more pixel values ​​are assigned.

[0328] Referring to Figure 14, in case E, you can check which sub-reference pictures contain a larger area and use the data from the sub-reference pictures containing the larger area to obtain the predicted value.

[0329] In the case of F, if the subreference picture containing the current block already contains a set number of pixels or more, or contains a predetermined area, the data of that subreference picture may be used. However, if this is not the case, it may be possible to use the data of the subreference picture of an area containing an even larger area, as in E.

[0330] In case G, the data is evenly distributed between both subreferenced pictures. In this case, priority may be given to the subreferenced picture containing the current block, and the data from that subreferenced picture may be used. This example may be included as a detailed condition for F and E.

[0331] In short, when the region indicated by the motion vector of the current block is the boundary between subreference pictures, the data of which subreference picture to use may be determined based on the coverage area, whether or not it is the same subreference picture as the current block, etc. Depending on the encoding / decoding settings, the data of which subreference picture can be used for interpretation may be determined, including all or part of the above conditions.

[0332] The example mentioned above involves splitting the reference picture into two and expanding each region, but this example will explain the case where even more sub-reference pictures are generated on a surface-by-surface basis.

[0333] In Figure 15, it is assumed that each surface unit is a subreference picture. Similarly, it is necessary to determine which pixel values ​​to obtain as predicted pixel values ​​from locations where two or more pixel values ​​are assigned.

[0334] Referring to Figure 15, in the case of H, data from a subreference picture containing more regions may be used, or data from a subreference picture containing the current block may be used.

[0335] For example, if a subreference picture containing the current block contains more than the number of pixels already set or the area already defined, the data of that subreference picture may be used. Alternatively, if data from a subreference picture containing a larger area is used, the data of that subreference picture may be used. However, if the contained area is the same, it may be restricted to using the data of the subreference picture containing the current block.

[0336] In case I, data from subreference pictures of regions containing even more regions may be used. In this example, subreference pictures can be handled on a surface-by-surface basis, but because the surface setting has the characteristic that the upper 3x1 region and the lower 3x1 region are not continuous, priority can be given to the subreference picture on the side with continuous characteristics over the subreference picture on the side with discontinuous characteristics.

[0337] In the case of J, data from subreference pictures of regions containing more data may be used. However, this is only if the contained regions are evenly distributed. In this example as well, as in I, the subreference picture containing the current block (the rightmost part of the 3x2 area) is not an object that spans across regions. Furthermore, the subreference picture in question (the leftmost and center parts of the 3x2 area) belongs to the same 3x1 region as the subreference picture containing the current block. In this case, priority can be given to the subreference picture that is closer to the subreference picture containing the current block.

[0338] In short, when the area indicated by the current block's motion vector is the boundary between subreferenced pictures, it may be possible to decide which subreferenced picture data to use based on factors such as the coverage area and the priority of using the subreferenced pictures.

[0339] On the other hand, based on Figures 7 to 10, we described a method for performing interpretation based on affine motion modeling. This can be similarly applied to encoding / decoding 360 video, and a detailed explanation of that is omitted here.

[0340] The method according to the present invention may be implemented in the form of program instructions that can be performed by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable medium may be specifically designed and configured for the present invention, or they may be publicly known and available to those skilled in the art of computer software.

[0341] Examples of computer-readable media include hardware devices specifically designed to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The aforementioned hardware devices may be configured to operate as at least one software module to perform the operations of the present invention, and vice versa.

[0342] Furthermore, the aforementioned methods or apparatus may be implemented by combining all or part of their configurations and functions, or by separating them.

[0343] While preferred embodiments of the present invention have been described above, those skilled in the art will understand that the present invention can be modified and altered in various ways without departing from the spirit and scope of the invention as described in the appended claims. [Industrial applicability]

[0344] This invention can be used for encoding / decoding video signals.

Claims

1. Currently, the steps involve building a list of candidates for merging blocks, A step of deriving motion information for the current block from the merge candidate list, wherein the motion information includes at least one of a predicted direction flag, a reference picture index, or a motion vector; Using the aforementioned motion information, the step of predicting the current block's interoperability, Includes, The merge candidate list includes combined merge candidates, and the combined merge candidate is derived by combining n merge candidates belonging to the merge candidate list. The n merge candidates include a first merge candidate and a second merge candidate, and the motion information of the combined merge candidate is derived by weighting the motion information of the first merge candidate and the motion information of the second merge candidate. The first merge candidate corresponds to the first unidirectional predicted motion vector, the second merge candidate corresponds to the second unidirectional predicted motion vector, and the directions of the first unidirectional predicted motion vector and the second unidirectional predicted motion vector are the same. The reference picture index of the aforementioned combination merge candidate is derived as the reference picture index of the first merge candidate. The step of using the aforementioned motion information to predict the current block's interoperability is as follows: Based on the reference picture index of the current block, select a reference picture from the reference picture list, Based on the motion vector of the current block, the reference block in the reference picture is identified, This includes generating predicted pixels for the current block using the reconstructed pixels of the identified reference block, The n merge candidates are the merge candidates corresponding to indices 0 to (n-1) in the merge candidate list, A video signal decoding method wherein the index of the combination merge candidate is greater than the index of the temporal merge candidate.

2. The video signal decoding method according to claim 1, wherein the weights of the weighted average are one of [1:1], [1:2], [1:3], or [2:3].

3. The method for decoding a video signal according to claim 1, comprising performing inverse quantization or inverse transformation on residual coefficients signaled by a bitstream to obtain the residual pixels.

4. Memory for storing a program for performing interpretation, A video signal decoding device comprising: a processor for executing the program described above and performing the video signal decoding method according to any one of claims 1 to 3.

5. Currently, the steps involve building a list of candidates for merging blocks, A step of deriving motion information for the current block from the merge candidate list, wherein the motion information includes at least one of a predicted direction flag, a reference picture index, or a motion vector; Using the aforementioned motion information, the step of predicting the current block's interoperability, Includes, The merge candidate list includes combined merge candidates, The aforementioned combination merge candidate is derived by combining n merge candidates belonging to the merge candidate list. The n merge candidates include a first merge candidate and a second merge candidate, and the motion information of the combined merge candidate is derived by weighting the motion information of the first merge candidate and the motion information of the second merge candidate. The first merge candidate corresponds to the first unidirectional predicted motion vector, the second merge candidate corresponds to the second unidirectional predicted motion vector, and the directions of the first unidirectional predicted motion vector and the second unidirectional predicted motion vector are the same. The reference picture index of the aforementioned combination merge candidate is derived as the reference picture index of the first merge candidate. The step of using the aforementioned motion information to predict the current block's interoperability is as follows: Based on the reference picture index of the current block, select a reference picture from the reference picture list, Based on the motion vector of the current block, the reference block in the reference picture is identified, This includes generating predicted pixels for the current block using the reconstructed pixels of the identified reference block, The n merge candidates are the merge candidates corresponding to indices 0 to (n-1) in the merge candidate list, A video signal encoding method wherein the index of the combination merge candidate is greater than the index of the temporal merge candidate.

6. The method for encoding a video signal according to claim 1, wherein the weights of the weighted average are one of [1:1], [1:2], [1:3], or [2:3].

7. Memory for storing a program for performing interpretation, A video signal encoding apparatus comprising a processor for executing the program described above to perform the video signal encoding method according to claim 5 or 6.

8. A computer-readable storage medium storing a computer program and a bitstream, wherein when the computer program is executed by a processor, the processor is instructed to perform the following operations to generate the bitstream, the operations being: Currently, the steps involve building a list of candidates for merging blocks, A step of deriving motion information for the current block from the merge candidate list, wherein the motion information includes at least one of a predicted direction flag, a reference picture index, or a motion vector; Using the aforementioned motion information, the step of predicting the current block's interoperability, Includes, The merge candidate list includes combined merge candidates, The aforementioned combination merge candidate is derived by combining n merge candidates belonging to the merge candidate list. The n merge candidates include a first merge candidate and a second merge candidate, and the motion information of the combined merge candidate is derived by weighting the motion information of the first merge candidate and the motion information of the second merge candidate. The first merge candidate corresponds to the first unidirectional predicted motion vector, the second merge candidate corresponds to the second unidirectional predicted motion vector, and the directions of the first unidirectional predicted motion vector and the second unidirectional predicted motion vector are the same. The reference picture index of the aforementioned combination merge candidate is derived as the reference picture index of the first merge candidate. The step of using the aforementioned motion information to predict the current block's interoperability is as follows: Based on the reference picture index of the current block, select a reference picture from the reference picture list, Based on the motion vector of the current block, the reference block in the reference picture is identified, This includes generating predicted pixels for the current block using the reconstructed pixels of the identified reference block, The n merge candidates are the merge candidates corresponding to indices 0 to (n-1) in the merge candidate list, A computer-readable storage medium in which the index of the aforementioned combination merge candidate is greater than the index of the temporal merge candidate.

Images