Inter prediction method and apparatus based on merge mode

Abstract

The present application relates to a kind of video encoding / decoding method and device, it is characterized in that including: the step of constituting the merging candidate list of current block, according to the motion information of merging candidate list export current block, utilize motion information to perform the interframe prediction of current block;Merging candidate list contains at least one of spatial merging candidate, time merging candidate or combination merging candidate, combination merging candidate is exported by combining n merging candidates belonging to merging candidate list, since not only utilize spatial / time merging candidate, but also utilize combination merging candidate, therefore have the effect of being able to improve the accuracy of motion information.

Description

[0001] This application is a divisional application of Chinese patent application No. 201980041821.0, entitled "Inter-frame prediction method and apparatus based on merging mode", which entered the Chinese national phase of PCT international patent application PCT / KR2019 / 007981 filed on July 1, 2019. Technical Field

[0002] This invention relates to an inter-frame prediction method and apparatus. Background Technology

[0003] Recently, there has been an increasing demand for high-resolution, high-quality video, such as HD (High Definition) video and UHD (Ultra High Definition) video, in various application fields, so efficient video compression technologies are being researched.

[0004] As video compression technologies, there are various techniques such as inter-frame prediction, which predicts the pixel values ​​of the current image based on images before or after the current image; intra-frame prediction, which predicts the pixel values ​​of the current image using pixel information in the current image; and entropy coding, which assigns short codes to values ​​that occur frequently and long codes to values ​​that occur infrequently. These video compression technologies can be used to effectively compress video data for transmission or storage. Summary of the Invention

[0005] [The problem the invention aims to solve]

[0006] The purpose of this invention is to provide an inter-frame prediction method and apparatus.

[0007] The purpose of this invention is to provide a method and apparatus for constructing a merged candidate list.

[0008] The purpose of this invention is to provide a motion compensation method and apparatus for sub-block units.

[0009] The purpose of this invention is to provide a method and apparatus for determining affine candidates.

[0010] The purpose of this invention is to provide an inter-frame prediction method and apparatus corresponding to the projection format of 360 video.

[0011] [Problem-solving methods]

[0012] The video encoding / decoding method and apparatus of the present invention can construct a merging candidate list for the current block, derive motion information of the current block based on the merging candidate list, and use the motion information to perform inter-frame prediction of the current block.

[0013] In the video encoding / decoding method and apparatus of the present invention, the merging candidate list may include at least one of spatial merging candidates, temporal merging candidates, or combined merging candidates.

[0014] In the video encoding / decoding method and apparatus of the present invention, the combined merging candidate can be derived by combining n merging candidates belonging to the merging candidate list.

[0015] In the video encoding / decoding method and apparatus of the present invention, the n merging candidates may be merging candidates corresponding to indices from 0 to (n-1) in the merging candidate list.

[0016] In the video encoding / decoding method and apparatus of the present invention, the index of the combined merging candidate can be greater than the index of the time merging candidate.

[0017] In the video encoding / decoding method and apparatus of the present invention, the n merging candidates include a first merging candidate and a second merging candidate, and the motion information of the combined merging candidate can be derived by considering the prediction direction of the first merging candidate and the prediction direction of the second merging candidate.

[0018] In the video encoding / decoding method and apparatus of the present invention, the motion information of the combined candidate can be derived by weighted averaging the motion information of the first combined candidate and the motion information of the second combined candidate.

[0019] In the video encoding / decoding method and apparatus of the present invention, the weighting value of the weighted average can be any one of [1∶1], [1∶2], [1∶3] or [2∶3].

[0020] The video encoding / decoding method and apparatus of the present invention can generate a candidate list for predicting motion information of the current block, derive the control point vector of the current block based on the candidate list and candidate index, derive the motion vector of the current block based on the control point vector of the current block, and perform inter-frame prediction on the current block using the motion vector.

[0021] In the video encoding / decoding apparatus of the present invention, the candidate list may contain multiple affine candidates.

[0022] In the video encoding / decoding apparatus of the present invention, the affine candidate may include at least one of spatial candidate, temporal candidate, or a combination thereof.

[0023] In the video encoding / decoding apparatus of the present invention, the motion vector of the current block can be derived on a sub-block basis.

[0024] In the video encoding / decoding apparatus of the present invention, the spatial candidate can be determined by considering whether the boundary of the current block is adjacent to the boundary (CTU boundary) of the coding tree block.

[0025] In the video encoding / decoding apparatus of the present invention, the constructed candidate can be determined based on the combination of at least two control point vectors among the control point vectors corresponding to each corner of the current block.

[0026] The video encoding / decoding method and apparatus of the present invention can obtain all or a portion of the pixels of the reference region by utilizing data from a correlated region when the reference region used for inter-frame prediction includes the boundary of a reference image or the boundary between discontinuous surfaces.

[0027] [The effects of the invention]

[0028] According to the present invention, not only spatial / temporal merging candidates are utilized, but also combined merging candidates are utilized, thereby improving the accuracy of motion information.

[0029] According to the present invention, the encoding / decoding performance of video can be improved by inter-frame prediction based on affine models.

[0030] According to the present invention, the accuracy of prediction can be improved by inter-frame prediction of sub-block units.

[0031] According to the present invention, affine candidates can be determined efficiently, thereby improving the encoding / decoding efficiency of inter-frame prediction.

[0032] According to the present invention, the reference region can be set by taking correlation into account, thereby improving the coding efficiency of inter-frame prediction. Attached Figure Description

[0033] Figure 1 This is a block diagram of a video encoding apparatus according to an embodiment of the present invention.

[0034] Figure 2 This is a block diagram of a video decoding device according to an embodiment of the present invention.

[0035] Figure 3 This is an example diagram representing a tree-based block structure.

[0036] Figure 4 This is an illustrative diagram showing the shapes of various blocks that can be obtained from the block division section of the present invention.

[0037] Figure 5 This illustrates a block partitioning process according to an embodiment of the present invention.

[0038] Figure 6 As an embodiment of the present invention, it is illustrated by a diagram showing a method for performing inter-frame prediction based on a block-unit merging pattern.

[0039] Figure 7 As an embodiment of the present invention, it is illustrated by a diagram showing an inter-frame prediction method based on an affine model.

[0040] Figure 8 As an embodiment of the present invention, it is illustrated by a diagram showing a method for deriving affine candidates based on the control point vectors of spatially / temporally adjacent blocks.

[0041] Figure 9 As an embodiment of the present invention, it is illustrated by a diagram showing a method for deriving the constructed candidate based on the combination of motion vectors of spatial / temporal adjacent blocks.

[0042] Figure 10 As an embodiment of the present invention, it is a diagram illustrating a method for deriving motion information based on temporal candidates of sub-blocks.

[0043] Figure 11 As an embodiment of the present invention, it is a diagram illustrating the inter-frame prediction method under the ERP (Equi-Rectangular Projection) projection format.

[0044] Figures 12 to 15 As an embodiment of the present invention, it is a diagram illustrating the inter-frame prediction method under the CMP (Cube Map Projection) projection format. Detailed Implementation

[0045] The video encoding / decoding method and apparatus of the present invention can construct a merging candidate list for the current block, derive motion information of the current block based on the merging candidate list, and use the motion information to perform inter-frame prediction of the current block.

[0046] In the video encoding / decoding method and apparatus of the present invention, the merging candidate list may include at least one of spatial merging candidates, temporal merging candidates, or combined merging candidates.

[0047] In the video encoding / decoding method and apparatus of the present invention, the combined merging candidate can be derived by combining n merging candidates belonging to the merging candidate list.

[0048] In the video encoding / decoding method and apparatus of the present invention, the n merging candidates may be merging candidates corresponding to indices from 0 to (n-1) in the merging candidate list.

[0049] In the video encoding / decoding method and apparatus of the present invention, the index of the combined merging candidate can be greater than the index of the time merging candidate.

[0050] In the video encoding / decoding method and apparatus of the present invention, the n merging candidates include a first merging candidate and a second merging candidate, and the motion information of the combined merging candidate can be derived by considering the prediction direction of the first merging candidate and the prediction direction of the second merging candidate.

[0051] In the video encoding / decoding method and apparatus of the present invention, the motion information of the combined candidate can be derived by weighted averaging the motion information of the first combined candidate and the motion information of the second combined candidate.

[0052] In the video encoding / decoding method and apparatus of the present invention, the weighting value of the weighted average can be any one of [1∶1], [1∶2], [1∶3] or [2∶3].

[0053] The video encoding / decoding method and apparatus of the present invention can generate a candidate list for predicting motion information of the current block, derive the control point vector of the current block based on the candidate list and candidate index, derive the motion vector of the current block based on the control point vector of the current block, and use the motion vector to perform inter-frame prediction on the current block.

[0054] In the video encoding / decoding apparatus of the present invention, the candidate list may contain multiple affine candidates.

[0055] In the video encoding / decoding apparatus of the present invention, the affine candidate may include at least one of spatial candidate, temporal candidate, or a combination thereof.

[0056] In the video encoding / decoding apparatus of the present invention, the motion vector of the current block can be derived on a sub-block basis.

[0057] In the video encoding / decoding apparatus of the present invention, the spatial candidate can be determined by considering whether the boundary of the current block is adjacent to the boundary (CTU boundary) of the coding tree block.

[0058] In the video encoding / decoding apparatus of the present invention, the constructed candidate can be determined based on the combination of at least two control point vectors among the control point vectors corresponding to each corner of the current block.

[0059] The video encoding / decoding method and apparatus of the present invention can obtain all or a portion of the pixels of the reference region by utilizing data from a correlated region when the reference region used for inter-frame prediction includes the boundary of a reference image or the boundary between discontinuous surfaces.

[0060] This invention can be modified in various ways and has multiple embodiments, with specific embodiments shown in the figures and described in detail. However, it should be understood that these specific embodiments are not intended to limit the invention to the specific implementation, but include all modifications, equivalents, or substitutions within the scope of the invention's ideas and techniques.

[0061] The terms "first," "second," "A," and "B," etc., may be used to describe constituent elements, but the constituent elements should not be limited by these terms. These terms are only used to distinguish one constituent element from other constituent elements. For example, a first constituent element may be named a second constituent element without departing from the scope of the invention; similarly, a second constituent element may be named a first constituent element. The term "and / or" refers to a combination of the plurality of related items described or any one of the plurality of related items described.

[0062] When a component is described as being "connected" or "linked" to another component, it should be understood that it can be directly connected or linked to the other component, or that there may be other components between the component and the other component. Conversely, when a component is described as being "directly connected" or "directly linked" to another component, it should be understood that there are no other components between the component and the other component.

[0063] The terminology used in this application is for illustrative purposes only and is not intended to limit the invention. Unless otherwise expressly indicated in the text, the singular expression includes the plural expression. In this application, terms such as "comprising" or "having" should be understood as indicating the presence of the features, numbers, steps, actions, constituent elements, parts, or combinations thereof described in the specification, and do not preclude the possibility or additional possibilities of the presence of one or more other features, numbers, steps, actions, constituent elements, parts, or combinations thereof.

[0064] Unless otherwise defined, all terms, including technical or scientific terms, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms as defined in commonly used dictionaries shall be interpreted as having the same meaning as in relevant technical literature, and shall not be construed as having an ideal or overly formal meaning unless expressly defined in this application.

[0065] Video encoding and decoding devices can be user terminals such as personal computers (PCs), laptops, personal digital assistants (PDAs), portable multimedia players (PMPs), portable game consoles (PSPs), wireless communication terminals, smartphones, televisions, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, head-mounted display (HMD) devices, and smart glasses, or server terminals such as application servers and service servers. They can include various devices such as: communication devices such as communication modems for communicating with various devices or wired / wireless communication networks; memory for storing various programs and data for intra-frame or inter-frame prediction for video encoding or decoding; and processors for running programs for computation and control. In addition, video encoded into a bitstream by a video encoding device can be transmitted in real time or non-real time to a video decoding device via wired and wireless communication networks such as the Internet, short-range wireless communication networks, wireless local area networks, wireless broadband networks, and mobile communication networks, or via various communication interfaces such as cables and Universal Serial Bus (USB), and then decoded by the video decoding device to reconstruct the video for playback.

[0066] In addition, video encoded into a bitstream by a video encoding device can also be transmitted from the encoding device to the decoding device via a recording medium that can be read by a computer.

[0067] The video encoding device and the video decoding device may be separate devices, but they can be configured as a single video encoding / decoding device depending on the specific implementation. In this case, a portion of the video encoding device may be implemented as a technical element substantially the same as a portion of the video decoding device, and may at least include the same structure or at least perform the same function.

[0068] Therefore, in the following detailed description of the technical elements and their operating principles, the repeated descriptions of the corresponding technical elements will be omitted.

[0069] Furthermore, the video decoding device is equivalent to a computing device that applies the video encoding method executed by the video encoding device to the decoding process; therefore, the following description will focus on the video encoding device.

[0070] The computing device may include: a memory storing programs or software modules that implement video encoding methods and / or video decoding methods; and a processor connected to the memory to execute the programs. Furthermore, the video encoding device may be referred to as an encoder, and the video decoding device as a decoder.

[0071] Typically, video can contain a series of still images, which can be divided into GOPs (Group of Pictures), and each still image can be called a picture. In this case, a picture can represent one of the following: a frame or a field in progressive or interlaced signal. When encoding / decoding is performed on a frame-by-frame basis, the video can be represented by "frames"; when encoding / decoding is performed on a field-by-field basis, the video can be represented by "fields". In this invention, progressive signal is used as an example, but it can also be applied to interlaced signal. As a higher-level concept, there can be units such as GOPs and sequences. Furthermore, each picture can be divided into specific regions such as segments, blocks, and chunks. Additionally, a GOP can contain I-images, P-images, B-images, etc. An I-image can refer to an image that is encoded / decoded automatically without using a reference image. P-images and B-images can refer to images that are encoded / decoded using a reference image through processes such as motion estimation and motion compensation. Typically, a P-image can use both an I-image and a P-image as a reference image, and a B-image can use both an I-image and a P-image as a reference image. However, these definitions can be modified depending on the encoding / decoding settings.

[0072] Here, the image referenced during encoding / decoding is called the reference picture, and the reference block or pixel is called the reference block or reference pixel. In addition to pixel values ​​in the spatial domain, reference data can also be coefficient values ​​in the frequency domain, and various encoding / decoding information generated and determined during the encoding / decoding process. For example, in the prediction unit, it can be intra-frame prediction-related information or motion-related information; in the transform / inverse transform unit, it can be transform-related information; in the quantization / inverse quantization unit, it can be quantization-related information; in the encoding / decoding unit, it can be encoding / decoding-related information (context information); and in the in-loop filtering unit, it can be filtering-related information, etc.

[0073] The smallest unit that makes up a video is a pixel, and the number of bits required to represent one pixel is called the bit depth. Typically, the bit depth is 8 bits, but depending on the encoding settings, bit depths of 8 bits or more can be supported.

[0074] The bit depth can support at least one bit depth according to the color space. Additionally, it can be composed of at least one color space depending on the video's color format. It can be composed of more than one image with a fixed size or more than one image with different sizes, depending on the color format. For example, in the case of YCbCr 4∶2∶0, it can include one luminance component (Y in this example) and two chrominance components (Cb / Cr in this example), where the ratio of the chrominance component to the luminance component can be 1:2 (horizontal to vertical). As another example, in the case of 4∶4∶4, it can have the same horizontal and vertical ratio. In the case of including more than one color space as described above, the image can be segmented in each color space.

[0075] In this invention, a portion of the color space (Y in this example) of a certain color format (YCbCr) is used as a reference for explanation. The same or similar application can be made in other color spaces based on the color format (Cb, Cr in this example) (depending on the settings of the specific color space). However, some differences can also be retained in each color space (independent of the settings of the specific color space). That is, settings dependent on each color space can refer to settings that are proportional to or dependent on the composition ratio of each component (e.g., determined according to 4:2:0, 4:2:2, 4:4:4, etc.), while settings independent of each color space can refer to settings that are unrelated to the composition ratio of each component or independently applicable only to the corresponding color space. In this invention, a portion of the composition can be set independently or dependently according to the encoder / decoder.

[0076] In video encoding, the setup information or syntax elements required can be determined at the unit level, such as video, sequence, image, segment, block, etc. These elements can be included in the bitstream and transmitted to the decoder in units such as VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), Slice Header, Tile Header, and Block Header. The decoder can then parse these units at the same level to reconstruct the setup information transmitted from the encoder for use in the video decoding process. Additionally, relevant information can be transmitted to the bitstream and parsed for use in the form of SEI (Supplement Enhancement Information) or metadata. Each parameter set has an inherent ID value, and lower-level parameter sets can have the ID values ​​of higher-level parameter sets that need to be referenced. For example, a lower-level parameter set can reference information from more than one higher-level parameter set with consistent ID values. In the examples of various units described above, when a unit contains more than one other unit, the corresponding unit can be called the superordinate unit, and the unit that is contained can be called the subordinate unit.

[0077] In the case of the setting information generated by the unit, each corresponding unit may contain independent setting content or content that depends on the settings of previous, subsequent, or superior units. Here, dependent settings can be understood as flag information used to indicate the setting information of the corresponding unit for conforming to the settings of previous, subsequent, or superior units (e.g., 1 bit flag indicates conformity, 0 indicates non-conformity). In this invention, the setting information is described with examples of independent settings as the main focus, but examples may also be included that add or replace content with a dependent relationship regarding the setting information of previous, subsequent, or superior units of the current unit.

[0078] Figure 1 This is a block diagram of a video encoding apparatus according to an embodiment of the present invention. Figure 2 This is a block diagram of a video decoding device according to an embodiment of the present invention.

[0079] Reference Figure 1 The video encoding apparatus may include a prediction unit, a subtraction unit, a transform unit, a quantization unit, an inverse quantization unit, an inverse transform unit, an addition unit, an in-loop filter unit, a memory, and / or an encoding unit. It may exclude some of the above-mentioned components, selectively include some or all of the components depending on the specific implementation, and may include another component not shown in the figure.

[0080] Reference Figure 2 The video decoding device may include a decoding unit, a prediction unit, an inverse quantization unit, an inverse transform unit, an addition unit, an in-loop filtering unit, and / or a memory. It may exclude some of the above components, selectively include some or all of the components depending on the specific implementation, and may include another component not shown in the figure.

[0081] The video encoding device and the video decoding device may be separate devices, but they can be configured as a single video encoding / decoding device depending on the specific implementation. In this case, a portion of the video encoding device may be implemented as a technical element substantially the same as a portion of the video decoding device, and may at least include the same structure or at least perform the same function.

[0082] Therefore, in the detailed explanation of the following technical elements and their operating principles, repeated descriptions of the corresponding technical elements will be omitted. A video decoding device is essentially a computing device that applies the video encoding method executed by a video encoding device to decoding; therefore, the following explanation will focus on the video encoding device. The video encoding device can be referred to as an encoder, and the video decoding device as a decoder.

[0083] The prediction unit may include an intra-prediction unit that performs intra-frame prediction and an inter-prediction unit that performs inter-frame prediction. Intra-prediction uses pixels from adjacent blocks as reference pixels to determine the intra-prediction mode and generates a prediction block using the intra-prediction mode. Inter-prediction uses one or more reference images to determine the motion information of the current block and performs motion compensation using the motion information to generate a prediction block. After determining whether to use intra-prediction or inter-prediction for the current block (coding unit or prediction unit), specific information corresponding to each prediction method (e.g., intra-prediction mode, motion vector, reference image, etc.) is determined. At this time, the processing unit performing the prediction and the processing unit determining the prediction method and its specific content can be determined based on the encoding / decoding settings. For example, the prediction method and prediction mode are determined by prediction unit (or coding unit), and the prediction is executed by prediction block unit (or coding unit, transform unit).

[0084] The subtraction unit generates a residual block by subtracting the prediction block from the current block. That is, the subtraction unit calculates the difference between the pixel value of each pixel in the current block to be encoded and the predicted pixel value of each pixel in the prediction block generated by the prediction unit to generate a residual signal in block form, i.e., a residual block.

[0085] The transformation unit can transform a signal belonging to the spatial domain into a signal belonging to the frequency domain. The signal obtained through the transformation process is called the transformed coefficient. For example, a residual block having a residual signal received from the subtraction unit can be transformed to obtain a transform block with transformed coefficients. The received signal is determined by the encoding settings and is not limited to the residual signal.

[0086] The transformation unit can use transformation methods such as Hadamard Transform, Discrete Sine Transform (DST Based-Transform), Discrete Cosine Transform (DCT Based-Transform) to transform the residual block, but is not limited to these methods. Various transformation methods obtained by improving and modifying these transformation methods can also be used.

[0087] Transformations can be performed along the horizontal or vertical direction. For example, a two-dimensional transformation can be performed by using the fundamental vectors in the transformation to perform a one-dimensional transformation along the horizontal direction and a one-dimensional transformation along the vertical direction, thereby transforming pixel values ​​in the spatial domain into pixel values ​​in the frequency domain.

[0088] In addition, the transformation unit can transmit the information required to generate the transformation block to the encoding unit to encode the information, and then record the information obtained therefrom into a bit stream and transmit the bit stream to the decoder. The decoding unit of the decoder can parse the information in the bit stream and use it for the inverse transformation process.

[0089] The quantization unit can quantize the received signal. The signal obtained through the quantization process is called the quantized coefficient. For example, a residual block with residual transform coefficients received from the transform unit can be quantized to obtain a quantized block with quantized coefficients. The received signal is determined by the encoding settings and is not limited to the residual transform coefficients.

[0090] The quantization department can use quantization methods such as Dead Zone Uniform Threshold Quantization and Quantization Weighted Matrix to quantize the transformed residual block, but is not limited to these methods. Various quantization methods obtained by improving and transforming these quantization methods can also be used.

[0091] The quantization unit can transmit the information required to generate the quantization block to the encoding unit to encode the information, and then record the resulting information into a bit stream and transmit the bit stream to the decoder. The decoding unit of the decoder can parse the information in the bit stream for use in the dequantization process.

[0092] In the above example, the transformation and quantization of the residual block using the transformation unit and the quantization unit were described. However, the residual block can be transformed into a residual signal to generate a residual block with transformation coefficients without performing the quantization process. Alternatively, the residual signal of the residual block can be transformed into transformation coefficients without performing the quantization process, or neither transformation nor quantization process can be performed. This can be determined according to the encoder settings.

[0093] The encoding unit can scan the quantization coefficients, transform coefficients, or residual signals of the generated residual blocks according to at least one scanning order (e.g., zigzag scanning, vertical scanning, horizontal scanning, etc.) to generate a quantization coefficient sequence, transform coefficient sequence, or signal sequence, and encode it using at least one entropy coding method. In this case, the information of the scanning order can be determined according to the encoding settings (e.g., encoding mode, prediction mode, etc.), and the relevant information can be implicitly determined or explicitly generated. For example, one of several scanning orders can be selected according to the intra-frame prediction mode. In this case, the scanning pattern can be set to one of various patterns such as zig-zag, diagonal, or raster.

[0094] Additionally, encoded data containing the encoded information transmitted from each component can be generated and output to a bitstream. This can be achieved using a multiplexer (MUX). At this point, methods such as Exponential Golomb, Context Adaptive Variable Length Coding (CAVLC), and Context Adaptive Binary Arithmetic Coding (CABAC) can be used for encoding, but this is not a limitation. Various encoding methods derived from improvements and variations of these methods can also be used.

[0095] When performing entropy encoding (in this example, CABAC) on the residual block data and syntax elements such as information generated during encoding / decoding, the entropy encoding device may include a binarizer, a context modeler, and a binary arithmetic coder. The binary arithmetic coder may then include a regular coding engine and a bypass coding engine. The regular coding engine can be a process executed in relation to the context modeler, while the bypass coding engine can be a process executed independently of the context modeler.

[0096] The syntax elements input to the entropy encoding device may not be binary values. Therefore, when the syntax elements are not binary values, the binarization unit can binarize the syntax elements and output a binary string consisting of 0s or 1s. In this case, the binary representation consists of bits consisting of 0s or 1s, which can be encoded by the binary arithmetic encoding unit. At this time, either the regular encoding unit or the bypass encoding unit can be selected based on the generation probabilities of 0s and 1s, and this can be determined according to the encoding / decoding settings. If the syntax elements are data with the same frequency of 0s and 1s, the bypass encoding unit can be used; otherwise, the regular encoding unit can be used, which can be referenced when executing the next regular encoding unit through context modeling (or context information update).

[0097] At this point, the context is information about the generation probability of the binary (Bin). Context modeling is the process of estimating the probability of the binary required for binary arithmetic encoding by taking the binarized result (Bin) as input. To estimate the probability, information such as the syntax elements of the binary, the position (index) of the binary in the binary string, and the probability of the binary contained in surrounding blocks can be used. Furthermore, at least one context table can be used for probability estimation. For example, information for a subset of flags can be obtained using multiple context tables depending on whether surrounding blocks use combinations of flags.

[0098] Various methods can be used when binarizing the syntax elements. For example, they can be categorized into fixed-length binarization and variable-length binarization. In the case of variable-length binarization, unary binarization (truncated unary binarization), rice binarization, K-th Exp-Golomb binarization, and truncated binary binarization can be used. Furthermore, signed or unsigned binarization can be performed depending on the range of values ​​possessed by the syntax element. The binarization process for syntax elements in this invention can perform not only the binarization methods described in the examples above, but also other binarization methods.

[0099] The dequantization unit and the inverse transform unit can reverse the processes in the transform unit and the quantization unit. For example, the dequantization unit can dequantize the quantized transform coefficients generated by the quantization unit, and the inverse transform unit can inverse transform the dequantized transform coefficients to generate the reconstructed residual block.

[0100] The addition unit adds the predicted block to the reconstructed residual block to reconstruct the current block. The reconstructed block can be stored in memory and used as reference data (for the prediction unit and filtering unit, etc.).

[0101] The in-loop filtering unit may include at least one post-processing filtering procedure such as deblocking filtering, Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF). Deblocking filtering removes block distortion occurring at the boundaries between blocks from the reconstructed video. ALF performs filtering based on values ​​obtained by comparing the reconstructed video with the input video. Specifically, filtering can be performed after filtering the blocks using deblocking filtering, based on values ​​obtained by comparing the reconstructed video with the input video. Alternatively, filtering can be performed after filtering the blocks using SAO, based on values ​​obtained by comparing the reconstructed video with the input video.

[0102] The memory can store reconstructed blocks or images. These reconstructed blocks or images stored in memory can be provided to the prediction unit, which performs intra-frame or inter-frame prediction. Specifically, the storage space in the form of a queue of bitstreams compressed in the encoder can be used as a Coded Picture Buffer (CPB), and the space storing decoded video in image units can be used as a Decoded Picture Buffer (DPB). In the CPB, decoded units can be stored in the decoding order. The decoding process is simulated in the encoder, and the bitstream compressed during the simulation is stored. The bitstream output from the CPB can be reconstructed through the decoding process, and the reconstructed video is stored in the DPB. The images stored in the DPB are referenced during subsequent video encoding and decoding processes.

[0103] The decoding unit can reverse the processes in the encoding unit. For example, it can receive and decode quantization coefficient sequences, transform coefficient sequences, or signal sequences from a bitstream, and can parse and transmit decoded data containing decoding information to the respective constituent units.

[0104] A video encoding apparatus and a video decoding apparatus according to an embodiment of the present invention may include a block segmentation unit. The basic encoding unit may refer to the basic (or initial) unit of prediction, transformation, quantization, etc., in the video encoding / decoding process. In this case, the encoding unit may consist of one luma encoding block and two chroma encoding blocks according to the color format (YCbCr in this example), and the size of each block can be determined according to the color format. In the examples described below, the block (luminance component in this example) is used as the basis for explanation. Here, it is assumed that the block is a unit that can be obtained after determining each unit, and similar settings can be applied to other types of blocks for explanation.

[0105] The block segmentation unit can be set according to the various components of the video encoding and decoding apparatus, and the size and shape of the block can be determined through this process. At this time, the set block can be defined differently according to the components; it can be a prediction block in the prediction unit, a transform block in the transform unit, a quantization block in the quantization unit, etc. It is not limited to this; block units corresponding to other components can also be defined. The size and shape of the block can be defined according to its horizontal and vertical dimensions.

[0106] In the block segmentation section, blocks can be represented as M×N, and the maximum and minimum values ​​of each block can be obtained within a range. For example, the block shape supports squares. When the maximum value of the block is set to 256×256 and the minimum value is set to 8×8, blocks of size 2m×2m (in this example, m is an integer from 3 to 8, such as 8×8, 16×16, 32×32, 64×64, 128×128, 256×256), blocks of size 2m×2m (in this example, m is an integer from 4 to 128), or blocks of size m×m (in this example, m is an integer from 8 to 256) can be obtained. Alternatively, the block shape supports squares and rectangles. When within the same range as the examples above, blocks of size 2m×2n can be obtained (in this example, m and n are integers from 3 to 8, with a maximum aspect ratio of 2:1, for example, 8×8, 8×16, 16×8, 16×16, 16×32, 32×16, 32×32, 32×64, 64×32, 64×64, 64×128, 128×64, 128×128, 128×256, 256×128, 256×256; depending on the encoding / decoding settings, the aspect ratio can be unlimited or may have a maximum value). Alternatively, blocks of size 2m×2n can be obtained (in this example, m and n are integers from 4 to 128). Alternatively, blocks of size m×n can be obtained (in this example, m and n are integers from 8 to 256).

[0107] In the block segmentation section, the input block can be divided into more than one block. For example, the input block can be directly output, or it can be divided into two or more sub-blocks. In this case, the number of sub-blocks can be 2, 3, 4, etc., which can be determined according to the segmentation method (segmentation type). The shape of the sub-blocks can be obtained based on the block before segmentation (the input block in this example), and can take various forms depending on the block segmentation settings.

[0108] In this invention, the description focuses on tree-based segmentation, but is not limited to this.

[0109] The available blocks can be determined based on the encoding / decoding settings (e.g., block type, segmentation method, segmentation settings, etc.). For example, an encoded block can obtain a 2m×2n block, a predicted block can obtain a 2m×2n or m×n block, and a transformed block can obtain a 2m×2n block. Based on the above settings, information such as block size and range (e.g., information about exponents, multiples, etc.) can be generated.

[0110] The range can be determined based on the type of block (in this example, based on the maximum and minimum values). Furthermore, some blocks can explicitly generate block range information, while others can implicitly determine it. For example, coded blocks and transform blocks can explicitly generate relevant information, while prediction blocks can implicitly process relevant information.

[0111] In the case of explicit specification, at least one range information can be generated. For example, in the case of an encoded block, information about the maximum and minimum values ​​can be generated as range information. Alternatively, it can be generated based on the difference between the maximum value and a preset minimum value (e.g., 8) (e.g., generating information about the exponential difference between the maximum and minimum values ​​according to the setting, etc.). In addition, information about multiple ranges of the horizontal and vertical lengths of a rectangular block can be generated.

[0112] When implied, range information can be obtained based on encoding / decoding settings (e.g., block type, segmentation method, segmentation settings, etc.). For example, in the case of a prediction block, maximum and minimum value information can be obtained from candidate groups (M×N and m / 2×n / 2 in this example), which can be obtained from the encoding block (e.g., the maximum size of the encoding block is M×N and the minimum size is m×n) as the higher-level unit, based on the segmentation settings of the prediction block (e.g., quadtree segmentation + segmentation depth 0).

[0113] The size and shape of the initial (or starting) block of the block segmentation can be determined based on the higher-level unit. In the case of a coded block, the basic coded block can be the initial block; in the case of a prediction block, the coded block can be the initial block; and in the case of a transform block, either the coded block or the prediction block can be the initial block, depending on the encoding / decoding settings. For example, in the intra encoding mode, the prediction block can be the higher-level unit of the transform block; in the inter encoding mode, the prediction block can be a unit independent of the transform block. The initial block, as the starting unit for segmentation, can be divided into smaller blocks. If the optimal segmentation size and shape of each block are determined, the corresponding block can be determined as the initial block of the lower-level unit. For example, the former can be a coded block, and the latter (lower-level unit) can be a prediction block or a transform block. As in the example above, if the initial block of the lower-level unit is determined, the segmentation process of finding the block with the optimal size and shape can be performed in the same way as the higher-level unit.

[0114] In summary, the block partitioning unit can partition a basic coding unit (or the largest coding unit) into at least one coding unit (or a lower-level coding unit). Furthermore, a coding unit can be partitioned into at least one prediction unit and at least one transform unit. A coding unit can be partitioned into at least one coding block, a coding block can be partitioned into at least one prediction block, and at least one transform block. A prediction unit can be partitioned into at least one prediction block, and a transform unit can be partitioned into at least one transform block.

[0115] As in the example above, when searching for blocks of optimal size and shape through a pattern determination process, pattern information (e.g., segmentation information) about the blocks is generated. This pattern information, along with information generated by the constituent parts to which the blocks belong (e.g., prediction-related information, transformation-related information, etc.), can be included in the bitstream and transmitted to the decoder, which can then parse it at the same level for use in the video decoding process.

[0116] The following example illustrates the segmentation method, using a square initial block as an example. However, a similar application can be achieved when the initial block is rectangular.

[0117] Figure 3 This is an example diagram representing a tree-based block structure.

[0118] 'a' represents a 2N×2N instance obtained without splitting, 'b' represents two 2N×N instances obtained through a partial split flag (horizontal split of a binary tree in this example), 'c' represents two N×2N instances obtained through a partial split flag (vertical split of a binary tree in this example), and 'd' represents four N×N instances obtained through a partial split flag (quad split of a quadtree in this example). The shape of the obtained blocks can be determined based on the type of tree used in the splitting. For example, in the case of performing a quadtree split, the candidate blocks that can be obtained are 'a' and 'd'. In the case of performing a binary tree split, the candidate blocks that can be obtained are 'a', 'b', and 'c'. Quadtrees support one split flag, where 'a' is obtained when the flag is '0' and 'd' is obtained when the flag is '1'. Binary trees support multiple split flags, one of which can be a flag indicating whether to perform a split, and another flag indicating whether to perform a horizontal / vertical split. Quadtrees can be a basic tree-based partitioning method. In addition, tree partitioning methods (binary trees in this example) can be included within tree-based partitioning. Multiple tree partitions can be performed if a flag allowing additional tree partitions is implicitly or explicitly activated. Tree-based partitioning can be a method capable of recursive partitioning. That is, the partitioned block can be set as the initial block again for tree-based partitioning, determined by partitioning settings such as the partitioning range and allowed partitioning depth. This type of partitioning can be an example of hierarchical partitioning.

[0119] Figure 4 This is an illustrative diagram showing the shapes of various blocks that can be obtained from the block division section of the present invention.

[0120] Reference Figure 4 It can obtain blocks from a to s according to the segmentation settings and segmentation method, and can also realize other block forms not shown in the figure.

[0121] As an example (1), tree-based partitioning allows for asymmetric partitioning. For example, in the case of a binary tree, blocks such as b and c, or blocks such as b to g, can be implemented. When the flag allowing asymmetric partitioning is explicitly or implicitly disabled according to the encoding / decoding settings, the candidate blocks that can be obtained are b or c. When the flag allowing asymmetric partitioning is activated, the candidate blocks that can be obtained are b, d, and e (horizontal partitioning in this example), or c, f, and g (vertical partitioning in this example).

[0122] In the above example, the length ratio of left:right or top:bottom of asymmetric segmentation is 1:3 or 3:1, but it is not limited to this. Candidate groups with other ratios may also be included depending on the encoding settings (e.g., 1:2, 1:4, 2:3, 2:5, 3:5, etc.).

[0123] Next, various examples of segmentation information generated in binary tree segmentation (candidate groups of 1:1, 1:3, and 3:1 in this example) are illustrated.

[0124] For example, a flag indicating the segmentation pattern can be generated from the flags indicating whether segmentation is performed and the flag indicating the segmentation direction. In this case, the segmentation pattern can refer to symmetrical or asymmetrical segmentation. Specifically, when asymmetrical segmentation is determined as the segmentation pattern, a flag indicating the segmentation ratio is generated, and an index can be assigned based on preset candidate groups. If the candidate groups support a segmentation ratio of 1:3 or 3:1, the segmentation ratio can be selected using a 1-bit flag.

[0125] Alternatively, a flag indicating the segmentation ratio can be generated separately from the flag indicating whether segmentation is performed and the flag indicating the segmentation direction. In this example, the candidate group for the segmentation ratio can be composed of candidates with a symmetrical ratio of 1:1.

[0126] In this invention (where the flag for asymmetric partitioning is enabled), it is assumed that the binary tree partitioning has the configuration shown in the example above. Unless otherwise specified, the binary tree refers to a symmetric binary tree.

[0127] As an example (2), tree-based partitioning allows for other tree partitioning. For example, partitioning of ternary trees, quad-type trees, octa trees, etc., can be performed, resulting in n partition blocks (3, 4, 8 in this example, where n is an integer). The number of blocks supported by a ternary tree (in this example, partitioning into multiple blocks) can be h to m, the number of blocks supported by a quad-type tree can be n to p, and the number of blocks supported by an octa tree can be q. Whether the tree-based partitioning is supported can be implicitly determined or explicitly generated based on the encoding / decoding settings. In addition, it can be used alone or in combination with binary tree partitioning, quad-type tree partitioning, etc., depending on the encoding / decoding settings.

[0128] For example, in the case of a binary tree, blocks such as b and c can be implemented. In the case of a mixed use of binary and ternary trees (in this example, it is assumed that the scope of the binary tree usage partially overlaps with the scope of the ternary tree usage), blocks such as b, c, i, and l can be implemented. If the flags that allow other splits besides the existing tree splits are explicitly or implicitly prohibited according to the encoding / decoding settings, the candidate blocks that can be obtained can be b or c. If activated, the candidate blocks that can be obtained can be b, i or b, h, i, j (horizontal split in this example), or c, l or c, k, l, m (vertical split in this example).

[0129] In the above example, it is assumed that the length ratio of the left:middle:right or top:middle:bottom of the ternary tree segment is 2:1:1, 1:2:1 or 1:1:2, but it is not limited to this and may have other ratios depending on the encoding settings.

[0130] Next, an example of the segmentation information generated in ternary tree segmentation (a 1:2:1 candidate in this example) is explained.

[0131] For example, a flag indicating the type of split can be generated in addition to the flag indicating whether a split is performed and the flag indicating the direction of the split. In this case, the type of split can refer to binary tree split or ternary tree split.

[0132] In this invention, adaptive encoding / decoding settings can be applied according to the segmentation method.

[0133] As an example, the segmentation method can be determined based on the type of block. For instance, the coding block and the transform block can be segmented using a quadtree, while the prediction block can be segmented using a quadtree and a binary tree (or ternary tree, etc.).

[0134] As an example, the segmentation method can be determined based on the size of the block. For instance, between the maximum and minimum values ​​of a block, a quadtree segmentation can be performed on a portion of the range (e.g., a×b to c×d, where the latter is larger), and a binary tree (or ternary tree, etc.) segmentation can be performed on another portion (e.g., e×f to g×h). In this case, the range information corresponding to the segmentation method can be explicitly generated or implicitly determined, and the ranges may overlap.

[0135] As an example, the segmentation method can be determined based on the shape of the block (or the block before segmentation). For instance, if the block is square, quadtree segmentation and binary tree (or ternary tree, etc.) segmentation can be performed. Alternatively, if the block is rectangular, binary tree (or ternary tree, etc.) based segmentation can be performed.

[0136] As an example, the segmentation settings can be determined based on the type of block. For instance, in tree-based segmentation, the coded block and the prediction block can be segmented using a quadtree, while the transform block can be segmented using a binary tree. Alternatively, in the case of the coded block, the allowed segmentation depth can be set to m; in the case of the prediction block, the allowed segmentation depth can be set to n; and in the case of the transform block, the allowed segmentation depth can be set to o. m, n, and o can be the same or different.

[0137] As an example, the partitioning settings can be determined based on the size of the block. For instance, a portion of the block's range (e.g., a×b~c×d) can be partitioned using a quadtree, a portion using a binary tree (e.g., e×f~g×h, where in this example c×d is assumed to be greater than g×h), and a portion using a ternary tree (e.g., i×j~k×l, where in this example g×h is assumed to be greater than or equal to k×l). In this case, the range can encompass all values ​​between the maximum and minimum values ​​of the block, and these ranges can be configured to either not overlap or overlap. For example, the minimum value of one range can be the same as the maximum value of another range, or the minimum value of one range can be less than the maximum value of another range. In the case of overlapping ranges, the partitioning method with the larger maximum value can be prioritized, or information about which partitioning method to use can be explicitly generated. That is, in partitioning methods with a priority order, the decision to execute the later-ranked partitioning method can be determined based on the partitioning results, or the partitioning method selection information can be used to determine which partitioning method to use.

[0138] As an example, the partitioning settings can be determined based on the shape of the block. For instance, if the block is square, a quadtree partition can be performed. Alternatively, if the block is rectangular, a binary tree partition or a ternary tree partition can be performed.

[0139] As an example, the segmentation settings can be determined based on encoding / decoding information (e.g., segment type, color composition, encoding mode, etc.). For instance, when the segment type is I, a range (e.g., a×b~c×d) can be segmented using a quadtree (or binary tree, ternary tree); when the segment type is P, a range (e.g., e×f~g×h) can be segmented using a quadtree (or binary tree, ternary tree); and when the segment type is B, a range (e.g., i×j~k×l) can be segmented using a quadtree (or binary tree, ternary tree). Furthermore, when the segment type is I, the allowed segmentation depth for quadtree (or binary tree, ternary tree) segmentation can be set to m; when the segment type is P, the allowed segmentation depth can be set to n; and when the segment type is B, the allowed segmentation depth can be set to o. m, n, and o can be the same or different. Some segment types can have the same settings as other segments (e.g., segments P and B).

[0140] As another example, when the color component is a luminance component, the allowable segmentation depth of the quadtree (or binary tree, ternary tree) can be set to m; when the color component is a chromaticity component, the allowable segmentation depth of the quadtree (or binary tree, ternary tree) can be set to n. m and n can be the same or different. Furthermore, the segmentation range of the quadtree (or binary tree, ternary tree) when the color component is a luminance component (e.g., a×b~c×d) can be the same or different from the segmentation range of the quadtree (or binary tree, ternary tree) when the color component is a chromaticity component (e.g., e×f~g×h).

[0141] As another example, in the intra-frame encoding mode, the allowed segmentation depth of a quadtree (or binary tree, ternary tree) can be m. In the inter-frame encoding mode, the allowed segmentation depth of a quadtree (or binary tree, ternary tree) can be n (in this example, we assume n is greater than m). m and n can be the same or different. Furthermore, the segmentation range of a quadtree (or binary tree, ternary tree) in the intra-frame encoding mode can be the same or different from that in the inter-frame encoding mode.

[0142] In the above example, information about whether to support the formation of adaptive segmentation candidate groups corresponding to the encoding / decoding information can be generated explicitly or implicitly.

[0143] The above examples illustrate how segmentation methods and settings can be determined based on encoding / decoding settings. These examples represent a subset of cases corresponding to various elements and can be modified to suit other scenarios. Furthermore, segmentation methods and settings can also be determined based on a combination of multiple elements. For example, segmentation methods and settings can be determined based on block type, size, shape, encoding / decoding information, etc.

[0144] Additionally, in the above examples, elements related to the segmentation method and segmentation settings can be implicitly determined, or information can be explicitly generated to determine whether an adaptive situation like the one in the above examples is allowed.

[0145] In the segmentation settings, the segmentation depth refers to the number of times the initial block is segmented spatially (in this example, the initial block's segmentation depth is 0). A deeper segmentation depth allows for smaller segments. The depth-related settings can be changed depending on the segmentation method. For example, in tree-based segmentation, different depths can be used depending on the type of tree. For instance, the segmentation depth for a binary tree and a ternary tree can use the same depth, while the segmentation depth for a quadtree and a binary tree can use different depths, and so on.

[0146] In the example above, when using different split depths depending on the type of tree, the split depth at the starting position of the tree (the block before splitting in this example) can be set to 0. The split depth can be calculated centered on the starting position of the split, rather than based on the split range of each tree (the maximum value in this example).

[0147] Figure 5 This illustrates a block partitioning process according to an embodiment of the present invention. More specifically, Figure 5 The text represents an example of the size and shape of a block that can be obtained from a basic coded block using more than one segmentation method.

[0148] In the diagram, thick solid lines represent basic coding blocks, thick dashed lines represent quadtree partitioning boundaries, double solid lines represent symmetric binary tree partitioning boundaries, solid lines represent ternary tree partitioning boundaries, and thin dashed lines represent asymmetric binary tree partitioning boundaries. Except for the thick solid lines, the remaining lines represent boundaries defined according to various partitioning methods. The partitioning settings described below (e.g., partitioning types, partitioning information, order of partitioning information composition, etc.) are not limited to this example and can be implemented in various variations.

[0149] For ease of explanation, let's assume that the basic coding block (2N×2N, 128×128) is used as a base, with different block partitioning settings in the top-left, top-right, bottom-left, and bottom-right blocks (N×R, 64×64). First, let's assume that the initial block is partitioned once (partitioning depth 0→1, i.e., the partitioning depth increases by 1), resulting in 4 sub-blocks. As for the quadtree partitioning settings, the maximum coding block size is 128×128, the minimum coding block size is 8×8, and the maximum partitioning depth is 4, which are common settings for all blocks.

[0150] (No. 1, top left block, A0~A6)

[0151] This example supports single-tree segmentation (quadtree in this case). The size and shape of the obtainable block can be determined based on block segmentation settings such as maximum coding block, minimum coding block, and segmentation depth. In this example, the block obtainable through segmentation is one (dividing it horizontally and vertically into two parts). The segmentation information required for one segmentation operation (based on the original block of 4M×4N, with the segmentation depth increased by 1) is a flag indicating whether to perform segmentation (in this example, if it is 0, no segmentation is performed; if it is 1, segmentation is performed). The obtainable candidates are 4M×4N and 2M×2N.

[0152] (No. 2, top right block, A7~A11)

[0153] This example supports multi-way tree partitioning (quadtree and binary tree in this case), and the size and shape of the blocks that can be obtained can be determined by multiple block partitioning settings. In this example, assuming a binary tree, the maximum coded block is 64×64, the length of one side of the minimum coded block is 4, and the maximum partitioning depth is 4.

[0154] In this example, there are two or more blocks that can be obtained through splitting (2 or 4 in this example). The splitting information required to perform a splitting action (increasing the quadtree splitting depth by 1) is a flag indicating whether to split, a flag indicating the type of split, a flag indicating the splitting shape, and a flag indicating the splitting direction. The possible candidates are 4M×4N, 4M×2N, 2M×4N, 4M×N / 4M×3N, 4M×3N / 4M×N, M×4N / 3M×4N, and 3M×4N / M×4N.

[0155] When the segmentation range of a quadtree overlaps with that of a binary tree (i.e., both quadtree and binary tree segmentation can be performed in the current step) and the current block (the state before segmentation) is a block obtained through quadtree segmentation (a block obtained from the parent block <where the segmentation depth is 1 less than the current block> through quadtree segmentation), the segmentation information can be categorized as follows. That is, when blocks supported by various segmentation settings can be obtained through multiple segmentation methods, the process for generating segmentation information can be divided into the following steps.

[0156] (1) Cases where quadtree partitioning and binary tree partitioning overlap.

[0157] [Table 1]

[0158] a b c d e QT 1 No Split 0 0 SBT hor 0 1 0 0 ABT hor 1 / 4 0 1 0 1 0 ABT hor 3 / 4 0 1 0 1 1 SBT ver 0 1 1 0 ABT ver 1 / 4 0 1 1 1 0 ABT ver 3 / 4 0 1 1 1 1

[0159] In the table above, 'a' is a flag indicating whether to perform a quadtree split. If it is 1, a quadtree split (QT) is performed. If the flag is 0, then the flag indicating whether to perform a binary tree split (b) is confirmed. If b is 0, no split is performed on the corresponding block (No Split); if it is 1, a binary tree split is performed.

[0160] `c` indicates the direction of the segmentation. A value of 0 indicates a horizontal segmentation (hor), and a value of 1 indicates a vertical segmentation (ver). `d` indicates the segmentation pattern. A value of 0 indicates a symmetrical segmentation (SBT, Symmetric Binary Tree), and a value of 1 indicates an asymmetrical segmentation (ABT, Asymmetric Binary Tree). Only when `d` is 1 is the information regarding the fineness ratio (1 / 4 or 3 / 4) in asymmetrical segmentation confirmed. When `d` is 0, in the left / right or top / bottom blocks, the left and top blocks have a 1 / 4 ratio, and the right and bottom blocks have a 3 / 4 ratio. When `d` is 1, the ratios are reversed.

[0161] (2) Cases where only binary tree partitioning is possible

[0162] In the table above, the segmentation information can be represented using the labels b through e, excluding a.

[0163] exist Figure 5 In the above, block A7 is the case where quadtree splitting can be performed in the block (A7~A11) before splitting (that is, although quadtree splitting can be performed, binary tree splitting is performed instead of quadtree splitting), so it is equivalent to generating the splitting information in (1).

[0164] Conversely, A8 to A11 are cases where a quadtree segmentation is no longer performed in the block (A8~A11) before the segmentation, but a binary tree segmentation is achieved (i.e., the block <A8~A11> cannot be segmented into a quadtree), which is equivalent to the case of generating the segmentation information in (2).

[0165] (No. 3, bottom left block, A12~A15)

[0166] This example supports multi-tree segmentation (quadtree, binary tree, and ternary tree in this case), and the size and shape of the blocks that can be obtained can be determined based on multiple block segmentation settings. In this example, assuming a binary / ternary tree case, the maximum coded block size is 64×64, the minimum coded block size has a side length of 4, and the maximum segmentation depth is 4.

[0167] In this example, more than two blocks can be obtained through segmentation (2, 3, and 4 blocks in this example). The segmentation information required for one segmentation action is a flag indicating whether segmentation is to be performed, a flag indicating the type of segmentation, and a flag indicating the direction of segmentation. The possible candidates are 4M×4N, 4M×2N, 2M×4N, 4M×N / 4M×2N / 4M×N, and M×4N / 2M×4N / M×4N.

[0168] When the segmentation range of a quadtree overlaps with the segmentation range of a binary / ternary tree, and the current block is a block obtained by segmenting a quadtree, the segmentation information can be divided into the following cases.

[0169] (1) Cases where quadtree partitioning overlaps with binary / tritree partitioning

[0170] [Table 2]

[0171] a b c d QT 1 No Split 0 0 BT hor 0 1 0 0 TT hor 0 1 0 1 Bt ver 0 1 1 0 TT ver 0 1 1 1

[0172] In the table above, 'a' is a flag indicating whether to perform quadtree splitting. If it is 1, quadtree splitting is performed. If the flag is 0, then the flag 'b' indicates whether to perform binary tree splitting or ternary tree splitting. If 'b' is 0, the corresponding block is not split; if it is 1, binary tree splitting or ternary tree splitting is performed.

[0173] c is a flag indicating the direction of the split. If it is 0, it indicates a horizontal split; if it is 1, it indicates a vertical split. d is a flag indicating the type of split. If it is 0, it indicates a binary tree split (BT); if it is 1, it indicates a ternary tree split (TT).

[0174] (2) Cases where only binary / ternary tree partitioning is possible

[0175] In the table above, the segmentation information can be represented using the labels b through d, excluding a.

[0176] exist Figure 5 In the above, blocks A12 and A15 are cases where quadtree partitioning can be implemented in the blocks (A12~A15) before partitioning, which is equivalent to generating the partitioning information in (1).

[0177] Conversely, A13 and A14 are cases where quadtree splitting is no longer performed in the blocks (A13, A14) before splitting, but ternary tree splitting is achieved instead, which is equivalent to the case of generating the splitting information in (2).

[0178] (No. 4, bottom left block, A16~A20)

[0179] This example supports multi-tree segmentation (quadtree, binary tree, and ternary tree in this case), and the size and shape of the blocks that can be obtained can be determined based on multiple block segmentation settings. In this example, assuming a binary / ternary tree case, the maximum coded block size is 64×64, the minimum coded block size has a side length of 4, and the maximum segmentation depth is 4.

[0180] In this example, more than two blocks can be obtained through segmentation (2, 3, and 4 blocks in this example). The segmentation information required for one segmentation action is a flag indicating whether segmentation is performed, a flag indicating the type of segmentation, a flag indicating the segmentation form, and a flag indicating the segmentation direction. The possible candidates are 4M×4N, 4M×2N, 2M×4N, 4M×N / 4M×3N, 4M×3N / 4M×N, M×4N / 3M×4N, 3M×4N / M×4N, 4M×N / 4M×2N / 4M×N, and M×4N / 2M×4N / M×4N.

[0181] When the segmentation range of a quadtree overlaps with the segmentation range of a binary / ternary tree, and the current block is a block obtained by segmenting a quadtree, the segmentation information can be divided into the following cases.

[0182] (1) Cases where quadtree partitioning overlaps with binary / tritree partitioning

[0183] [Table 3]

[0184] a b c d e f QT 1 No Split 0 0 TT hor 0 1 0 0 SBT hor 0 1 0 1 0 ABT hor 1 / 4 0 1 0 1 1 0 ABT hor 3 / 4 0 1 0 1 1 1 TT ver 0 1 1 0 SBT ver 0 1 1 1 0 ABT ver 1 / 4 0 1 1 1 1 0 ABT ver 3 / 4 0 1 1 1 1 1

[0185] In the table above, 'a' is a flag indicating whether to perform quadtree splitting. If it is 1, quadtree splitting is performed. If the flag is 0, then the flag indicating whether to perform binary tree splitting, 'b', is confirmed. If 'b' is 0, the corresponding block is not split; if it is 1, binary tree splitting or ternary tree splitting is performed.

[0186] `c` is a flag indicating the direction of the split; 0 indicates a horizontal split, and 1 indicates a vertical split. `d` is a flag indicating the type of split; 0 indicates a ternary split, and 1 indicates a binary tree split. When `d` is 1, the flag regarding the split shape, `e`, is confirmed. When `e` is 0, a symmetrical split is performed; when `e` is 1, an asymmetrical split is performed. When `e` is 1, information regarding the fineness ratio in asymmetrical splits is confirmed, as in the example above.

[0187] (2) Cases where only binary / ternary tree partitioning is possible

[0188] In the table above, the segmentation information can be represented using the labels b through f, excluding a.

[0189] exist Figure 5 In this context, block A20 is a case where quadtree partitioning can be performed on the blocks (A16-A19) before partitioning, which is equivalent to generating the partitioning information in (1).

[0190] Conversely, A16 to A19 represent cases where a binary tree split is achieved instead of a quadtree split in the blocks (A16 to A19) before the split, which is equivalent to generating the split information in (2).

[0191] The following describes the inter-frame prediction method based on merge mode. In merge mode, motion information can be derived on a block-by-block basis. Here, a block can refer to a coded block determined by the block segmentation. Alternatively, a block as a leaf node can refer to a block that cannot be further subdivided into smaller coded blocks. In the merge mode, motion information can be derived on a sub-block basis belonging to a block. For this purpose, a block can be further subdivided into multiple sub-blocks. For example, affine motion modeling can be performed based on the merge mode of the sub-block units. The current block can selectively utilize either the block-level merge mode or the sub-block-level merge mode, for which specific flags can be used. Here, the flags can be information indicating whether the sub-block-level merge mode can be applied. These flags can be encoded and transmitted in the encoding device based on specific encoding parameters, or derived from the decoding device. The encoding parameters can refer to the block size / shape, component type, segmentation type, segmentation depth, etc. For example, if the current block size is greater than or equal to N×M, the flag can be passed; otherwise, the flag can be derived as a predefined value. Here, N and M can be integers of 8, 16, or higher. N and M can be the same or different. See reference. Figure 6 The block unit merging mode is explained in the following description. Figures 7 to 10 The merging mode of sub-block units is explained.

[0192] Figure 6 As an embodiment of the present invention, it is a diagram illustrating a method for performing inter-frame prediction based on a block-unit merging mode.

[0193] Reference Figure 6 This can form a candidate list for merging the current block (S600).

[0194] The merge candidate list may contain at least one of spatial merge candidates or temporal merge candidates for the current block.

[0195] Motion information for spatial merging candidates can be derived based on the motion information of spatially adjacent blocks of the current block. Here, a spatially adjacent block, as a block belonging to the same image as the current block, refers to a block adjacent to the current block. A spatially adjacent block may include a block adjacent to at least one of the following: the left side, top edge, top right edge, bottom left edge, or top left edge of the current block. The top left adjacent block can only be utilized if at least one of the blocks adjacent to the left side, top edge, top right edge, and bottom left edge is unavailable.

[0196] Motion information for temporal merging candidates can be derived based on the motion information of temporally adjacent blocks of the current block. A temporally adjacent block, belonging to an image different from the current block, can be defined as a block at the same position as the current block. Here, a block at the same position can refer to at least one of the following: a block adjacent to the lower right corner of the current block (BR), a block containing the central sample of the current block (CTR), or a block containing the upper left sample of the current block (TL). Alternatively, a block at the same position can also refer to a block containing a position offset from the upper left sample of the current block by a specific disparity vector. Here, the disparity vector can be determined based on any one of the motion vectors of the spatially adjacent blocks. Alternatively, the disparity vector can be determined based on a combination of at least two motion vectors of the spatially adjacent blocks. The combination can refer to operations such as maximum value, minimum value, median value, weighted average value, etc. For example, the disparity vector can be set based on the motion vectors of the left-side adjacent block or the upper-side adjacent block. Alternatively, the disparity vector can be set based on the median or average value between the motion vectors of the left-side adjacent block and the motion vector of the lower left adjacent block.

[0197] The motion vectors and reference image indices of the time merging candidates can be derived separately based on the motion vectors and reference image indices of the temporally adjacent blocks. Alternatively, the motion vectors of the time merging candidates can be derived based on the motion vectors of the temporally adjacent blocks, and the reference image indices of the time merging candidates can be set independently of the temporally adjacent blocks based on a preset default value (e.g., 0) in the decoding device.

[0198] The merge candidate list can also contain combined merge candidates. Combined merge candidates can be derived by combining n merge candidates that belong to the generated merge candidate list.

[0199] Here, n can be an integer of 2, 3, 4, or higher. The number (n) of merge candidates as combination objects can also be a fixed value preset in the encoding / decoding device, or it can be encoded and transmitted in the encoding device. The transmission can be performed in at least one unit of sequence, image, segment, block, sub-block (brick), or specific block. The number (n) of merge candidates as combination objects can also be variably determined based on the number of remaining merge candidates. Here, the number of remaining merge candidates can refer to the difference between the maximum number of merge candidates that can be included in the merge candidate list and the current number of merge candidates belonging to the merge candidate list. The maximum number can be a preset number in the encoding / decoding device, or it can be encoded and transmitted in the encoding device. The current number can refer to the number of merge candidates formed before adding the combined merge candidates. For example, when the number of remaining merge candidates is 1, two merge candidates can be used, and when the number of remaining merge candidates is greater than 1, three or more merge candidates can be used.

[0200] The positions of the n merge candidates can be preset positions in the merge candidate list. For example, each merge candidate in the merge candidate list can be assigned an index (0 to (k-1)). Here, k can refer to the total number of merge candidates in the merge candidate list. In this case, the positions of the n merge candidates can correspond to indices 0 to (n-1) in the merge candidate list. Alternatively, the prediction direction of each merge candidate in the merge candidate list can be considered to determine the n merge candidates. For example, only merge candidates with bidirectional prediction or only merge candidates with unidirectional prediction can be selectively used among the merge candidates in the merge candidate list.

[0201] Merge candidates can be derived using both spatial and temporal merge candidates, or only using either one. For example, it can be restricted to deriving merge candidates using only spatial merge candidates. In this case, the number of merge candidates that can be combined can be limited to the number of spatial merge candidates that belong to the already generated merge candidate list.

[0202] The combined merge candidate can be appended to the merge candidate list after the spatial / temporal merge candidate. That is, the index of the combined merge candidate can be greater than the index of the spatial / temporal merge candidate. Alternatively, the combined merge candidate can also be appended to the merge candidate list between the spatial and temporal merge candidates. That is, the index of the combined merge candidate can be greater than the index of the spatial merge candidate and less than the index of the temporal merge candidate. Alternatively, the position of the combined merge candidate can be variably determined by considering the prediction direction of the combined merge candidate. The position of the combined merge candidate in the merge candidate list can be rearranged according to whether the prediction direction of the combined merge candidate is bidirectional. For example, if the combined merge candidate is bidirectional, it can be assigned an index less than that of the spatial or temporal merge candidate; otherwise, it can be assigned an index greater than that of the spatial or temporal merge candidate.

[0203] The following section explains, for ease of explanation, the method of deriving combined merge candidates based on two merge candidates.

[0204] Motion information of the combined merging candidates can be derived by weighted averaging of the motion information of the first and second merging candidates. Here, the weighting values ​​for the weighted average are [1:1], [1:2], [1:3], [2:3], etc., but are not limited to these. The weighting values ​​can be predefined in the encoding / decoding device or derived from the decoding device. In this case, at least one of the distance between the current image and the reference image of the merging candidate or the prediction direction of the merging candidate can be considered to derive the weighting values. Alternatively, the motion information of the combined merging candidates can be derived by obtaining motion information in the L0 direction from the first merging candidate, obtaining motion information in the L1 direction from the second merging candidate, and combining the motion information. The motion information of the combined merging candidates can be derived based on at least one of the derived methods, as described below, by considering the prediction direction of the merging candidate as the object of combination.

[0205] In this specification, motion information may include at least one of a predicted direction marker, a reference image index, or a motion vector. The motion information may be defined separately for L0 prediction and L1 prediction. Here, L0 prediction may refer to a prediction made with reference to a list of reference images L0, and L1 prediction may refer to a prediction made with reference to a list of reference images L1.

[0206] 1. The case where both the first and second merge candidates are unidirectional predictions.

[0207] (Example 1) When the first merging candidate is an L0 prediction and the second merging candidate is an L1 prediction, the reference image index of the combined merging candidate in the L0 direction can be derived from the reference image index of the first merging candidate. The prediction direction flag of the combined merging candidate in the L0 direction can be derived as 1. The motion vector of the combined merging candidate in the L0 direction can be derived from the motion vector of the first merging candidate. The reference image index of the combined merging candidate in the L1 direction can be derived from the reference image index of the second merging candidate. The prediction direction flag of the combined merging candidate in the L1 direction can be derived as 1. The motion vector of the combined merging candidate in the L1 direction can be derived from the motion vector of the second merging candidate.

[0208] (Example 2) When the first merging candidate is an L1 prediction and the second merging candidate is an L0 prediction, the reference image index of the combined merging candidate in the L0 direction can be derived from the reference image index of the second merging candidate. The prediction direction flag of the combined merging candidate in the L0 direction can be derived as 1. The motion vector of the combined merging candidate in the L0 direction can be derived from the motion vector of the second merging candidate. The reference image index of the combined merging candidate in the L1 direction can be derived from the reference image index of the first merging candidate. The prediction direction flag of the combined merging candidate in the L1 direction can be derived as 1. The motion vector of the combined merging candidate in the L1 direction can be derived from the motion vector of the first merging candidate.

[0209] (Example 3) When both the first and second merge candidates are L0 predictions, the reference image index of the combined merge candidate in the L0 direction can be derived from the reference image index of either the first or second merge candidate. For example, the reference image index of the merge candidate with the smallest index among the first and second merge candidates can be set as the reference image index of the combined merge candidate in the L0 direction. The prediction direction flag of the combined merge candidate in the L0 direction can be derived as 1. The motion vector of the combined merge candidate in the L0 direction can be derived from the weighted average of the motion vectors of the first and second merge candidates. The reference image index of the combined merge candidate in the L1 direction can be derived as -1, the prediction direction flag in the L1 direction can be derived as 0, and the motion information in the L1 direction can be derived as 0.

[0210] (Example 4) When both the first and second merge candidates are L1 predicted, the reference image index for the L0 direction of the combined merge candidate can be derived as -1, the prediction direction flag for the L0 direction can be derived as 0, and the motion information for the L0 direction can be derived as 0. The reference image index for the L1 direction of the combined merge candidate can be derived from the reference image index of either the first or second merge candidate. For example, the reference image index of the merge candidate with the smallest index among the first and second merge candidates can be set as the reference image index for the L1 direction of the combined merge candidate. The prediction direction flag for the L1 direction of the combined merge candidate can be derived as 1. The motion vector for the L1 direction of the combined merge candidate can be derived from the weighted average of the motion vectors of the first and second merge candidates.

[0211] 2. Case where both the first and second merge candidates are bidirectional predictions.

[0212] (Example 5) The reference image index for the L0 direction of the combined merging candidate can be derived from the reference image index of either the first or second merging candidate. For example, the reference image index of the merging candidate with the smallest index among the first and second merging candidates can be set as the reference image index for the L0 direction of the combined merging candidate. The predicted direction flag for the L0 direction of the combined merging candidate can be derived as 1. The motion vector for the L0 direction of the combined merging candidate can be derived from the weighted average of the motion vectors of the first and second merging candidates. The reference image index for the L1 direction of the combined merging candidate can be derived from the reference image index of either the first or second merging candidate. For example, the reference image index of the merging candidate with the smallest index among the first and second merging candidates can be set as the reference image index for the L1 direction of the combined merging candidate. The predicted direction flag for the L1 direction of the combined merging candidate can be derived as 1. The motion vector for the L1 direction of the combined merging candidate can be derived from the weighted average of the motion vectors of the first and second merging candidates.

[0213] 3. The case where the first merge candidate is a two-way prediction and the second merge candidate is a one-way prediction.

[0214] (Example 6) When the second merging candidate is predicted as L0, the reference image index of the combined merging candidate in the L0 direction can be derived based on the reference image index of either the first or second merging candidate. For example, the reference image index of the merging candidate with the smallest index among the first and second merging candidates can be set as the reference image index of the combined merging candidate in the L0 direction. The prediction direction flag of the combined merging candidate in the L0 direction can be derived as 1. The motion vector of the combined merging candidate in the L0 direction can be derived based on the weighted average of the motion vectors of the first and second merging candidates. The reference image index of the combined merging candidate in the L1 direction can be derived based on the reference image index of the first merging candidate. The prediction direction flag of the combined merging candidate in the L1 direction can be derived as 1. The motion vector of the combined merging candidate in the L1 direction can be derived based on the motion vector of the first merging candidate.

[0215] (Example 7) When the second merging candidate is an L1 prediction, the reference image index of the combined merging candidate in the L0 direction can be derived from the reference image index of the first merging candidate. The prediction direction flag of the combined merging candidate in the L0 direction can be derived as 1. The motion vector of the combined merging candidate in the L0 direction can be derived from the motion vector of the first merging candidate. The reference image index of the combined merging candidate in the L1 direction can be derived from the reference image index of either the first or second merging candidate. For example, the reference image index of the merging candidate with the smallest index among the first and second merging candidates can be set as the reference image index of the combined merging candidate in the L1 direction. The prediction direction flag of the combined merging candidate in the L1 direction can be derived as 1. The motion vector of the combined merging candidate in the L1 direction can be derived from the weighted average of the motion vectors of the first and second merging candidates.

[0216] Reference Figure 6 The motion information of the current block can be exported based on the candidate list for merging (S610).

[0217] Specifically, the merge index of the current block can be passed. The merge index can specify a particular merge candidate from a list of merge candidates. That is, merge candidates with the same index as the merge index can be extracted, and the motion information of the current block can be derived using the motion information of the extracted merge candidates.

[0218] Reference Figure 6 The derived motion information can be used to perform inter-frame prediction for the current block (S620).

[0219] Specifically, a reference image can be selected from a list of reference images based on the reference image index of the current block. A reference block can be identified within the reference image based on the motion vector of the current block. The reconstructed pixels of the identified reference block can be used to generate the predicted pixels of the current block. The current block can be reconstructed by adding the predicted pixels of the current block to the residual pixels. Here, at least one of inverse quantization or inverse transform can be performed on the residual coefficients transmitted via the bitstream to derive the residual pixels.

[0220] Figure 7 As an embodiment of the present invention, it is illustrated by a diagram showing an inter-frame prediction method based on an affine model.

[0221] Reference Figure 7 It can generate a candidate list for predicting the motion information of the current block (S700).

[0222] The candidate list may contain one or more affine model-based candidates (hereinafter referred to as affine candidates). An affine candidate may refer to a candidate that has a control point vector. The control point vector is the motion vector used for the control points of the affine model, and the control point vector can be defined for the corner position of the block (e.g., at least one of the top left corner, top right corner, bottom left corner, or bottom right corner).

[0223] Affine candidates may include at least one of spatial candidates, temporal candidates, or a combination of both. Here, spatial candidates can be derived from the vectors of spatially adjacent blocks, and temporal candidates can be derived from the vectors of temporally adjacent blocks. Here, the adjacent blocks may refer to blocks encoded based on an affine model. The vectors may refer to motion vectors or control point vectors.

[0224] Reference Figure 8 The method for deriving spatial / temporal candidates based on vectors of spatial / temporal neighboring blocks is explained in detail.

[0225] On the other hand, the candidates can be derived based on the combination of motion vectors between spatially / temporally adjacent blocks of the current block, referring to... Figure 9 This situation will be explained in detail.

[0226] The plurality of affine candidates can be arranged in the candidate list based on a specific priority order. For example, the plurality of affine candidates can be arranged in the candidate list in the order of spatial candidates, temporal candidates, and the combined candidates. Alternatively, the plurality of affine candidates can also be arranged in the candidate list in the order of temporal candidates, spatial candidates, and the combined candidates. However, this is not a limitation; temporal candidates can also be placed after the combined candidates. Alternatively, they can be arranged such that some of the combined candidates are placed before the spatial candidates, and the remaining ones are placed after the spatial candidates.

[0227] The candidate list may also include temporal candidates based on sub-blocks. Motion information of these temporal candidates can be derived from the collocated block corresponding to the current block, as shown in the reference. Figure 10 This situation will be explained in detail.

[0228] The control point vector of the current block can be derived based on the candidate list and candidate index (S710).

[0229] A candidate index can refer to an index encoded for deriving the control point vector of the current block. The candidate index can specify one of multiple affine candidates belonging to a candidate list. The control point vector of the current block can be derived using the control point vector of the affine candidate specified by the candidate index.

[0230] For example, suppose the affine model of the current block is a four-parameter affine model (i.e., determined to use two control point vectors for the current block). In this case, if the affine candidate specified by the candidate index has three control point vectors, only two of the three control point vectors (e.g., control point vectors with Idx = 0 and 1) can be selected and set as the control point vectors of the current block. Alternatively, all three control point vectors of the specified affine candidate can be set as the control point vectors of the current block. In this case, the affine model type of the current block can be updated to a six-parameter affine model.

[0231] Conversely, suppose the affine model of the current block is a six-parameter affine model (i.e., determined to use 3 control point vectors for the current block). In this case, if the affine candidate identified by the candidate index has two control point vectors, an additional control point vector can be generated, and the two control point vectors of the affine candidate and the additional control point vector are set as the control point vectors of the current block. The additional control point vector can be derived based on at least one of the two control point vectors of the affine candidate, the size or position information of the current / neighboring block.

[0232] Alternatively, the two control point vectors of the specific affine candidate can be set as the control point vectors of the current block. In this case, the type of the affine model of the current block can be updated to a four-parameter affine model.

[0233] The motion vector of the current block can be derived based on the control point vector of the current block (S720).

[0234] The motion vector can be derived on a per-sub-block basis. For this purpose, the current block can be divided into multiple N×M sub-blocks. Here, the N×M sub-blocks can be rectangular (N>M or N<M) or square (N=M). The values ​​of N and M can be 4, 8, 16, 32, or higher.

[0235] For example, the size / shape of a sub-block can be a fixed size / shape defined in the decoding device. For example, the size / shape of a sub-block can be a square such as 4×4, 8×8, 16×16, or a rectangle such as 2×4, 2×8, 4×8, 4×16. Alternatively, the sub-block can also be defined as a block whose width and height sum to an integer of 8, 12, 16, or higher. Alternatively, the sub-block can also be defined as a block whose width and height product is an integer of 16, 32, 64, or higher.

[0236] Alternatively, the size / shape of the sub-blocks can be variably derived based on the properties of the block. For example, if the size of the current block is greater than or equal to a certain threshold size, the current block can be divided into units of first sub-blocks (e.g., 8×8, 16×16); otherwise, the current block can be divided into units of second sub-blocks (e.g., 4×4).

[0237] Alternatively, information about the size / shape of a sub-block can be encoded and transmitted in an encoding device. This information represents at least one of the following: sub-block size or shape, and can be transmitted at at least one level: sequence, image, block group, block, CTU.

[0238] The control point representing the current block may include the upper left and upper right positions. However, it is not limited to this; the control point may also include three positions: the upper left, the upper right, and the lower left, or multiple other positions.

[0239] When using two control points, the motion vectors of each sub-block of the current block can be derived using at least one of the first control point vector corresponding to the first control point, the second control point vector corresponding to the second control point, the position (x, y) of the sub-block, or the size (width or height) of the current block.

[0240] When using three control points, the motion vectors of each sub-block of the current block can be derived using at least one of the following: the first control point vector corresponding to the first control point, the second control point vector corresponding to the second control point, the third control point vector corresponding to the third control point, the position (x, y) of the sub-block, or the size (width or height) of the current block. In this case, the difference vector between the second and first control point vectors can be used, as well as the difference vector between the third and first control point vectors. These difference vectors can be calculated separately for the horizontal direction (x-axis direction) and the vertical direction (y-axis direction).

[0241] The motion vector derivation process may further include a specific offsetting process on the motion vector derived based on the control point vector. The offset may refer to a vector used to improve the derived motion vector. The offset may be determined based on at least one of the magnitude or direction. The magnitude (absolute) may be an integer of 1, 2, 3, or higher. The direction may include at least one of left, right, top, or bottom directions. The magnitude and / or direction information of the offset may be encoded and transmitted in the encoding device. Alternatively, the magnitude of the offset may be a preset fixed value in the decoding device.

[0242] The derived motion vectors can be used to perform inter-frame prediction on the current block (S730).

[0243] Specifically, a reference block can be identified using the motion vector of the current block. The reference block can be identified separately for each sub-block of the current block. Each sub-block's reference block can belong to a single reference image. That is, sub-blocks belonging to the current block can share a single reference image. Alternatively, a reference image index can be set individually for each sub-block of the current block.

[0244] On the other hand, the derived motion vector may include at least one of the L0 motion vector or the L1 motion vector. If the motion vector includes both L0 and L1 motion vectors, the decoding device may also set either the L0 or L1 motion vector to 0 to perform unidirectional prediction.

[0245] The settings can be selectively executed by considering at least one of the block size or prediction mode. Here, a block can refer to the current block or a sub-block of the current block. The prediction mode can refer to skip mode, merge mode, AMVP (Advanced Motion Vector Prediction) mode, affine model-based prediction mode, current image reference mode, etc.

[0246] For example, if the size of the current block (or a sub-block of the current block) is less than or equal to a specific threshold size, the L1 motion vector can be set to 0 to perform unidirectional prediction; otherwise, both the L0 and L1 motion vectors can be used to perform bidirectional prediction. Here, the threshold size can be defined as a block size with at least one of a width and a height of 4, 8, or 16.

[0247] Alternatively, if the current block has been encoded using the first prediction mode, the L1 motion vector can be set to 0 to perform unidirectional prediction; otherwise, the L0 and L1 motion vectors can be used to perform bidirectional prediction.

[0248] The specified reference block can be set as the prediction block for the current block. The current block can be reconstructed by adding the prediction block to the residual block.

[0249] Figure 8 As an embodiment of the present invention, it is illustrated by a diagram showing a method for deriving affine candidates based on the control point vectors of spatially / temporally adjacent blocks.

[0250] The width and height of the current block 800 are cbW and cbH, respectively, and its position is (xCb, yCb). The width and height of the spatially adjacent blocks 810-850 are nbW and nbH, respectively, and their positions are (xNb, yNb). A spatially adjacent block may include at least one of the following: the left-hand block 810, the lower-left block 840, the upper-right block 830, the upper block 820, or the upper-left block 850 of the current block. Alternatively, the spatially adjacent block may also include at least one of the blocks adjacent to the right side of the upper-left block 850 or the block adjacent to its lower side.

[0251] A spatial candidate may have n control point vectors (cpMV). Here, the value of n can be an integer of 1, 2, 3, or higher. The value of n can be determined based on at least one of the following: information about whether decoding can be performed on a sub-block basis, information about whether it is a block encoded based on an affine model, or information about the type of affine model (four-parameter or six-parameter).

[0252] For example, based on the information, if the corresponding block can be decoded at the sub-block level or if the corresponding block is a block encoded based on an affine model, the block will have two control point vectors. Otherwise, affine model-based prediction may not be performed on the block.

[0253] Alternatively, based on the information, if the corresponding block is an affine model-based block and the affine model type is a six-parameter affine model, the block may have three control point vectors. Otherwise, the block may not perform affine model-based prediction.

[0254] The information can also be encoded and transmitted in an encoding device. Alternatively, all or part of the information can be derived from a decoding device based on the attributes of the block. Here, a block can refer to the current block or its spatially / temporally adjacent blocks. The attributes can refer to parameters such as size, shape, position, segmentation type, inter-frame mode, and residual coefficients. The inter-frame mode, as a mode defined in the decoding device, can refer to merge mode, skip mode, AMVP mode, affine model, intra / inter-frame combination mode, current image reference mode, etc. Alternatively, the value of n can be derived from the decoding device based on the attributes of the block.

[0255] In this embodiment, the n control point vectors can be represented as the first control point vector (cpMV[0]), the second control point vector (cpMV[1]), the third control point vector (cpMV[2]), ..., the nth control point vector (cpMV[n-1]).

[0256] As an example, the first control point vector (cpMV[0]), the second control point vector (cpMV[1]), the third control point vector (cpMV[2]), and the fourth control point vector (cpMV[3]) can be vectors corresponding to the positions of the upper left, upper right, lower left, and lower right samples of the block, respectively. Here, it is assumed that the spatial candidate has 3 control point vectors, which can be any control point vectors selected from the first control point vector to the nth control point vector. However, it is not limited to this; the spatial candidate can have two control point vectors, which can be any control point vectors selected from the first control point vector to the nth control point vector.

[0257] The following explains the method for deriving the spatial candidate control point vectors.

[0258] 1. The case where the boundary of the current block does not connect with the boundary of the CTU.

[0259] The first control point vector can be derived based on at least one of the following: the first control point vector of spatially adjacent blocks, a specific difference value, the position information of the current block (xCb, yCb), or the position information of spatially adjacent blocks (xNb, yNb).

[0260] The number of difference values ​​can be one, two, three, or more. The number of difference values ​​can be variably determined considering the attributes of the block, or it can be a fixed value preset in the decoding device. Each difference value can be defined as the difference between any one of a plurality of control point vectors and another control point vector. For example, the difference value can include at least one of the following: a first difference value between a second control point vector and a first control point vector; a second difference value between a third control point vector and a first control point vector; a third difference value between a fourth control point vector and a third control point vector; or a fourth difference value between a fourth control point vector and a second control point vector.

[0261] For example, the first control point vector can be derived using the following formula 1.

[0262] [Formula 1]

[0263] cpMvLX[0][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb-yNb))

[0264] cpMvLX[0][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb-yNb))

[0265] In Equation 1, variables mvScaleHor and mvScaleVer can refer to the first control point vector of spatially adjacent blocks, or to values ​​derived by performing a shift operation of k on the first control point vector. Here, k can be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9, or higher. Variables dHorX and dVerX correspond to the x and y components of the first difference value between the second and first control point vectors, respectively. Variables dHorY and dVerY correspond to the x and y components of the second difference value between the third and first control point vectors, respectively. These variables can be derived through the following Equation 2.

[0266] [Formula 2]

[0267] mvScaleHor=CpMvLX[xNb][yNb][0][0]<<7

[0268] mvScaleVer=CpMvLX[xNb][yNb][0][1]<<7

[0269] dHorX=(CpMvLX[xNb+nNbW-1][yNb][1][0]-CpMvLX[xNb][yNb][0][0])<<(7-log2NbW)

[0270] dVerX=(CpMvLX[xNb+nNbW-1][yNb][1][1]-CpMvLX[xNb][yNb][0][1])<<(7-log2NbW)

[0271] dHorY=(CpMvLX[xNb][yNb+nNbH-1][2][0]-CpMvLX[xNb][yNb][2][0])<<(7-log2NbH)

[0272] dVerY=(CpMvLX[xNb][yNb+nNbH-1][2][1]-CpMvLX[xNb][yNb][2][1])<<(7-log2NbH)

[0273] The second control point vector can be derived based on at least one of the following: a first control point vector of spatially adjacent blocks, a specific difference value, the position information of the current block (xCb, yCb), the block size (width or height), or the position information of spatially adjacent blocks (xNb, yNb). Here, the block size can refer to the size of the current block and / or spatially adjacent blocks. The difference value is the same as that described in the first control point vector, and detailed description is omitted here. However, the range and / or number of difference values ​​used in deriving the second control point vector may differ from those in the first control point vector.

[0274] For example, the second control point vector can be derived using the following equation 3.

[0275] [Formula 3]

[0276] cpMvLX[1][0]=(mvScaleHor+dHorX*(xCb+cbWidth-xNb)+dHorY*(yCb-yNb))

[0277] cpMvLX[1][1]=(mvScaleVer+dVerX*(xCb+cbWidth-xNb)+dVerY*(yCb-yNb))

[0278] In equation 3, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are the same as in equation 1, and detailed explanations are omitted here.

[0279] A third control point vector can be derived based on at least one of the following: a first control point vector of spatially adjacent blocks, a specific difference value, the position information of the current block (xCb, yCb), the block size (width or height), or the position information of spatially adjacent blocks (xNb, yNb). Here, the block size can refer to the size of the current block and / or spatially adjacent blocks. The difference value is the same as that described in the first control point vector, and detailed description is omitted here. However, the range and / or number of difference values ​​used in deriving the third control point vector may differ from those of the first or second control point vector.

[0280] For example, the third control point vector can be derived using the following equation 4.

[0281] [Formula 4]

[0282] cpMvLX[2][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb+cbHeight-yNb))

[0283] cpMvLX[2][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb+cbHeight-yNb))

[0284] In Equation 4, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are the same as in Equation 1, and detailed explanations are omitted here. On the other hand, the spatial candidate vector of the nth control point can be derived through the above process.

[0285] 2. The case where the current block boundary is adjacent to the CTU boundary.

[0286] The first control point vector can be derived based on at least one of the motion vectors (MV) of spatially adjacent blocks, a specific difference value, the position information of the current block (xCb, yCb), or the position information of spatially adjacent blocks (xNb, yNb).

[0287] The motion vector can be the motion vector of the lowest sub-block located in a spatially adjacent block. The sub-block can be the leftmost, center, or rightmost sub-block among multiple sub-blocks located at the lowest point of a spatially adjacent block. Alternatively, the motion vector can also refer to the average, maximum, or minimum value of the motion vectors of the sub-blocks.

[0288] The number of difference values ​​can be one, two, three, or more. The number of difference values ​​can be variably determined considering the attributes of the block, or it can be a fixed value preset in the decoding device. The difference value can be defined as the difference between any one motion vector and another motion vector among multiple motion vectors stored in sub-blocks within a spatially adjacent block. For example, the difference value can refer to the difference between the motion vector of the lower right terminal block and the motion vector of the lower left terminal block of a spatially adjacent block.

[0289] For example, the first control point vector can be derived using the following formula 5.

[0290] [Formula 5]

[0291] cpMvLX[0][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb-yNb))

[0292] cpMvLX[0][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb-yNb))

[0293] In Equation 5, the variables mvScaleHor and mvScaleVer can refer to the motion vector (MV) of the spatially adjacent block or the value derived by performing a shift operation of k on the motion vector. Here, k can be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9 or higher.

[0294] The variables dHorX and dVerX correspond to the x and y components of a specific difference value, respectively. Here, the difference value refers to the difference between the motion vector of the lower right terminal block and the motion vector of the lower left terminal block in a spatially adjacent block. The variables dHorY and dVerY can be derived based on the variables dHorX and dVerX. These variables can be derived using the following equation 6.

[0295] [Formula 6]

[0296] mvScaleHor=MvLX[xNb][yNb+nNbH-1][0]<<7

[0297] mvScaleVer=MvLX[xNb][yNb+nNbH-1][1]<<7

[0298] dHorX=(MvLX[xNb+nNbW-1][yNb+nNbH-1][0]-MvLX[xNb][yNb+nNbH-1][0])<<(7-log2NbW)

[0299] dVerX=(MvLX[xNb+nNbW-1][yNb+nNbH-1][1]-MvLX[xNb][yNb+nNbH-1][1])<<(7-log2NbW)

[0300] dHorY = -dVerX

[0301] dVerY=dHorX

[0302] The second control point vector can be derived based on at least one of the following: motion vectors (MV) of spatially adjacent blocks, specific difference values, the position information of the current block (xCb, yCb), the block size (width or height), or the position information of spatially adjacent blocks (xNb, yNb). Here, the block size can refer to the size of the current block and / or spatially adjacent blocks. The motion vectors and difference values ​​are the same as those described in the first control point vector, and detailed descriptions are omitted here. However, the position of the motion vectors, the range of difference values, and / or the number of difference values ​​used in deriving the second control point vector may differ from those in the first control point vector.

[0303] For example, the second control point vector can be derived using the following formula 7.

[0304] [Formula 7]

[0305] cpMvLX[1][0]=(mvScaleHor+dHorX*(xCb+cbWidth-xNb)+dHorY*(yCb-yNb))

[0306] cpMvLX[1][1]=(mvScaleVer+dVerX*(xCb+cbWidth-xNb)+dVerY*(yCb-yNb))

[0307] In equation 7, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are the same as in equation 5, and detailed explanations are omitted here.

[0308] The third control point vector can be derived based on at least one of the following: motion vectors (MV) of spatially adjacent blocks, specific difference values, the position information of the current block (xCb, yCb), the block size (width or height), or the position information of spatially adjacent blocks (xNb, yNb). Here, the block size can refer to the size of the current block and / or spatially adjacent blocks. The motion vectors and difference values ​​are the same as those described in the first control point vector, and detailed descriptions are omitted here. However, the position of the motion vectors, the range of difference values, and / or the number of difference values ​​used in deriving the third control point vector may differ from those of the first or second control point vectors.

[0309] For example, the third control point vector can be derived using the following formula 8.

[0310] [Formula 8]

[0311] cpMvLX[2][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb+cbHeight-yNb))

[0312] cpMvLX[2][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb+cbHeight-yNb))

[0313] In Equation 8, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are the same as in Equation 5, and detailed explanations are omitted here. On the other hand, the spatial candidate vector of the nth control point can be derived through the above process.

[0314] The process of deriving the affine candidate can be performed on each of the defined spatially adjacent blocks. The defined spatially adjacent blocks may include at least one of the left block, lower left block, upper right block, upper block, or upper left block of the current block.

[0315] Alternatively, the process of deriving the affine candidates can be performed according to the groups of spatially adjacent blocks. Here, the spatially adjacent blocks can be divided into a first group containing the left-side block and the lower-left block, and a second group containing the upper-right block, the upper block, and the upper-left block.

[0316] For example, an affine candidate can be derived from spatially adjacent blocks belonging to the first group. This derivation can be performed based on a specific preferred order until a usable affine candidate is found. The preferred order can be left-side block → lower-left block, or vice versa. Based on this preferred order, it can be determined whether the corresponding spatially adjacent blocks within the first group are blocks decoded through affine model-based prediction, and the block decoded first through affine model-based prediction is selected as an affine candidate.

[0317] Similarly, an affine candidate can be derived from the spatially adjacent blocks belonging to the second group. This derivation can be performed based on a specific preferred order until a usable affine candidate is found. The preferred order can be upper right block → upper block → upper left block, or the reverse order. Based on this preferred order, it can be determined whether the corresponding spatially adjacent blocks within the second group are blocks decoded through affine model-based prediction, and the block decoded first through affine model-based prediction is selected as an affine candidate.

[0318] The above embodiments can also be applied in the same or similar way to temporally adjacent blocks. Here, a temporally adjacent block belongs to an image different from the current block, but can be a block at the same position as the current block. A block at the same position can be a block containing the position of the upper left sample of the current block, the center position, or the position of a sample adjacent to the lower right sample of the current block.

[0319] Alternatively, a temporally adjacent block can refer to a block that is displaced by a specific displacement vector from the block at the same location. Here, the displacement vector can be determined based on the motion vector of any spatially adjacent block among the spatially adjacent blocks of the current block.

[0320] Figure 9 As an embodiment of the present invention, it is illustrated by a diagram showing a method for deriving the constructed candidate based on the combination of motion vectors of spatial / temporal adjacent blocks.

[0321] The candidate vectors constructed in this invention can be derived based on a combination of at least two control point vectors from the control point vectors (hereinafter referred to as control point vectors (cpMVCorner[n])) corresponding to each corner of the current block. Here, n can be 0, 1, 2, or 3.

[0322] The control point vector (cpMVCorner[n]) can be derived based on the motion vectors of spatially adjacent blocks and / or temporally adjacent blocks. Here, a spatially adjacent block may include at least one of the following: a first adjacent block (A, B, or C) adjacent to the upper-left sample of the current block; a second adjacent block (D or E) adjacent to the upper-right sample of the current block; or a third adjacent block (F or G) adjacent to the lower-left sample of the current block. A temporally adjacent block, as a block belonging to an image different from the current block, may refer to a block at the same position as the current block (hereinafter referred to as the fourth adjacent block (Col)). Here, the fourth adjacent block may refer to a block (H, I, or J) containing the position of the upper-left sample, upper-right sample, or lower-left sample of the current block, or a block adjacent to the position of the lower-right sample of the current block.

[0323] The first adjacent block can refer to the block adjacent to the upper left (A), upper (B), or left (C) end of the current block. The usability of the motion vectors of adjacent blocks A, B, and C can be determined according to a specific preferred order, and the control point vector is determined using the motion vectors of the available adjacent blocks. The usability determination can continue until an adjacent block with a usable motion vector is found. Here, the preferred order can be A→B→C. However, it is not limited to this; the order can also be A→C→B, C→A→B, or B→A→C.

[0324] The second adjacent block can refer to the block adjacent to the top (D) or top right (E) of the current block. Similarly, the availability of motion vectors for adjacent blocks D and E can be determined according to a specific preferred order, and the control point vector is determined using the motion vectors of the available adjacent blocks. The availability determination can be performed until an adjacent block with a usable motion vector is found. Here, the preferred order can be either D→E or E→D.

[0325] The third adjacent block can refer to the adjacent block to the left (F) or lower left (G) of the current block. Similarly, the availability of motion vectors for adjacent blocks can be determined according to a specific preferred order, and the control point vector is determined using the motion vectors of available adjacent blocks. The availability determination can be performed until an adjacent block with a usable motion vector is found. Here, the preferred order can be G→F or F→G.

[0326] For example, the first control point vector (cpMVCorner[0]) can be set based on the motion vector of the first adjacent block, the second control point vector (cpMVCorner[1]) can be set based on the motion vector of the second adjacent block, and the third control point vector (cpMVCorner[2]) can be set based on the motion vector of the third adjacent block. The fourth control point vector (cpMVCorner[3]) can be set based on the motion vector of the fourth adjacent block.

[0327] Alternatively, the first control point vector can be derived using the motion vectors of at least one of the first or fourth adjacent blocks, where the fourth adjacent block can be the block containing the position of the upper left sample (H). The second control point vector can be derived using the motion vectors of at least one of the second or fourth adjacent blocks, where the fourth adjacent block can be the block containing the position of the upper right sample (I). The third control point vector can be derived using the motion vectors of at least one of the third or fourth adjacent blocks, where the fourth adjacent block can be the block containing the position of the lower left sample (J).

[0328] Alternatively, any one of the control point vectors from the first to the fourth control point vector can be derived based on another control point vector. For example, a second control point vector can be derived by applying a specific offset vector to the first control point vector. The offset vector can be the difference vector between the third and first control point vectors, or it can be derived by applying a specific scale vector to the difference vector. The scale vector can be determined based on at least one of the width or height of the current block and / or adjacent blocks.

[0329] The K candidates (ConstK) constituted by the present invention can be determined by combining at least two control point vectors from the first to the fourth control point vectors. The value of K can be an integer of 1, 2, 3, 4, 5, 6, 7 or higher. The value of K can be derived based on information transmitted from the encoding device, or it can be a preset value in the decoding device. The information may include information indicating the maximum number of candidates that can be included in the candidate list.

[0330] Specifically, the first candidate (Const1) can be derived by combining the first control point vector to the third control point vector. For example, the first candidate (Const1) can have control point vectors as shown in Table 4 below. On the other hand, if the reference image information of the first adjacent block is the same as the reference image information of the second and third adjacent blocks, the control point vectors can also be constructed in a limited manner as shown in Table 4. Here, the reference image information can refer to the reference image index indicating the position of the reference image in the reference image list, or it can refer to the POC (picture order count) value indicating the output order.

[0331] [Table 4]

[0332] Idx Control point vector 0 cpMvCorner[0] 1 cpMvCorner[1] 2 cpMvCorner[2]

[0333] The second candidate (Const2) can be derived by combining the first, second, and fourth control point vectors. For example, the second candidate (Const2) can have control point vectors as shown in Table 5 below. On the other hand, if the reference image information of the first adjacent block is the same as the reference image information of the second and fourth adjacent blocks, the control point vectors can also be constructed in a limited manner as shown in Table 5. Here, the reference image information is as described above.

[0334] [Table 5]

[0335]

[0336] The third candidate (Const3) can be derived by combining the first, third, and fourth control point vectors. For example, the third candidate (Const3) can have control point vectors as shown in Table 6 below. On the other hand, if the reference image information of the first adjacent block is the same as the reference image information of the third and fourth adjacent blocks, the control point vectors can also be constructed in a limited manner as shown in Table 6. Here, the reference image information is as described above.

[0337] [Table 6]

[0338]

[0339] The fourth candidate (Const4) can be derived by combining the second, third, and fourth control point vectors. For example, the fourth candidate (Const4) can have control point vectors as shown in Table 7 below. On the other hand, if the reference image information of the second adjacent block is the same as that of the third and fourth adjacent blocks, it can also be configured in a limited manner as shown in Table 7. Here, the reference image information is as described above.

[0340] [Table 7]

[0341]

[0342] The fifth candidate (Const5) can be derived by combining the first and second control point vectors. For example, the fifth candidate (Const5) can have control point vectors as shown in Table 8 below. On the other hand, if the reference image information of the first adjacent block is the same as the reference image information of the second adjacent block, the control point vectors can also be constructed in a limited manner as shown in Table 8. Here, the reference image information is as described above.

[0343] [Table 8]

[0344] Idx Control point vector 1 cpMvCorner[0] 2 cpMvCorner[1]

[0345] The sixth candidate (Const6) can be derived by combining the first and third control point vectors. For example, the sixth candidate (Const6) can have control point vectors as shown in Table 9 below. On the other hand, if the reference image information of the first adjacent block is the same as the reference image information of the third adjacent block, the control point vectors can also be constructed in a limited manner as shown in Table 9. Here, the reference image information is as described above.

[0346] [Table 9]

[0347] Idx Control point vector Control point vector 1 cpMvCorner[0] cpMvCorner[0] 2 cpMvCorner[2] cpMvCorner[1]

[0348] In Table 9, cpMvCorner[1] can be a second control point vector derived based on the first control point vector and the third control point vector. The second control point vector can be derived based on at least one of the first control point vector, a specific difference value, or the size of the current / neighboring block. For example, the second control point vector can be derived using the following equation 9.

[0349] [Formula 9]

[0350] cpMvCorner[1][0]=(cpMvCorner[0][0]<<7)+((cpMvCorner[2][1]-cpMvCorner[0][1])<<(7+Log2(cbHeight / cbWidth)))

[0351] cpMvCorner[1][1]=(cpMvCorner[0][1]<<7)+((cpMvCorner[2][0]-cpMvCorner[0][0])<<(7+Log2(cbHeight / cbWidth)))

[0352] In the candidate list (ConstK) consisting of the above K candidates, the value of K does not limit the position or priority of the candidates in the candidate list.

[0353] In addition, the candidates formed by the first to the sixth group can all be included in the candidate list, or only a portion of them can be included in the candidate list.

[0354] For example, if it is determined that the current block uses 3 control point vectors, only the candidates generated by combining 3 control point vectors from the first to the fourth control point vectors can be used. If it is determined that the current block uses 2 control point vectors, the candidates generated by combining at least two control point vectors from the first to the fourth control point vectors can be used, or the candidates generated by combining only two control point vectors from the first to the fourth control point vectors can be used.

[0355] Alternatively, one could consider a maximum number of affine candidates that can be included in the candidate list, and then include only a portion of them in the candidate list. Here, the maximum number can be determined based on the maximum number information transmitted by the encoding device, or it can be variably determined by considering the attributes of the current block. In this case, the K value of the constructed candidate (ConstK) can also refer to the priority order in which it is inserted into the candidate list.

[0356] On the other hand, if the difference between two control point vectors in the constructed candidate list is less than a certain threshold, the constructed candidate may not be able to be inserted into the candidate list. The difference between the two control point vectors can be divided into a horizontal difference and a vertical difference. Here, the horizontal difference may refer to the difference between the first control point vector 910 and the second control point vector 920, and the vertical difference may refer to the difference between the first control point vector 910 and the third control point vector 930. The threshold may refer to 0 or a vector of a preset size in the encoding / decoding device.

[0357] Figure 10 As an embodiment of the present invention, it is a diagram illustrating a method for deriving motion information based on temporal candidates of sub-blocks.

[0358] The motion information of the temporal candidate of this invention can be derived from the motion information of the collocated block. The motion information can be derived on a sub-block basis.

[0359] Here, a juxtaposed block can be a block belonging to an image different from the current block 1010 (i.e., a juxtaposed image), or it can be a block 1040 at the same position as the current block, or a block 1030 at a position shifted from the current block's position according to a temporal vector. The temporal vector can be determined based on the motion vectors of the spatially adjacent blocks of the current block. A spatially adjacent block can refer to a block that is adjacent to at least one of the left, lower left, upper, upper right, or upper left ends of the current block. The temporal vector can be determined using only adjacent blocks at a preset position in the encoding / decoding device. For example, the preset position can be the left side 1011 or the upper end 1012, or both the left side 1011 and the upper end 1012. If there are multiple adjacent blocks on the left side, a block located at any of the lowermost, uppermost, or center positions among the multiple adjacent blocks can be used. If there are multiple adjacent blocks at the upper end, a block located at any of the leftmost, rightmost, or center positions among the multiple adjacent blocks can be used.

[0360] When deriving time candidates based on sub-blocks, the current block and / or juxtaposed block can be divided into multiple sub-blocks.

[0361] Here, the sub-block may have a fixed size / shape preset in the encoding / decoding device. For example, the sub-block may be represented as an N×M block, where N and M are integers of 4, 8, or higher. The sub-block may be a square (N=M) or a rectangle (N>M, N<M). The dimensions may be specified. Alternatively, the encoding device may encode and transmit information about the size / shape of the sub-block, and the decoding device may determine the size / shape of the sub-block based on the transmitted information.

[0362] Alternatively, the current block and / or juxtaposed blocks can be divided into multiple sub-blocks based on a specific number. Here, the number can be a fixed number preset in the encoding / decoding device, or it can be variably determined considering the block size / shape. Alternatively, the number can be determined based on the number information transmitted by the encoding device.

[0363] The following describes a method for deriving motion information of time candidates on a sub-block basis. The motion vectors of the time candidates can be set based on the motion vectors stored in each sub-block of the juxtaposed block. The reference image index of the time candidate can be set to a value already defined in the encoding / decoding device (e.g., 0). Alternatively, the reference image index of the time candidate can be set based on the reference image index of the juxtaposed block or the reference image index of the upper left terminal block in the sub-block. Alternatively, the reference image index of the time candidate can be set based on the reference image index stored in each sub-block, similar to the motion vector.

[0364] However, if an unusable sub-block exists within a juxtaposed block, the motion vector of the unusable sub-block can be replaced by the motion vector of an available sub-block within the juxtaposed block. Here, an available sub-block can refer to a block adjacent to any of the unusable sub-blocks on its left, right, top, or bottom. Alternatively, an available sub-block can be a block at a preset position in the encoding / decoding device. For example, the preset position can be the position of the lower right terminal block within the juxtaposed block, or the position of a sub-block containing the center position of the juxtaposed block. If the available sub-block exists, the sub-block-based time candidate can be added to the candidate list. Alternatively, if the sub-block at the preset position is available, the sub-block-based time candidate can be added to the candidate list.

[0365] Figure 11 As an embodiment of the present invention, it is a diagram illustrating the inter-frame prediction method under the ERP projection format.

[0366] Reference Figure 11 In the ERP projection format, there will be a situation where the boundary of the reference region specified by the motion vector overlaps with that of the reference image.

[0367] In this case, the boundary of the corresponding reference image can be expanded, and data adjacent to the boundary can be filled into the expanded area horizontally (see A). Alternatively, the boundary of the corresponding reference image can be expanded, and data adjacent to the opposite boundary (data that is continuous or related to the boundary) can be filled into the expanded area (see B). Alternatively, data in areas that do not exist can be filled using data adjacent to the opposite boundary (see C).

[0368] While A and B require expanding the data of the reference image, the computation process can be simplified. C, although it does not require additional memory due to expansion, may require additional computation.

[0369] In cases A and B, the size of the current image may differ from the size of the (enlarged) reference image. The process of obtaining the predicted pixels may include specific adjustments to account for this size difference.

[0370] When comparing only the horizontal length, the current image can have a range of 0 to (pic_width - 1), and the reference image (enlarged as shown on the right) can have a range of -L_offset to (pic_width - 1 + R_offset) or a range of 0 to (pic_width - 1 + L_offset + R_offset). In the former case, the predicted pixel can be obtained directly in the enlarged reference image based on the position obtained by adding the coordinates of the current block to the motion vector. In the latter case, the predicted pixel can be obtained at the readjusted position after adjusting the obtained position. Here, L_offset and R_offset can refer to the length of the horizontal enlargement. In this invention, it is assumed that the coordinates are calculated based on the range of the current image or the existing reference image, as in the past.

[0371] In case C, the position in the reference image can be calculated by adding the coordinates of the current block to the motion vector. It can be confirmed whether the calculated pixel position is outside the range of the reference image (<0, 0> to <pic_width - 1, pic_height - 1>). If the calculated pixel position is within the range of the reference image, the predicted pixel can be obtained at the calculated pixel position. Conversely, if the calculated pixel position is outside the range of the reference image, the position corresponding to the calculated position can be calculated at the opposite side (right side if left, left side if right) inside the corresponding boundary (image interior), and the predicted pixel can be obtained at the calculated corresponding position.

[0372] Inter-frame prediction on the reference image boundary can be performed based on at least one of the embodiments described above. The inter-frame prediction method on the reference image boundary can be a preset method in the encoding / decoding device, or it can be determined based on selection information of a specific embodiment described above and information related to the selected method (data processing method, enlargement information, etc.). The same applies to the embodiments described below.

[0373] Figure 12 and Figure 15 As an embodiment of the present invention, it is a diagram illustrating the inter-frame prediction method under the CMP projection format.

[0374] Reference Figure 12 In the CMP projection format, due to region-wise packing, there is three-dimensional spatial continuity between the upper and lower 3×1 surfaces. However, there may be no three-dimensional spatial continuity between the upper and lower 3×1 surfaces.

[0375] Similarly, situations can arise where the reference region, identified by the motion vector, overlaps with the boundary of the reference image or with a portion of the surface boundary. In case A, although it is inside the reference image, it is located on a portion of the surface boundary (lacking continuity in three-dimensional space, thus having low correlation). In case B, the reference region is located on the boundary of the reference image.

[0376] In particular, in case A, although it is indicated that the reference image is inside, the accuracy of inter-frame prediction is reduced because the reference region contains discontinuous surface boundaries.

[0377] In cases A and B, the position of the reference region in the reference image can be calculated by adding the coordinates of the current block to the motion vector. This can confirm whether at least a portion of the corresponding reference region is outside the scope of the reference image or whether it contains discontinuous boundaries within the reference image.

[0378] When the reference region is outside the scope of the reference image, a position that is continuous or related in three-dimensional space can be determined, and the predicted pixel can be obtained based on the determined position.

[0379] Conversely, if the reference region does not deviate from the scope of the reference image, it can be determined whether at least one pixel within the reference region pointed to by the motion vector of the current block belongs to more than two surfaces. Here, the determination process can be performed based on the pixel positions of the current block, such as the upper left, upper right, lower left, and lower right.

[0380] If at least one pixel within the reference region does not belong to more than two surfaces, a predicted pixel can be obtained based on the reference region. Otherwise, it can be determined whether the surface is continuous with the surface to which the current block belongs. In the case of reference region A', the upper region is continuous with the surface to which the current block belongs, while the lower region is not continuous with the surface to which the current block belongs.

[0381] Where the aforementioned continuity exists, the predicted pixel can be obtained from the reference region. Otherwise, the predicted pixel can be obtained from a region that is continuous or correlated in three-dimensional space.

[0382] The inter-frame prediction order is one example; various variations can be achieved by changing the order or altering a portion of the composition.

[0383] In cases C and D, the reference image can be enlarged. The enlarged region can be filled with data adjacent to the boundary along the horizontal / vertical direction, or filled with data from a correlated region. Case D is the same as described in ERP, so detailed explanation is omitted here. In case C, the reference region is located inside the reference image, similar to case A, but includes surface boundaries with lower correlation. Therefore, without generating an additional enlarged region inside the reference image, the accuracy of inter-frame prediction decreases. Figure 13 This will be explained in detail.

[0384] In cases C and D, a situation arises where the size of the current image differs from the size of the reference image due to its enlargement. In this case, the reference image can have a range of (-L_offset, -U_offset) to (pic_width - 1 + R_offset, pic_height - 1 + D_offset). Here, L_offset, R_offset, U_offset, and D_offset can refer to the length of the enlargement in the reference image in the left, right, up, and down directions, respectively. These offsets can have the same value or different values. The length of the enlargement based on the reference image can be explicitly stated or implied.

[0385] For example, a length that can be supported by the block segmentation unit, such as 2, 4, 8, etc., can be supported. Alternatively, the extended length can be determined based on the minimum coded block, maximum coded block, minimum prediction block, maximum prediction block, etc. Or, it can be determined in units such as blocks, bricks, etc.

[0386] For example, the prediction can be expanded according to the length of the maximum prediction block. In this case, prediction blocks for all blocks can be obtained within the expanded range. If a situation arises where the region pointed to by the motion vector of the current block still falls outside the range of the expanded reference image even after the reference image is expanded, prediction pixels for the portion corresponding to the fallen range can be obtained by filling in the horizontal, vertical, and diagonal directions.

[0387] Reference Figure 13 A through D can be equivalent to enlarging a reference image using one or more surface units. A and B are examples of enlarging a reference image (or sub-reference image) by combining surfaces with continuity. C and D are examples of enlarging a reference image individually using surface units. The data processing for the enlarged areas is the same as / similar to the embodiments described above. In the embodiments described above, there is a range of reference image, but in this example, the reference image is divided into two or more parts to enlarge each region (called sub-reference images), thus resulting in overlapping ranges.

[0388] In examples A and B, the enlarged sub-reference image located at the top has a range of (-L_offset, -U_offset) to (pic_width - 1 + R_offset, pic_height / 2 - 1 + D_offset), and the enlarged sub-reference image located at the bottom has a range of (-L_offset, pic_height / 2 - U_offset) to (pic_width - 1 + R_offset, pic_height - 1 + D_offset). It is assumed that the enlarged lengths to the left, right, top, and bottom are the same in each sub-reference image. However, the enlargement lengths in the left, right, top, and bottom directions can be set exactly the same as or different from the enlargement unit. Furthermore, the enlargement length can also be set individually according to the enlargement unit.

[0389] At this point, the coordinates of the range of the upper sub-reference image that is expanded downwards are the same as the coordinates inside the lower sub-reference image. However, the corresponding area of ​​the upper sub-reference image contains data that is correlated after expansion, while the lower sub-reference image contains the original data before expansion.

[0390] In summary, the reference image can have a range of (-L_offset, -U_offset) to (pic_width - 1 + R_offset, pic_height - 1 + D_offset). Normally, one pixel value should be assigned to each location. However, due to the expanded reference image that considers the characteristics of the entire video and is divided into more than two regions, some ranges may be assigned more than one pixel value. In this case, it will be required to determine which pixel value to set as the predicted pixel value at locations where more than one pixel value is assigned.

[0391] Reference Figure 14 In case E, after identifying which sub-reference image contains more regions, the data from the sub-reference image containing more regions is used to obtain the predicted value.

[0392] In case F, if the sub-reference image containing the current block contains a preset number of pixels or more, or contains a specific width, the data of the corresponding sub-reference image can be used. Otherwise, as in case E, the data of the sub-reference image containing a larger area can be used for processing.

[0393] G is the case where the data is uniformly contained within two sub-reference images. In this case, priority can be assigned to the sub-reference image containing the current block to use the data from that sub-reference image. This example can include detailed conditions for F and E.

[0394] In summary, when the region pointed to by the motion vector of the current block is the boundary between sub-reference images, the data from which to use can be determined based on factors such as the included area and whether it is the same sub-reference image as the current block. Inter-frame prediction can be performed by determining which sub-reference image data to use based on all or some of the above conditions through encoding / decoding settings.

[0395] In the example above, the reference image was divided into two parts to expand each region, but in this example, we will illustrate the case of generating more sub-reference images on a surface unit basis.

[0396] exist Figure 15 In this context, each surface unit is assumed to be a sub-reference image. Similarly, it is necessary to determine which pixel value to set as the predicted pixel value at locations where more than two pixel values ​​are assigned.

[0397] Reference Figure 15 In the case of H, data from a sub-reference image containing more regions can be used, or data from a sub-reference image containing the current block can be used.

[0398] For example, if a sub-reference image containing the current block is included by a preset number of pixels or a defined width, the data of that sub-reference image can be used. Alternatively, if data of a sub-reference image containing a larger area is used, the data of that sub-reference image can be used. However, if the included areas are the same, the data of the sub-reference image containing the current block can be used restrictively.

[0399] In case I, data from a sub-reference image containing a larger region can be used. This example supports sub-reference images in surface units, but the surface setup has the characteristic of discontinuity between the upper 3×1 region and the lower 3×1 region. Therefore, sub-reference images with continuous characteristics can be given priority over sub-reference images with discontinuity characteristics.

[0400] In case J, sub-reference image data containing more regions can be used. However, although this example shows a uniform region containing the sub-reference images, like in case I, the sub-reference image containing the current block (the right side in a 3×2) is not an overlapping object. Furthermore, the corresponding sub-reference image (the upper left and center in a 3×2) belongs to the same 3×1 region as the sub-reference image containing the current block. In this case, priority can be given to the sub-reference image closer to the sub-reference image containing the current block.

[0401] In summary, when the region pointed to by the motion vector of the current block is the boundary between sub-reference images, the data of which sub-reference image can be used can be determined based on factors such as the area contained and the priority of using the sub-reference image.

[0402] On the other hand, refer to Figures 7 to 10 A method for performing inter-frame prediction based on affine motion modeling is described, which can be applied in the same or similar way to the encoding / decoding of 360 video; detailed descriptions are omitted.

[0403] The method of this invention can be implemented in the form of program commands executable by various computer devices and recorded on a computer-readable medium. The computer-readable medium may contain program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the computer-readable medium may be program commands specifically designed and configured for this invention, or program commands known and used by those skilled in the art of computer software.

[0404] Examples of computer-readable media include hardware devices such as read-only memory (ROM), random access memory (RAM), and flash memory, specifically configured to store and execute program commands. Examples of program commands include not only machine language code produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter. The hardware device can be configured to operate via at least one software module to perform the actions of the present invention, and vice versa.

[0405] Furthermore, in the aforementioned methods or apparatus, their structure or function can be implemented in a form that combines all or part of them, or in a separate form.

[0406] The above description refers to preferred embodiments of the present invention. However, it should be understood that those skilled in the art can make various modifications and changes to the present invention without departing from the spirit and scope of the invention as set forth in the appended claims.

[0407] Industrial availability

[0408] This invention can be used to encode / decode video signals.

Claims

1. A method for decoding a video signal, the method comprising: Construct an affine candidate list for the current block, wherein the affine candidate list includes the constructed candidates and at least one of spatial candidates or temporal candidates, and the maximum number of affine candidates contained in the affine candidate list is determined based on the maximum number information parsed from the bit stream. Parse the affine candidate indexes from the bitstream; The control point vector of the current block is determined based on the affine candidate list and the affine candidate index; The motion vector of the current block is determined based on the control point vector of the current block; and Based on the motion vector of the current block, perform a prediction based on the affine pattern for the current block; The method further includes: Parse information about the type of affine mode from the bitstream, wherein the information indicates whether a four-parameter-based affine mode or a six-parameter-based affine mode is applied. The candidate configuration is determined based on a combination of at least two control point vectors; Specifically, when applying the four-parameter affine pattern, the candidate configuration is determined based on the combination of two control point vectors; or, When the six-parameter-based affine pattern is applied, the candidate configuration is determined based on the combination of three control point vectors.

2. The method according to claim 1, wherein, When the four-parameter-based affine pattern is applied, one affine candidate in the affine candidate list includes two control point vectors. Determining the control point vector of the current block based on the affine candidate list and the affine candidate index includes: Determine the two control point vectors indicated by the affine candidate index.

3. The method according to claim 1, wherein, When the six-parameter-based affine pattern is applied, one affine candidate in the affine candidate list includes three control point vectors; Determining the control point vector of the current block based on the affine candidate list and the affine candidate index includes: Determine the three control point vectors indicated by the affine candidate index.

4. The method according to claim 2, wherein, The two control point vectors include a first control point vector and a second control point vector. Determining the motion vector of the current block based on the control point vector of the current block includes: The motion vector of the current block is determined based on the first control point vector, the second control point vector, and the size of the current block.

5. The method according to claim 3, wherein, The three control point vectors include a first control point vector, a second control point vector, and a third control point vector; Determining the motion vector of the current block based on the control point vector of the current block includes: The motion vector of the current block is determined based on the first control point vector, the second control point vector, the third control point vector, and the size of the current block.

6. A video signal decoding device, comprising: A processor configured to perform the method as described in any one of claims 1 to 5.

7. A method for encoding a video signal, the method comprising: Construct an affine candidate list for the current block, wherein the affine candidate list includes candidates formed by the block and at least one of spatial candidates or temporal candidates, and write information about the maximum number of affine candidates contained in the affine candidate list into the bitstream. The control point vector of the current block is determined based on the affine candidates in the affine candidate list, wherein the affine candidates are indicated by the affine candidate index; The motion vector of the current block is determined based on the control point vector of the current block; Based on the motion vector of the current block, perform affine pattern-based prediction on the current block; and Encode the affine candidate index of the current block into a bitstream; The method further includes: Information about the type of affine mode is encoded into the bitstream, wherein the information indicates whether a four-parameter-based affine mode or a six-parameter-based affine mode is applied. The candidate configuration is determined based on a combination of at least two control point vectors; Specifically, when applying the four-parameter affine pattern, the candidate configuration is determined based on the combination of two control point vectors; or, When the six-parameter-based affine pattern is applied, the candidate configuration is determined based on the combination of three control point vectors.

8. The method according to claim 7, wherein, When the four-parameter-based affine pattern is applied, one of the affine candidates in the affine candidate list includes two control point vectors.

9. The method according to claim 7, wherein, in, When the six-parameter-based affine pattern is applied, one of the affine candidates in the affine candidate list includes three control point vectors.

10. The method according to claim 8, wherein, The two control point vectors include a first control point vector and a second control point vector. Determining the motion vector of the current block based on the control point vector of the current block includes: The motion vector of the current block is determined based on the first control point vector, the second control point vector, and the size of the current block.

11. The method according to claim 9, wherein, The three control point vectors include a first control point vector, a second control point vector, and a third control point vector; Determining the motion vector of the current block based on the control point vector of the current block includes: The motion vector of the current block is determined based on the first control point vector, the second control point vector, the third control point vector, and the size of the current block.

12. A video signal encoding device, comprising: A processor configured to perform the method according to any one of claims 7 to 11.

13. A non-transitory computer-readable storage medium storing a computer program and a bit stream thereon, characterized in that, When executed by a processor, the computer program implements the encoding method for the video signal according to any one of claims 7 to 11 to generate the bitstream.

Images