Video signal decoding and encoding method and storage medium

Abstract

The invention discloses a video signal decoding and encoding method and a storage medium. Specifically, a method for processing a video signal by using affine prediction may comprise the steps of: checking whether affine prediction is applied to a current block; when it is confirmed that the affine prediction is applied to the current block, acquiring at least one syntax element indicating a resolution of a motion vector difference for the affine prediction; deriving a control point motion vector of the current block based on the at least one syntax element; deriving a motion vector for each of a plurality of sub-blocks included in the current block based on the control point motion vector; and generating a prediction sample of the current block using the motion vector of each sub-block.

Description

[0001] This application is a divisional application of the original invention patent application with application number 201980045121.9 (International application number: PCT / KR2019 / 008087, application date: July 2, 2019, invention name: Method and device for processing video signals by using affine prediction). Technical Field

[0002] The present disclosure relates to a method and apparatus for processing a video signal using affine prediction, and more particularly, to a method and apparatus for processing a video signal by controlling the resolution of an affine motion vector used in the affine prediction. Background Art

[0003] Compression coding refers to a series of signal processing techniques used to transmit digitized information over communication lines, or to store information in a form suitable for storage media. Media including pictures, images, and audio can be targets for compression coding, and in particular, the technology used to perform compression coding on pictures is called video image compression.

[0004] It is generally believed that the next generation of video content has the characteristics of high spatial resolution, high frame rate and high dimensionality of scene representation. In order to process such content, it will lead to a sharp increase in memory storage capacity, memory access rate and processing power.

[0005] Therefore, there is a need to design a coding tool for efficiently processing next-generation video content. Summary of the Invention

[0006] Technical issues

[0007] The present disclosure aims to propose a method for controlling the resolution of affine motion vectors used in affine prediction to improve the accuracy of affine prediction.

[0008] Furthermore, an object of the present disclosure is to propose an entropy coding method that relies on unique statistics of a motion model when performing entropy coding on MVD, rather than a constant entropy coding method.

[0009] The technical objectives to be achieved in the present disclosure are not limited to the above-mentioned technical objectives, and other technical objectives not described above can be clearly understood by those skilled in the art in the art to which the present disclosure belongs based on the following description.

[0010] Technical Solution

[0011] In one aspect of the present disclosure, a method for processing a video signal using affine prediction may include the following steps: checking whether affine prediction is applied to a current block; if affine prediction is applied as a result of the checking, obtaining at least one syntax element indicating a resolution of a motion vector difference used in the affine prediction; deriving a control point motion vector of the current block based on the at least one syntax element; deriving a motion vector for each of a plurality of sub-blocks included in the current block based on the control point motion vector; and generating a prediction sample of the current block using the motion vector of each sub-block.

[0012] Preferably, the step of obtaining at least one syntax element may include the following steps: obtaining a first syntax element indicating whether the resolution of the motion vector difference is a preset default resolution; and when the resolution of the motion vector difference is not the default resolution, obtaining a second syntax element indicating the resolution of the motion vector difference among the remaining resolutions other than the default resolution.

[0013] Preferably, the default resolution may be pre-set to 1 / 4 pixel accuracy.

[0014] Preferably, each residual resolution may include at least one of integer pixel precision, 4 pixel precision, 1 / 8 pixel precision or 1 / 16 pixel precision.

[0015] Preferably, the step of deriving the control point motion vector may include the steps of: determining a resolution of the motion vector difference using at least one syntax element; and obtaining the motion vector difference based on the resolution of the motion vector difference.

[0016] Preferably, the step of obtaining the motion vector difference may include the following steps: obtaining a flag indicating whether the motion vector difference is greater than 0; and when the motion vector difference is greater than 0, obtaining a flag indicating whether the motion vector difference is greater than a predefined specific value.

[0017] Preferably, when the motion vector difference is greater than 0 and less than or equal to a predefined specific value, the motion vector difference may be binarized using an exponential Golomb code with an order of 1. When the motion vector difference is greater than the predefined specific value, the motion vector difference may be binarized using a truncated binarization method.

[0018] In another aspect of the present disclosure, a device for processing a video signal using affine prediction may include: an affine prediction mode recognition unit configured to check whether affine prediction is applied to a current block; a syntax element acquisition unit configured to obtain at least one syntax element indicating a resolution of a motion vector difference used in affine prediction if affine prediction is applied as a result of the check; a control point motion vector derivation unit configured to derive a control point motion vector of the current block based on the at least one syntax element; a sub-block motion vector derivation unit configured to derive a motion vector for each of a plurality of sub-blocks included in the current block based on the control point motion vector; and a prediction sample generation unit configured to generate a prediction sample of the current block using the motion vector of each sub-block.

[0019] Preferably, the syntax element acquisition unit can be configured to obtain a first syntax element indicating whether the resolution of the motion vector difference is a preset default resolution, and when the resolution of the motion vector difference is not the default resolution, obtain a second syntax element indicating the resolution of the motion vector difference among the remaining resolutions other than the default resolution.

[0020] Preferably, the default resolution may be pre-set to 1 / 4 pixel accuracy.

[0021] Preferably, each residual resolution may include at least one of integer pixel precision, 4 pixel precision, 1 / 8 pixel precision or 1 / 16 pixel precision.

[0022] Preferably, the control point motion vector deriving unit may be configured to determine a resolution of the motion vector difference using the at least one syntax element, and obtain the motion vector difference based on the resolution of the motion vector difference.

[0023] Preferably, the control point motion vector deriving unit may be configured to obtain a flag indicating whether the motion vector difference is greater than 0, and when the motion vector difference is greater than 0, obtain a flag indicating whether the motion vector difference is greater than a predefined specific value.

[0024] Preferably, when the motion vector difference is greater than 0 and less than or equal to a predefined specific value, the motion vector difference may be binarized using an exponential Golomb code with an order of 1. When the motion vector difference is greater than the predefined specific value, the motion vector difference may be binarized using a truncated binarization method.

[0025] Beneficial effects

[0026] According to one embodiment of the present disclosure, the accuracy of affine motion prediction can be improved and compression efficiency can be improved by controlling the motion vector accuracy of a control point used in affine prediction.

[0027] Furthermore, according to one embodiment of the present disclosure, encoding efficiency and compression performance may be improved by adaptively setting a binarization method for each partitioned MVD region.

[0028] Effects obtainable in the present disclosure are not limited to the above-described effects, and other technical effects not described above can be clearly understood by those skilled in the art to which the present disclosure pertains from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To help understanding the present disclosure, the accompanying drawings, which are included as a part of the detailed description, provide embodiments of the present disclosure and describe technical features of the present disclosure together with the detailed description.

[0030] Figure 1 is a schematic block diagram of an encoding apparatus in which encoding of a video / image signal is performed as an embodiment to which the present disclosure is applied.

[0031] Figure 2 is a schematic block diagram of a decoding apparatus in which decoding of a video / image signal is performed as an embodiment to which the present disclosure is applied.

[0032] Figure 3 is a diagram illustrating an example of a multi-type tree structure to which the present disclosure can be applied.

[0033] Figure 4 is a diagram illustrating a signaling mechanism for partition information of a quadtree having a nested multi-type tree structure as an embodiment to which the present disclosure can be applied.

[0034] Figure 5 1 is a diagram illustrating a method of splitting a CTU into multiple CUs based on a quadtree and a nested multi-type tree structure as an embodiment to which the present disclosure can be applied.

[0035] Figure 6 is a diagram illustrating a method for limiting ternary tree partitioning as an embodiment to which the present disclosure can be applied.

[0036] Figure 7 : is a diagram showing a redundant partitioning pattern that may occur in binary tree partitioning and ternary tree partitioning as an embodiment to which the present disclosure can be applied.

[0037] Figure 8 and Figure 9 2 is a diagram illustrating a video / image encoding method based on inter-frame prediction according to an embodiment of the present disclosure and an inter-frame prediction unit in an encoding device according to an embodiment of the present disclosure.

[0038] Figure 10 and Figure 112 is a diagram illustrating a video / image decoding method based on inter-frame prediction according to an embodiment of the present disclosure and an inter-frame prediction unit in a decoding device according to an embodiment of the present disclosure.

[0039] Figure 12 is a diagram for describing neighboring blocks used in a merge mode or a skip mode as an embodiment to which the present disclosure is applied.

[0040] Figure 13 is a flowchart illustrating a method for configuring a merge candidate list according to an embodiment to which the present disclosure is applied.

[0041] Figure 14 is a flowchart illustrating a method for configuring a merge candidate list according to an embodiment to which the present disclosure is applied.

[0042] Figure 15 An example of a motion model according to one embodiment of the present disclosure is shown.

[0043] Figure 16 An example of control point motion vectors for affine motion prediction according to one embodiment of the present disclosure is shown.

[0044] Figure 17 An example of a motion vector for each subblock of a block to which affine motion prediction according to one embodiment of the present disclosure has been applied is shown.

[0045] Figure 18 An example of neighboring blocks used for affine motion prediction in affine merging mode according to one embodiment of the present disclosure is shown.

[0046] Figure 19 An example of a block in which affine motion prediction is performed using a neighboring block to which affine motion prediction according to an embodiment of the present disclosure has been applied is shown.

[0047] Figure 20 is a diagram for describing a method of generating a merge candidate list using peripheral affine coding blocks according to one embodiment of the present disclosure.

[0048] Figure 21 and Figure 22 is a diagram for describing a method of configuring an affine merging candidate list using neighboring blocks encoded by affine prediction according to one embodiment of the present disclosure.

[0049] Figure 23 An example of neighboring blocks used for affine motion prediction in affine inter mode according to one embodiment of the present disclosure is shown.

[0050] Figure 24An example of neighboring blocks used for affine motion prediction in affine inter mode according to one embodiment of the present disclosure is shown.

[0051] Figure 25 and Figure 26 2 is a diagram illustrating a method of deriving motion vector candidates using motion information of neighboring blocks in an affine inter mode according to one embodiment of the present disclosure.

[0052] Figure 27 An example of a method of deriving an affine motion vector field in units of sub-blocks according to one embodiment of the present disclosure is shown.

[0053] Figure 28 A method of generating a prediction block and a motion vector in inter prediction to which the affine motion model according to an embodiment of the present disclosure has been applied is shown.

[0054] Figure 29 is a diagram illustrating a method of performing motion compensation based on a motion vector of a control point according to one embodiment of the present disclosure.

[0055] Figure 30 is a diagram illustrating a method of performing motion compensation based on motion vectors of control points in an irregular block according to one embodiment of the present disclosure.

[0056] Figure 31 is a diagram illustrating a method of performing motion compensation based on motion vectors of control points in an irregular block according to one embodiment of the present disclosure.

[0057] Figures 32 to 38 is a diagram illustrating a method of performing motion compensation based on a motion vector of a control point in an irregular block according to one embodiment of the present disclosure.

[0058] Figure 39 An overall coding structure for deriving motion vectors according to one embodiment of the present disclosure is shown.

[0059] Figure 40 An example of an MVD coding structure according to one embodiment of the present disclosure is shown.

[0060] Figure 41 An example of an MVD coding structure according to one embodiment of the present disclosure is shown.

[0061] Figure 42 An example of an MVD coding structure according to one embodiment of the present disclosure is shown.

[0062] Figure 43 An example of an MVD coding structure according to one embodiment of the present disclosure is shown.

[0063] Figure 44 is a diagram illustrating a method of deriving affine motion vector difference information according to one embodiment to which the present disclosure is applied.

[0064] Figure 45 2 is a diagram illustrating a coding structure of a motion vector difference according to an embodiment to which the present disclosure is applied.

[0065] Figure 46 is a diagram illustrating a method of deriving an affine motion vector based on accuracy information according to one embodiment of the present disclosure.

[0066] Figure 47 2 is a diagram illustrating a coding structure of a motion vector difference according to an embodiment to which the present disclosure is applied.

[0067] Figure 48 is a flowchart illustrating a method of generating an inter-frame prediction block based on affine prediction according to one embodiment to which the present disclosure is applied.

[0068] Figure 49 is a diagram illustrating an inter-frame prediction apparatus based on affine prediction according to one embodiment to which the present disclosure is applied.

[0069] Figure 50 A video encoding system to which the present disclosure is applied is shown.

[0070] Figure 51 is one embodiment to which the present disclosure is applied, and shows the structure of a content streaming system. DETAILED DESCRIPTION

[0071] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Figure 1 The detailed description disclosed above is intended to describe some embodiments of the present disclosure and is not intended to describe the only embodiment of the present disclosure. The following detailed description includes more details in order to provide a comprehensive understanding of the present disclosure. However, it will be understood by those skilled in the art that the present disclosure can be practiced without such more details.

[0072] In some cases, in order to avoid obscuring the concepts of the present disclosure, known structures and devices are omitted or may be shown in the form of block diagrams based on the core functions of each structure and device.

[0073] Although most of the terms used in this disclosure are selected from common terms widely used in the art, the applicant has arbitrarily selected some terms and explained their meanings in detail in the following description as needed. Therefore, the disclosure should be understood with the intended meanings of the terms rather than their simple names or meanings.

[0074] The specific terms used in the following description are provided to help understand the present disclosure, and the use of these specific terms can be changed in various forms without departing from the technical spirit of the present disclosure. For example, signals, data, samples, pictures, frames, and blocks can be appropriately replaced and interpreted in each encoding process.

[0075] In this specification, a "processing unit" refers to a unit in which encoding / decoding processing such as prediction, transform and / or quantization is performed. Hereinafter, for convenience of description, a processing unit may be referred to as a "processing block" or a "block."

[0076] Furthermore, a processing unit may be interpreted as including a unit for luma components and a unit for chroma components. For example, a processing unit may correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0077] In addition, the processing unit can be interpreted as a unit for the luma component or a unit for the chroma component. For example, the processing unit can correspond to a coding tree block (CTB), coding block (CB), prediction unit PU, or transform block (TB) for the luma component. In addition, the processing unit can correspond to a CTB, CB, PU, ​​or TB for the chroma component. In addition, the processing unit is not limited to this and can be interpreted as including the meaning of a unit for the luma component and a unit for the chroma component.

[0078] Additionally, the processing unit is not necessarily limited to a square block and may be configured in a polygonal shape having three or more vertices.

[0079] In addition, in this specification, a pixel is referred to as a sample. In addition, using a sample may mean using a pixel value or the like.

[0080] Figure 1 1 is a schematic block diagram of an encoding apparatus for encoding a video / image signal as an embodiment to which the present disclosure is applied.

[0081] Reference Figure 1, the encoding device 100 can be configured to include an image divider 110, a subtractor 115, a transformer 120, a quantizer 130, a dequantizer 140, an inverse transformer 150, an adder 155, a filter 160, a memory 170, an inter-frame predictor 180, an intra-frame predictor 185, and an entropy encoder 190. The inter-frame predictor 180 and the intra-frame predictor 185 can be collectively referred to as a predictor. In other words, the predictor can include the inter-frame predictor 180 and the intra-frame predictor 185. The transformer 120, the quantizer 130, the dequantizer 140, and the inverse transformer 150 can be included in a residual processor. The residual processor can also include the subtractor 115. In one embodiment, the image divider 110, the subtractor 115, the transformer 120, the quantizer 130, the dequantizer 140, the inverse transformer 150, the adder 155, the filter 160, the inter-frame predictor 180, the intra-frame predictor 185, and the entropy encoder 190 may be configured as a hardware component (e.g., an encoder or a processor). In addition, the memory 170 may include a decoded picture buffer (DPB) and may be implemented by a digital storage medium.

[0082] The image divider 110 can divide the input image (or picture or frame) input to the encoding device 100 into one or more processing units. For example, the processing unit can be referred to as a coding unit (CU). In this case, the coding unit can be recursively split from the coding tree unit (CTU) or the maximum coding unit (LCU) based on a quadtree binary tree (QTBT) structure. For example, based on a quadtree structure and / or a binary tree structure, one coding unit can be split into multiple coding units with a deeper depth. In this case, for example, the quadtree structure can be applied first, and then the binary tree structure can be applied. Alternatively, the binary tree structure can be applied first. The encoding process according to the present disclosure can be performed based on the final coding unit that is no longer split. In this case, the maximum coding unit can be directly used as the final coding unit based on coding efficiency according to image characteristics, or the coding unit can be recursively split into deeper coding units if necessary. Therefore, the coding unit with the optimal size can be used as the final coding unit. In this case, the encoding process may include processes such as prediction, transformation, or reconstruction, which will be described later. For another example, the processing unit may also include a prediction unit (PU) or a transform unit (TU). In this case, each of the prediction unit and the transform unit may be divided or partitioned from each final coding unit. The prediction unit may be a unit for sample prediction, and the transform unit may be a unit from which a transform coefficient is derived and / or a unit in which a residual signal is derived from the transform coefficient.

[0083] Depending on the situation, a unit may be used interchangeably with a block or region. In general, an M×N block may indicate a set of samples or a set of transform coefficients configured with M columns and N rows. In general, a sample may indicate a pixel or a pixel value, and may indicate only a pixel / pixel value of a luma component or only a pixel / pixel value of a chroma component. In a sample, a picture (or image) may be used as a term corresponding to a pixel or a pel.

[0084] The encoding device 100 can generate a residual signal (residual block or residual sample array) by subtracting the prediction signal (prediction block or prediction sample array) output by the inter predictor 180 or the intra predictor 185 from the input image signal (original block or original sample array). The generated residual signal is sent to the transformer 120. In this case, as shown in the figure, the unit within the encoding device 100 that subtracts the prediction signal (prediction block or prediction sample array) from the input image signal (original block or original sample array) can be referred to as a subtractor 115. The predictor can perform prediction on a processing target block (hereinafter referred to as a current block) and can generate a prediction block including prediction samples for the current block. The predictor can determine whether to apply intra prediction or inter prediction in the current block or CU unit. The predictor can generate various information related to prediction, such as the prediction mode information described later in the description of each prediction mode, and can send this information to the entropy encoder 190. The information related to the prediction can be encoded in the entropy encoder 190 and output in the form of a bitstream.

[0085] The intra-frame predictor 185 can predict the current block with reference to samples within the current picture. The position of the referenced sample may be adjacent to the current block or may be spaced apart from the current block according to the prediction mode. In intra-frame prediction, the prediction mode may include multiple non-angle modes and multiple angle modes. The non-angle mode may include, for example, a DC mode and a planar mode. For example, depending on the degree of refinement of the prediction direction, the angle mode may include 33 angle prediction modes or 65 angle prediction modes. In this case, for example, depending on the configuration, more or less than 33 angle prediction modes or 65 angle prediction modes may be used. The intra-frame predictor 185 may use the prediction mode applied to the neighboring blocks to determine the prediction mode applied to the current block.

[0086] The inter-frame predictor 180 can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector in a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted into blocks, subblocks, or sample units based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can also include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction). In the case of inter-frame prediction, neighboring blocks can include spatially neighboring blocks within the current picture and temporally neighboring blocks within a reference picture. The reference picture including the reference block and the reference picture including the temporally neighboring block can be the same or different. Temporally neighboring blocks can be referred to as collocated reference blocks or collocated CUs (colCUs). A reference picture including temporally neighboring blocks can be referred to as collocated pictures (colPics). For example, the inter-frame predictor 180 can construct a motion information candidate list based on the neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index for the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 180 can use the motion information of the adjacent block as the motion information of the current block. In skip mode, unlike merge mode, a residual signal may not be sent. In motion vector prediction (MVP) mode, the motion vector of the adjacent block can be used as the motion vector predictor. The motion vector of the current block can be indicated by signaling the motion vector difference.

[0087] The prediction signal generated by the inter-frame predictor 180 or the intra-frame predictor 185 may be used to generate a reconstructed signal or a residual signal.

[0088] The transformer 120 may generate transform coefficients by applying a transform scheme to the residual signal. For example, the transform scheme may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loève transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). In this case, GBT refers to a transform obtained from a graph if the relationship information between pixels is represented as the graph. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process may be applied to square pixel blocks of the same size, or may be applied to blocks of variable size other than square blocks.

[0089] The quantizer 130 can quantize the transform coefficients and send them to the entropy encoder 190. The entropy encoder 190 can encode the quantized signal (information about the quantized transform coefficients) and output it in the form of a bitstream. The information about the quantized transform coefficients can be called residual information. The quantizer 130 can rearrange the quantized transform coefficients in block form in the form of a one-dimensional vector based on the coefficient scan sequence, and can generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of a one-dimensional vector. The entropy encoder 190 can perform various encoding methods such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). In addition to the quantized transform coefficients, the entropy encoder 190 can also encode information necessary for video / image reconstruction (e.g., syntax element values) together or separately. The encoded information (e.g., the encoded video / image information) can be transmitted or stored in the form of a bitstream in units of network abstraction layer (NAL) units. The bitstream can be transmitted over a network or can be stored in a digital storage medium. In this case, the network can include a broadcast network and / or a communication network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitter (not shown) that transmits the signal output by the entropy encoder 190 and / or a memory (not shown) for storing the signal may be configured as an internal / external element of the encoding device 100, or the transmitter may be an element of the entropy encoder 190.

[0090] The quantized transform coefficients output by the quantizer 130 can be used to generate a prediction signal. For example, a residual signal can be reconstructed by applying dequantization and inverse transformation to the quantized transform coefficients via a dequantizer 140 and an inverse transformer 150 within a loop. The adder 155 can add the reconstructed residual signal to the prediction signal output by the inter-frame predictor 180 or the intra-frame predictor 185, thereby generating a reconstructed signal (reconstructed image, reconstructed block, or reconstructed sample array). If there is no residual for the processing target block, as in the case where skip mode has been applied, the prediction block can be used as a reconstructed block. The adder 155 can be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, and can be used for inter-frame prediction of the next picture through filtering, which will be described later.

[0091] The filter 160 can improve the subjective / objective picture quality by applying filtering to the reconstructed signal. For example, the filter 160 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture. The modified reconstructed picture can be stored in the memory 170, more specifically, in the DPB of the memory 170. Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering. The filter 160 can generate various information for filtering (as will be described later in the description of each filtering method) and can send them to the entropy encoder 190. The filter information can be encoded by the entropy encoder 190 and output in the form of a bitstream.

[0092] The modified reconstructed picture transmitted to the memory 170 may be used as a reference picture in the inter predictor 180. If inter prediction is applied, the encoding apparatus may avoid prediction mismatch in the encoding apparatus 100 and the decoding apparatus, and may improve encoding efficiency.

[0093] The DPB of the memory 170 may store the modified reconstructed picture for use as a reference picture in the inter-frame predictor 180. The memory 170 may store the motion information of the block from which the motion information in the current picture is derived (or encoded) and / or the motion information of the block in the reconstructed picture. The stored motion information may be forwarded to the inter-frame predictor 180 for use as motion information of spatially neighboring blocks or motion information of temporally neighboring blocks. The memory 170 may store the reconstructed samples of the reconstructed block in the current picture and forward them to the intra-frame predictor 185.

[0094] Figure 2 1 is a schematic block diagram of a decoding device for decoding a video / image signal, to which the present disclosure is applied.

[0095] Reference Figure 2 , the decoding device 200 can be configured to include an entropy decoder 210, a dequantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-frame predictor 260, and an intra-frame predictor 265. The inter-frame predictor 260 and the intra-frame predictor 265 can be collectively referred to as a predictor. That is, the predictor can include the inter-frame predictor 180 and the intra-frame predictor 185. The dequantizer 220 and the inverse transformer 230 can be collectively referred to as a residual processor. That is, the residual processor can include the dequantizer 220 and the inverse transformer 230. The entropy decoder 210, the dequantizer 220, the inverse transformer 230, the adder 235, the filter 240, the inter-frame predictor 260, and the intra-frame predictor 265 can be configured as a hardware component (e.g., a decoder or a processor) according to one embodiment. In addition, the memory 250 may include a decoded picture buffer (DPB) and may be implemented by a digital storage medium.

[0096] When a bit stream including video / image information is input, the decoding device 200 can Figure 1 The image is reconstructed by processing video / image information in the encoding device. For example, the decoding device 200 can perform decoding using the processing unit applied in the encoding device. Therefore, for example, the processing unit used for decoding can be a coding unit. The coding unit can be divided from the coding tree unit or the maximum coding unit according to the quadtree structure and / or binary tree structure. In addition, the reconstructed image signal decoded and output by the decoding device 200 can be played back by a playback device.

[0097] The decoding device 200 may receive the Figure 1 The signal output by the encoding device. The received signal can be decoded by the entropy decoder 210. For example, the entropy decoder 210 can derive information (e.g., video / image information) for image reconstruction (or picture reconstruction) by parsing the bitstream. For example, the entropy decoder 210 can decode the information within the bitstream based on a coding method such as exponential Golomb, CAVLC, or CABAC, and can output the value of the syntax element for image reconstruction or the quantized value of the transform coefficient of the residual. More specifically, in the CABAC entropy decoding method, a binary number (bin) corresponding to each syntax element can be received from the bitstream, and the context model can be determined using the decoding target syntax element information and the decoding information of the adjacent and decoding target blocks or the information of the symbol / binary number decoded in the previous step. The probability of the binary number appearing can be predicted based on the determined context model, and the symbol corresponding to the value of each syntax element can be generated by performing arithmetic decoding on the binary number. In this case, in the CABAC entropy decoding method, after determining the context model, the context model can be updated using the information of the symbol / binary number decoded by the context model for the next symbol / binary number. Information on prediction among the information decoded in the entropy decoder 2110 may be provided to the predictor (inter-frame predictor 260 and intra-frame predictor 265). Parameter information (i.e., quantized transform coefficients) related to the residual value on which entropy decoding has been performed in the entropy decoder 210 may be input to the dequantizer 220. In addition, information on filtering among the information decoded in the entropy decoder 210 may be provided to the filter 240. In addition, a receiver (not shown) that receives a signal output by the encoding device may be further configured as an internal / external element of the decoding device 200, or the receiver may be an element of the entropy decoder 210.

[0098] The dequantizer 220 may dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 220 may rearrange the quantized transform coefficients in a two-dimensional block form. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding device. The dequantizer 220 may dequantize the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) and may obtain the transform coefficients.

[0099] The inverse transformer 230 may output a residual signal (a residual block or a residual sample array) by applying inverse transform to the transform coefficients.

[0100] The predictor may perform prediction on the current block and may generate a prediction block including prediction samples for the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on the information about prediction output by the entropy decoder 210, and may determine a specific intra / inter prediction mode.

[0101] The intra-frame predictor 265 can predict the current block with reference to samples within the current picture. Depending on the prediction mode, the location of the reference sample can be adjacent to the current block or can be spaced apart from the current block. In intra-frame prediction, the prediction mode can include multiple non-angle modes and multiple angle modes. The intra-frame predictor 265 can use the prediction mode applied to the neighboring blocks to determine the prediction mode applied to the current block.

[0102] The inter-frame predictor 260 can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted as a block, sub-block, or sample unit based on the correlation of motion information between neighboring blocks and the current block. Motion information can include a motion vector and a reference picture index. Motion information can also include inter-frame prediction direction information (L0 prediction, L1 prediction, and Bi prediction). In the case of inter-frame prediction, neighboring blocks can include spatial neighboring blocks within the current picture and temporal neighboring blocks within the reference picture. For example, the inter-frame predictor 260 can configure a motion information candidate list based on neighboring blocks and can derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes. Information about the prediction can include information indicating the mode of inter-frame prediction for the current block.

[0103] The adder 235 can generate a reconstructed signal (reconstructed picture, reconstructed block, or reconstructed sample array) by adding the obtained residual signal to the prediction signal (prediction block or prediction sample array) output by the inter-frame predictor 260 or the intra-frame predictor 265. If there is no residual for the processing target block as in the case where the skip mode is applied, the prediction block can be used as the reconstructed block.

[0104] The adder 235 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra-frame prediction of the next processing target block in the current picture, and may be used for inter-frame prediction of the next picture through filtering, which will be described later.

[0105] The filter 240 can improve the subjective / objective picture quality by applying filtering to the reconstructed signal. For example, the filter 240 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and can send the modified reconstructed picture to the memory 250, more specifically, to the DPB of the memory 250. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), adaptive loop filtering (ALF), and bilateral filtering.

[0106] The (modified) reconstructed picture stored in the DPB of the memory 250 can be used as a reference picture in the inter-frame predictor 260. The memory 250 can store the motion information of the block from which the motion information in the current picture is derived (or decoded) and / or the motion information of the block in the reconstructed picture. The stored motion information can be forwarded to the inter-frame predictor 260 to be used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory 170 can store the reconstructed samples of the reconstructed block in the current picture and forward them to the intra-frame predictor 265.

[0107] In the present disclosure, the embodiments described in the filter 160, the inter-frame predictor 180, and the intra-frame predictor 185 of the encoding device 100 may be applied to the filter 240, the inter-frame predictor 260, and the intra-frame predictor 265 of the decoding device 200, respectively, identically, or in a corresponding manner.

[0108] Block Partitioning

[0109] The video / image encoding method according to the present disclosure can be performed based on various detailed techniques, and each of the various detailed techniques is described below. It is obvious to those skilled in the art that the techniques described herein can be associated with related processes such as prediction, residual processing ((inverse) transform and (de)quantization, etc.), syntax element encoding, filtering, partitioning / segmentation, etc. in the video / image encoding / decoding process described above and / or below.

[0110] The block partitioning process according to the present disclosure can be performed in the image divider 110 of the encoding device described above, and the partition-related information can be processed (encoded) in the entropy encoder 190 and forwarded to the decoding device in a bitstream format. The entropy decoder 210 of the decoding device can obtain the block partition structure of the current picture based on the partition-related information obtained from the bitstream, and based on this, a series of processes for image decoding (e.g., prediction, residual processing, block reconstruction, and loop filtering, etc.) can be performed.

[0111] Partition the image into CTUs

[0112] A picture can be divided into a sequence of coding tree units (CTUs). A CTU can correspond to a coding tree block (CTB). Alternatively, a CTU can include a coding tree block of luma samples and two corresponding coding tree blocks of chroma samples. In other words, for a picture including three types of sample arrays, a CTU can include an N×N block of luma samples and two corresponding samples of chroma samples.

[0113] The maximum supported size of a CTU for coding and prediction may be different from the maximum supported size of a CTU for transform. For example, the maximum supported size of a luma block in a CTU may be 128×128.

[0114] Partition CTU using tree structure

[0115] A CTU can be divided into CUs based on a quadtree (QT) structure. The quadtree structure can be called a quaternary structure. This is to reflect various local features. In addition, in the present disclosure, a CTU can be divided based on a multi-type tree structure partition including a binary tree (BT) and a ternary tree (TT) as well as a quadtree. Hereinafter, a QTBT structure may include a quadtree and a binary tree structure, and a QTBTTT may include a partition structure based on a binary tree and a ternary tree. Alternatively, a QTBT structure may also include a partition structure based on a quadtree, a binary tree, and a ternary tree. In the coding tree structure, a CU may have a square shape or a rectangular shape. First, a CTU may be divided into a quadtree structure. And then, the leaf nodes of the quadtree structure may be further divided by a multi-type tree structure.

[0116] Figure 3 is a diagram illustrating an example of a multi-type tree structure to which an embodiment of the present disclosure can be applied.

[0117] In one embodiment of the present disclosure, the multi-type tree structure may include: Figure 3There are four types of splitting as shown. These four types of splitting may include vertical binary splitting (SPLIT_BT_VER), horizontal binary splitting (SPLIT_BT_HOR), vertical ternary splitting (SPLIT_TT_VER), and horizontal ternary splitting (SPLIT_TT_HOR). The leaf nodes of the multi-type tree structure may be referred to as CUs. Such CUs may be used for prediction and transform processes. In the present disclosure, generally, CUs, PUs, and TUs may have the same block size. However, in the case where the maximum supported transform length is less than the width or height of the color component, CUs and TUs may have different block sizes.

[0118] Figure 4 : is a diagram illustrating a signaling mechanism of partition splitting information of a quadtree having a nested multi-type tree structure as an embodiment to which the present disclosure can be applied.

[0119] Here, the CTU can be regarded as the root of a quadtree and is initially partitioned into a quadtree structure. Each quadtree leaf node can then be further partitioned into a multi-type tree structure. In this multi-type tree structure, a first flag (e.g., mtt_split_cu_flag) is signaled to indicate whether the corresponding node is further partitioned. In the case where the corresponding node is further partitioned, a second flag (e.g., mtt_split_cu_vertical_flag) can be signaled to indicate the partition direction. Thereafter, a third flag (e.g., mtt_split_cu_binary_flag) can be signaled to indicate whether the partition type is binary or ternary. For example, based on mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, a multi-type tree partition mode (MttSplitMode) can be derived as shown in Table 1 below.

[0120] [Table 1]

[0121] MttSplitMode mtt_split_cu_vertical_flag mtt_split_cu_binary_flag SPLIT_TT_HOR 0 0 SPLIT_BT_HOR 0 1 SPLIT_TT_VER 1 0 SPLIT_BT_VER 1 1

[0122] Figure 5 2 is a diagram illustrating a method of partitioning a CTU into multiple CUs based on a quadtree and a nested multi-type tree structure as an embodiment to which the present disclosure may be applied.

[0123] Here, the bold block edges represent quadtree partitions, and the remaining edges represent multi-type tree partitions. Quadtree partitions with nested multi-type trees can provide a content-adaptive coding tree structure. A CU may correspond to a coding block (CB). Alternatively, a CU may include a coding block of luminance samples and two coding blocks of corresponding chrominance samples. The size of a CU can be as large as a CTU or as small as 4×4 (in units of luminance samples). For example, in the case of a 4:2:0 color format (or chrominance format), the maximum chrominance CB size may be 64×64 and the minimum chrominance CB size may be 2×2.

[0124] In the present disclosure, for example, the maximum supported luma TB size may be 64 × 64, and the maximum supported chroma TB size may be 32 × 32. In the case where the width or height of the CB partitioned according to the tree structure is greater than the maximum transform width or height, the CB may be further partitioned until the TB size restrictions in the horizontal and vertical directions are automatically (or implicitly) met.

[0125] Furthermore, for a quadtree coding tree scheme with nested multiple types, the following parameters may be defined or identified as SPS syntax elements.

[0126] -CTU size: the root node size of the quadtree

[0127] -MinQTSize: Minimum allowed quadtree leaf node size

[0128] -MaxBtSize: Maximum allowed binary tree root node size

[0129] -MaxTtSize: Maximum allowed ternary tree root node size

[0130] -MaxMttDepth: The maximum allowed hierarchy depth of a multi-type tree split from a quad leaf.

[0131] -MinBtSize: minimum allowed binary tree leaf node size

[0132] -MinTtSize: Minimum allowed ternary tree leaf node size

[0133] As an example of a quadtree coding tree scheme with nested multi-type trees, the CTU size can be set to a 64×64 block of 128×128 luma samples and two corresponding chroma samples (in 4:2:0 chroma samples). In this case, MinOTSize can be set to 16×16, MaxBtSize can be set to 128×128, MaxTtSzie can be set to 64×64, MinBtSize and MinTtSize (for both width and height) can be set to 4×4, and MaxMttDepth can be set to 4. Quadtree partitioning can be applied to the CTU and a quadtree leaf node can be generated. The quadtree leaf node can be referred to as a leaf QT node. The size of the quadtree leaf node can range from a size of 16×16 (i.e., MinOTSize) to a size of 128×128 (i.e., CTU size). In the case where the leaf QT node is 128×128, the leaf QT node may not be partitioned into a binary tree / ternary tree. This is because, even in the case where the leaf QT node is partitioned, the leaf QT node exceeds MaxBtsize and MaxTtszie (i.e., 64×64). In other cases, the leaf QT node can be additionally partitioned into a multi-type tree. Therefore, the leaf QT node can be the root node of the multi-type tree, and the leaf QT node can have a multi-type tree depth (mttDepth) value of 0. In the case where the multi-type tree depth reaches MaxMttdepth (for example, 4), additional partitioning can no longer be considered. In the case where the width of the multi-type tree node is equal to MinBtSize and less than or equal to 2×MinTtSize, additional horizontal partitioning can no longer be considered. In the case where the height of the multi-type tree node is equal to MinBtSize and less than or equal to 2×MinTtSize, additional vertical partitioning can no longer be considered.

[0134] Figure 6 is a diagram illustrating a method for limiting ternary tree partitioning as an embodiment to which the present disclosure can be applied.

[0135] Reference Figure 6 In order to support 64×64 luma blocks and 32×32 chroma pipeline design in hardware decoders, TT splitting can be restricted in certain cases. For example, when the width or height of the luma coding block is greater than a predetermined specific value (e.g., 32, 64), Figure 6 As shown, the TT split can be restricted.

[0136] In the present disclosure, the coding tree scheme can support luma and chroma blocks to have their own block tree structures. For P slices and B slices, the luma and chroma CTBs in a single CTU can be restricted to have the same coding tree structure. However, for I slices, luma and chroma blocks can have their own separate block tree structures. When a separate block tree mode is applied, the luma CTB can be partitioned into CUs based on a specific coding tree structure, and the chroma CTB can be partitioned into chroma CUs based on different coding tree structures. This may mean that the CU in the I slice may include a coding block of a chroma component or a coding block of two chroma components, and the CU in the P slice or B slice may include blocks of three color components.

[0137] In the above-mentioned "Partitioning CTU using a tree structure", a quadtree coding tree scheme with nested multi-type trees is described, but the structure for partitioning the CU is not limited thereto. For example, the BT structure and the TT structure can be interpreted as concepts included in the multi-partition tree (MPT) structure, and can be interpreted as the CU being partitioned by the QT structure and the MPT structure. In the example of partitioning the CU by the QT structure and the MPT structure, a syntax element (e.g., MPT_split_type) including information on the number of blocks into which the leaf node of the QT structure is partitioned and a syntax element (e.g., MPT_split_mode) including information on the direction in which the leaf node of the QT structure is partitioned among the vertical and horizontal directions can be signaled, and the partitioning structure can be determined.

[0138] In another example, the CU may be partitioned in a method different from the QT structure, the BT structure, or the TT structure. That is, instead of partitioning a CU of a lower depth into 1 / 4 the size of a CU of a higher depth according to the QT structure, partitioning a CU of a lower depth into 1 / 2 the size of a CU of a higher depth according to the BT structure, or partitioning a CU of a lower depth into 1 / 4 or 1 / 2 the size of a CU of a higher depth according to the TT structure, in some cases, the CU of the lower depth may be partitioned into 1 / 5, 1 / 3, 3 / 8, 3 / 5, 2 / 3, or 5 / 8 the size of the CU of the higher depth, but the method of partitioning the CU is not limited thereto.

[0139] In the case where a portion of a tree node block exceeds the bottom or right picture boundary, the corresponding tree node block may be restricted so that all samples of all coded CUs are within the picture boundary. In this case, for example, the following partitioning rules may be applied.

[0140] - if part of the tree node block exceeds both the bottom and right picture boundaries,

[0141] - If the chunk is a QT node and the chunk size is larger than the minimum QT size, the chunk is forced to be split using the QT split mode.

[0142] - Otherwise, the block will be forced to use SPLIT_BT_HOR mode for splitting

[0143] - Otherwise, if part of the tree node block exceeds the bottom picture boundary,

[0144] - If the chunk is a QT node, and the chunk's size is larger than the minimum QT size, and the chunk's size is larger than the maximum BT size, the chunk is forced to be split using the QT split mode.

[0145] Otherwise, if the chunk is a QT node and the chunk's size is greater than the minimum QT size, and the chunk's size is less than or equal to the maximum BT size, the chunk is forced to be split using the QT split mode or the SPLIT_BT_HOR mode.

[0146] Otherwise (the block is a BTT node, or the block size is less than or equal to the minimum QT size), the block is forced to be split using the SPLIT_BT_HOR mode.

[0147] - Otherwise, if part of the tree node block exceeds the right picture boundary,

[0148] - If the chunk is a QT node, and the chunk's size is larger than the minimum QT size, and the chunk's size is larger than the maximum BT size, the chunk is forced to be split using the QT split mode.

[0149] Otherwise, if the chunk is a QT node and the chunk's size is greater than the minimum QT size, and the chunk's size is less than or equal to the maximum BT size, the chunk is forced to be split using the QT split mode or SPLIT_BT_VER mode.

[0150] - Otherwise (the block is a BTT node, or the block's size is less than or equal to the minimum QT size), the block is forced to be split using the SPLIT_BT_VER mode.

[0151] The quadtree coding block structure with a multi-type tree provides a highly flexible block partitioning structure. Due to the various types of partitioning supported by the multi-type tree, different partitioning modes may, in some cases, result in the same coding block structure. This approach limits the generation of redundant partitioning modes to reduce the amount of partition information data. This is described with reference to the following figures.

[0152] Figure 7 : is a diagram showing a redundant partitioning pattern that may occur in binary tree partitioning and ternary tree partitioning as an embodiment to which the present disclosure can be applied.

[0153] like Figure 7As shown, two levels of consecutive binary partitions in one direction have the same coding block structure as the binary partition for the middle partition after the ternary partition. In this case, the binary tree partition for the middle partition of the ternary partition can be restricted (along a given direction). This restriction can be applied to CUs of all pictures. When a specific partition is restricted, the signaling of the syntax elements can be modified by reflecting this restriction, and the number of bits signaled for the partition can be reduced by the modified signaling. For example, with Figure 7 Similar to the example shown, when binary tree partitioning for the middle partition of a CU is restricted, the syntax element mtt_split_cu_binary_flag indicating whether the partition is binary or ternary may not be signaled and the decoder may infer the value to be 0.

[0154] predict

[0155] To reconstruct the current processing unit in which decoding is performed, a decoded portion of a current picture or other pictures including the current processing unit may be used.

[0156] A picture that uses only the current picture for reconstruction (i.e., performs intra prediction) may be referred to as an intra picture or I picture (slice), a picture (slice) that uses up to one motion vector and reference index to predict each unit may be referred to as a predicted picture or P picture (slice), and a picture (slice) that uses up to two motion vectors and reference indices may be referred to as a bilaterally predicted picture (bi-predictive picture) or B picture (slice).

[0157] Intra-frame prediction refers to a prediction method that derives the pixel values ​​of the current processing block from data elements (e.g., sample values) of the same decoded picture (or slice). In other words, intra-frame prediction refers to a method that predicts the pixel values ​​of the current processing block by referencing a reconstructed area in the current picture.

[0158] Hereinafter, inter prediction will be described in more detail.

[0159] Inter-frame prediction

[0160] Inter-frame prediction refers to a method for deriving prediction values ​​for the current block based on data elements (e.g., sample values ​​or motion vectors) from pictures other than the current one. In other words, inter-frame prediction refers to a method for predicting pixel values ​​for the current block by referencing reconstructed regions in reconstructed pictures other than the current one.

[0161] Inter prediction (inter-picture prediction), which is a technique for eliminating redundancy existing between pictures, is mainly performed through motion estimation and motion compensation.

[0162] In this disclosure, the above Figure 1 and Figure 2 The detailed description of the inter-frame prediction method described in the following and the decoder can be represented as will be described below Figure 10 Video / image decoding method based on inter-frame prediction and Figure 11 In addition, the encoder can be represented as will be described below Figure 8 Video / image coding method based on inter-frame prediction and Figure 9 In addition, it can be stored in the form of a bit stream Figure 8 and Figure 9 The encoded data.

[0163] The prediction unit of the encoding device / decoding device can derive prediction samples by performing inter-frame prediction in units of blocks. Inter-frame prediction can refer to prediction derived by a method that relies on data elements (e.g., sample values ​​or motion information) of pictures other than the current picture. When inter-frame prediction is applied to the current block, the prediction block (prediction sample array) for the current block can be derived based on the reference block (reference sample array) specified by the motion vector on the reference picture indicated by the reference picture index.

[0164] In this case, to reduce the amount of motion information sent in inter-frame prediction mode, the motion information of the current block can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between adjacent blocks and the current block. Motion information can include motion vectors and reference picture indexes. Motion information can also include inter-frame prediction type information (such as L0 prediction, L1 prediction, and Bi prediction).

[0165] When inter-frame prediction is applied, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same as or different from each other. The temporal neighboring block may be referred to as a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block may be referred to as a collocated picture (colPic). For example, a motion information candidate list may be configured based on the neighboring blocks of the current block, and a flag or index information indicating which candidate is selected (used) may be signaled to facilitate derivation of the motion vector and / or reference picture index of the current block.

[0166] Inter-frame prediction can be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the motion information of the current block can be the same as the motion information of the selected neighboring block. In the case of skip mode, a residual signal may not be sent as in merge mode. In the case of motion vector prediction (MVP) mode, the motion vector of the selected neighboring block can be used as a motion vector predictor, and the motion vector difference can be signaled. In this case, the motion vector of the current block can be derived by using the sum of the motion vector predictor and the motion vector difference.

[0167] Figure 8 and Figure 9 2 is a diagram illustrating a video / image encoding method based on inter-frame prediction according to an embodiment of the present disclosure and an inter-frame prediction unit in an encoding device according to an embodiment of the present disclosure.

[0168] refer to Figure 8 and Figure 9 , S801 may be performed by the inter-frame prediction unit (inter-frame predictor) 180 of the encoding device and S802 may be performed by the residual processing unit of the encoding device. Specifically, S802 may be performed by the subtractor 115 of the encoding device. In S803, prediction information may be derived by the inter-frame prediction unit (inter-frame predictor) 180 and may be encoded by the entropy encoder 190. In S803, residual information may be derived by the residual processing unit and encoded by the entropy encoder 190. The residual information is information about the residual samples. The residual information may include information about the quantized transform coefficients of the residual samples.

[0169] As described above, the residual samples may be derived as transform coefficients by the transformer 120 of the encoding apparatus, and the transform coefficients may be derived as quantized transform coefficients by the quantizer 130. Information about the quantized transform coefficients may be encoded by the entropy encoder 190 through a residual encoding process.

[0170] The encoding device performs inter-frame prediction for the current block (S801). The encoding device can derive the inter-frame prediction mode and motion information of the current block and generate prediction samples of the current block. Here, the inter-frame prediction mode determination process, the motion information derivation process, and the prediction sample generation process can be performed simultaneously, and any one process can be performed earlier than the other processes. For example, the inter-frame predictor 180 of the encoding device may include a prediction mode determination unit 181, a motion information derivation unit 182, and a prediction sample derivation unit 183, and the prediction mode determination unit 181 may determine the prediction mode for the current block, the motion information derivation unit 182 may derive the motion information of the current block, and the prediction sample derivation unit 183 may derive the prediction sample of the current block.

[0171] For example, the inter-frame prediction unit (inter-frame predictor) 180 of the encoding device can search for a block similar to the current block in a predetermined area (search area) of the reference picture through motion estimation, and derive a reference block whose difference with the current block is the smallest or equal to or less than a predetermined standard. Based on this, a reference picture index indicating the reference picture where the reference block is located can be derived, and a motion vector can be derived based on the position difference between the reference block and the current block. The encoding device can determine a mode to be applied to the current block from various prediction modes. The encoding device can compare the RD costs (RD cost) for various prediction modes and determine the optimal prediction mode for the current block.

[0172] For example, when skip mode or merge mode is applied to the current block, the encoding device may configure a merge candidate list (merging candidate list) to be described below, and derive a reference block whose difference with the current block is the smallest or equal to or less than a predetermined standard among the reference blocks indicated by the merge candidates included in the merge candidate list. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to the decoding device. The motion information of the current block may be derived by using the motion information of the selected merge candidate.

[0173] As another example, when the (A)MVP mode is applied to the current block, the encoding device may configure the (A)MVP candidate list to be described below, and use the motion vector of the selected MVP candidate among the motion vector predictor (MVP) candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, the motion vector indicating the reference block derived by motion estimation may be used as the motion vector of the current block. And, the MVP candidate having the motion vector with the smallest difference from the motion vector of the current block among the MVP candidates may become the selected MVP candidate. A motion vector difference (MVD) as the difference obtained by subtracting the MVP from the motion vector of the current block may be derived. In this case, information about the MVD may be signaled to the decoding device. In addition, when the (A)MVP mode is applied, the value of the reference picture index may be configured as reference picture index information and signaled separately to the decoding device.

[0174] The encoding apparatus may derive residual samples based on the prediction samples (S802). The encoding apparatus may derive residual samples by comparing original samples of the current block with the prediction samples.

[0175] The encoding device encodes the image information including the prediction information and the residual information (S803). The encoding device can output the encoded image information in the form of a bitstream. The prediction information may include information about the prediction mode information (e.g., a skip flag, a merge flag, or a mode index, etc.) and information about the motion information as information related to the prediction process. The information about the motion information may include candidate selection information (e.g., a merge index, an mvp flag, or an mvp index) as information for deriving a motion vector. In addition, the information about the motion information may include information about the MVD and / or reference picture index information.

[0176] In addition, the information about motion information may include information indicating whether L0 prediction, L1 prediction, or bilateral prediction is applied. The residual information is information about residual samples. The residual information may include information about quantized transform coefficients of the residual samples.

[0177] The output bitstream may be stored in a (digital) storage medium and transmitted to a decoding device or transmitted to a decoding device via a network.

[0178] In addition, as described above, the encoding device can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is to derive the same prediction result as the prediction result performed by the decoding device, and thereby improve the coding efficiency. Therefore, the encoding device can store the reconstructed picture (or reconstructed sample or reconstructed block) in a memory and use the reconstructed picture as a reference picture. As described above, the loop filtering process can be further applied to the reconstructed picture.

[0179] Figure 10 and Figure 11 2 is a diagram illustrating a video / image decoding method based on inter-frame prediction according to an embodiment of the present disclosure and an inter-frame prediction unit in a decoding device according to an embodiment of the present disclosure.

[0180] Reference Figure 10 and Figure 11 , the decoding device may perform an operation corresponding to the operation performed by the encoding device. The decoding device may perform prediction for the current block based on the received prediction information and derive a prediction sample.

[0181] S1001 to S1003 may be performed by the inter-frame prediction unit (inter-frame predictor) 260 of the decoding device, and the residual information of S1004 may be obtained from the bitstream by the entropy decoder 210 of the decoding device. The residual processing unit of the decoding device may derive residual samples for the current block based on the residual information. Specifically, the dequantizer 220 of the residual processing unit may derive transform coefficients by performing dequantization based on the quantized transform coefficients derived based on the residual information, and the inverse transformer 230 of the residual processing unit may derive residual samples for the current block by performing inverse transform on the transform coefficients. S1005 may be performed by the adder 235 or the reconstruction unit of the decoding device.

[0182] Specifically, the decoding device may determine a prediction mode for the current block based on the received prediction information (S1001). The decoding device may determine which inter-frame prediction mode to apply to the current block based on prediction mode information in the prediction information.

[0183] For example, whether to apply merge mode or (A)MVP mode to the current block may be determined based on the merge flag. Alternatively, one of various inter-frame prediction mode candidates may be selected based on the mode index. Inter-frame prediction mode candidates may include skip mode, merge mode, and / or (A)MVP mode, or may include various inter-frame prediction modes described below.

[0184] The decoding device derives motion information of the current block based on the determined inter-frame prediction mode (S1002). For example, when the skip mode or merge mode is applied to the current block, the decoding device may configure a merge candidate list to be described below and select a merge candidate from among the merge candidates included in the merge candidate list. The selection may be performed based on selection information (merge index). The motion information of the current block may be derived by using the motion information of the selected merge candidate. The motion information of the selected merge candidate may be used as the motion information of the current block.

[0185] As another example, when the (A)MVP mode is applied to the current block, the decoding device may configure the (A)MVP candidate list described below and use the motion vector of the selected motion vector predictor (MVP) candidate among the motion vector predictor (MVP) candidates included in the (A)MVP candidate list as the MVP of the current block. The selection may be performed based on selection information (MVP flag or MVP index). In this case, the MVD of the current block may be derived based on information about the MVD, and the motion vector of the current block may be derived based on the MVP and MVD of the current block. In addition, the reference picture index of the current block may be derived based on the reference picture index information. The picture indicated by the reference picture index in the reference picture list for the current block may be derived as the reference picture referenced for inter-frame prediction of the current block.

[0186] In addition, the motion information of the current block can be derived without the candidate list configuration as described below, and in this case, the motion information of the current block can be derived according to the process disclosed in the prediction mode to be described below. In this case, the candidate list configuration can be omitted.

[0187] The decoding device may generate prediction samples for the current block based on the motion information of the current block (S1003). In this case, a reference picture may be derived based on a reference picture index of the current block, and the prediction samples of the current block may be derived by using samples of the reference block indicated by the motion vector of the current block on the reference picture. In this case, as described below, in some cases, a prediction sample filtering process may be further performed on all or some of the prediction samples of the current block.

[0188] For example, the inter-frame prediction unit (inter-frame predictor) 260 of the decoding device may include a prediction mode determination unit 261, a motion information derivation unit 262 and a prediction sample derivation unit 263, and the prediction mode determination unit 261 can determine the prediction mode for the current block based on the received prediction mode information, the motion information derivation unit 262 can derive the motion information (motion vector and / or reference picture index) of the current block based on the information about the received motion information, and the prediction sample derivation unit 263 can derive the prediction sample of the current block.

[0189] The decoding device generates residual samples for the current block based on the received residual information (S1004). The decoding device may generate reconstructed samples for the current block based on the predicted samples and the residual samples, and generate a reconstructed picture based on the generated reconstructed samples (S1005). Thereafter, as described above, the in-loop filtering process may be further applied to the reconstructed picture.

[0190] As described above, the inter prediction process may include an inter prediction mode determining step, a motion information deriving step according to the determined prediction mode, and a prediction performing (prediction sample generating) step based on the derived motion information.

[0191] Determination of inter-frame prediction mode

[0192] Various inter-frame prediction modes can be used to predict the current block in the picture. For example, various modes including merge mode, skip mode, MVP mode, and affine mode can be used. Decoder side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, etc. can be further used as ancillary mode. Affine mode can be called affine motion prediction mode. MVP mode can be called advanced motion vector prediction (AMVP) mode.

[0193] Prediction mode information indicating the inter-frame prediction mode of the current block can be signaled from the encoding device to the decoding device. The prediction mode information can be included in the bitstream and received by the decoding device. The prediction mode information may include index information indicating one of multiple candidate modes. Alternatively, the inter-frame prediction mode can be indicated by hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags.

[0194] For example, a skip flag may be signaled to indicate whether skip mode is applied, a merge flag may be signaled to indicate whether merge mode is applied when skip mode is not applied, and an MVP mode may be applied when merge mode is not applied, or a flag for additional distinction may be further signaled. Affine mode may be signaled as an independent mode or as a dependent mode of a merge mode or MVP mode. For example, affine mode may be configured as a candidate in a merge candidate list or an MVP candidate list as described below.

[0195] Derivation of motion information based on inter prediction mode

[0196] Inter-frame prediction can be performed by using the motion information of the current block. The encoding device can derive the optimal motion information for the current block through a motion estimation process. For example, the encoding device can search for a similar reference block with high correlation in a predetermined search range in the reference picture in units of fractional pixels by using the original block in the original picture for the current block, and derive motion information through the searched reference block. The similarity of the blocks can be derived based on the difference in sample values ​​based on the phase. For example, the similarity of the blocks can be calculated based on the SAD between the current block (or the template of the current block) and the reference block (or the template of the reference block). In this case, the motion information can be derived based on the reference block with the smallest SAD in the search area. The derived motion information can be signaled to the decoding device according to various methods based on the inter-frame prediction mode.

[0197] Merge mode and skip mode

[0198] Figure 12 is a diagram for describing neighboring blocks used in a merge mode or a skip mode as an embodiment to which the present disclosure is applied.

[0199] When the merge mode is applied, the motion information of the current prediction block is not directly transmitted, and the motion information of the current prediction block is derived by using the motion information of the adjacent prediction blocks. Therefore, flag information indicating the use of the merge mode and a merge index indicating which adjacent prediction block to use are transmitted to indicate the motion information of the current prediction block.

[0200] The encoder can search for a merge candidate block for deriving motion information of the current prediction block in order to perform a merge mode. For example, up to five merge candidate blocks can be used, but the present disclosure is not limited thereto. In addition, the maximum number of merge candidate blocks can be sent in a slice header (or tile group header), and the present disclosure is not limited thereto. After finding the merge candidate block, the encoder can generate a merge candidate list and select the merge candidate block with the minimum cost among the merge candidate blocks as the final merge candidate block.

[0201] The present disclosure provides various implementations for merging candidate blocks constituting a merging candidate list.

[0202] As a merge candidate list, for example, five merge candidate blocks can be used. For example, four spatial merge candidates and one temporal merge candidate can be used. As a specific example, in the case of spatial merge candidates, Figure 12 The blocks shown may be used as spatial merging candidates.

[0203] Figure 13is a flowchart illustrating a method for configuring a merge candidate list according to an embodiment to which the present disclosure is applied.

[0204] Reference Figure 13 , the encoding device (encoder / decoder) inserts the spatial merge candidates obtained by searching the spatial adjacent blocks of the current block into the merge candidate list (S1301). For example, the spatial adjacent blocks may include the lower left adjacent block, the left adjacent block, the upper right adjacent block, the upper adjacent block, and the upper left adjacent block of the current block. However, this is an example, and in addition to the above-mentioned spatial adjacent blocks, additional adjacent blocks including the right adjacent block, the lower adjacent block, and the lower right adjacent block may be used as spatial adjacent blocks. The encoding device can derive available blocks by searching the spatial adjacent blocks based on priority, and can derive motion information of the detected blocks as spatial merge candidates. For example, the encoder and the decoder may search in the order of A1, B1, B0, A0, and B2. Figure 12 The five blocks shown are indexed in sequence and the available candidates are configured as a merge candidate list.

[0205] The encoding device inserts a temporal merge candidate obtained by searching for temporally adjacent blocks of the current block into a merge candidate list (S1302). The temporally adjacent blocks may be located in a reference picture that is a different picture from the current picture in which the current block is located. The reference picture in which the temporally adjacent blocks are located may be referred to as a collocated picture or col picture. The temporal adjacent blocks may be searched for in the col picture in the order of the lower right corner adjacent block and the lower right middle block of the collocated block of the current block.

[0206] In addition, when motion data compression is applied, specific motion information can be stored as representative motion information in the col picture for each predetermined storage unit. In this case, it is not necessary to store the motion information of all blocks in the predetermined storage unit, and as a result, a motion data compression effect can be obtained. In this case, the above-mentioned predetermined storage unit can be predetermined for each 16×16 sample unit or 8×8 sample unit, or the size information of the predetermined storage unit can be signaled from the encoder to the decoder. When motion data compression is applied, the motion information of the temporally adjacent blocks can be replaced with the representative motion information of the predetermined storage unit where the temporally adjacent blocks are located.

[0207] In other words, in this case, in terms of implementation, a temporal merge candidate can be derived based on the motion information of a prediction block (instead of the prediction block located at the coordinates of the temporally adjacent block) that covers a position where the coordinates (top-left sample position) of the temporally adjacent blocks are arithmetically right-shifted by a predetermined value and then arithmetically left-shifted. For example, when the predetermined storage unit is a 2n×2n sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb>>n)<<n), (yTnb>>n)<<n)) can be used for the temporal merge candidate.

[0208] Specifically, for example, when the predetermined storage unit is a 16×16 sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb>>4)<<4), (yTnb>>4)<<4)) can be used for the temporal merge candidate. Alternatively, for example, when the predetermined storage unit is an 8×8 sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb>>3)<<3), (yTnb>>3)<<3)) can be used for the temporal merge candidate.

[0209] The encoding device can check whether the number of current merge candidates is less than the maximum number of merge candidates (S1303). The maximum number of merge candidates can be predefined or signaled from the encoder to the decoder. For example, the encoder can generate information about the maximum number of merge candidates, encode the generated information, and transmit the encoded information to the decoder in the form of a bitstream. When the maximum number of merge candidates is completely exhausted, the subsequent candidate addition process may not be performed.

[0210] As a check result, when the number of current merge candidates is less than the maximum number of merge candidates, the encoding device inserts additional merge candidates into the merge candidate list (S1304). The additional merge candidates can include, for example, ATMVP, combined bi-prediction merge candidates (when the slice type of the current slice is type B), and / or zero vector merge candidates.

[0211] As a check result, when the number of current merge candidates is not less than the maximum number of merge candidates, the encoding device can terminate the configuration of the merge candidate list. In this case, the encoder can select an optimal merge candidate among the merge candidates constituting the merge candidate list based on the rate-distortion (RD) cost, and signal selection information (e.g., a merge index) indicating the selected merge candidate to the decoder. The decoder can select the optimal merge candidate based on the merge candidate list and the selection information.

[0212] The motion information of the selected merge candidate may be used as the motion information of the current block as described above, and the prediction samples of the current block may be derived based on the motion information of the current block. The encoder may derive residual samples of the current block based on the prediction samples and signal the residual information for the residual samples to the decoder. The decoder may generate reconstructed samples based on the residual samples derived based on the residual information and the prediction samples as described above, and generate a reconstructed picture based on the generated reconstructed samples.

[0213] When skip mode is applied, the motion information of the current block can be derived by the same method as the case of applying merge mode as described above. However, when skip mode is applied, the residual signal for the corresponding block is omitted, and as a result, the prediction sample can be directly used as the reconstructed sample.

[0214] MVP pattern

[0215] Figure 14 is a flowchart illustrating a method for configuring a merge candidate list according to an embodiment to which the present disclosure is applied.

[0216] When the motion vector prediction (MVP) mode is applied, the reconstructed spatial neighboring blocks (e.g., the above Figure 12 A motion vector predictor (MVP) candidate list is generated based on the motion vectors of the reconstructed spatial neighboring blocks and / or the motion vectors corresponding to the temporal neighboring blocks (or Col blocks). In other words, the reconstructed motion vectors of the spatial neighboring blocks and / or the motion vectors corresponding to the temporal neighboring blocks can be used as motion vector predictor candidates.

[0217] The information about the prediction may include selection information (e.g., an MVP flag or an MVP index) indicating the optimal motion vector predictor candidate selected from among the motion vector predictor candidates included in the list. In this case, the predictor can select the motion vector predictor of the current block from among the motion vector predictor candidates included in the motion vector candidate list by using the selection information. The predictor of the encoding device can obtain the motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, and encode the obtained MVD and output the encoded MVD in the form of a bitstream. In other words, the MVD can be obtained by subtracting the motion vector predictor from the motion vector of the current block. In this case, the predictor of the decoding device can obtain the motion vector difference included in the information about the prediction, and derive the motion vector of the current block by adding the motion vector difference and the motion vector predictor. The predictor of the decoding device can obtain or derive the reference picture index indicating the reference picture from the information about the prediction. For example, it can be as follows Figure 14 The configuration shown is a motion vector prediction value candidate list.

[0218] Affine motion prediction

[0219] Figure 15 An example of a motion model according to one embodiment of the present disclosure is shown.

[0220] Conventional image compression technology (e.g., High Efficiency Video Coding (HEVC)) uses a motion vector to represent the motion of a coding block. Although the optimal motion in units of blocks can be represented for each block in the method using one motion vector, the optimal motion may not actually be the optimal motion for each pixel. Therefore, if the optimal motion vector is determined in units of pixels, the coding efficiency will be improved. Therefore, one embodiment of the present disclosure describes a motion prediction method for encoding or decoding a video signal using a multi-motion model. In particular, a motion vector at two to four control points can be used to represent a motion vector in each pixel unit or sub-block unit of a block. The prediction scheme using motion vectors of such multiple control points may be referred to as affine motion prediction, affine prediction, or the like.

[0221] The affine motion model according to one embodiment of the present disclosure can represent four motion models such as Figure 15 The four motion models shown. An affine motion model that represents three motions (translation, scaling, and rotation) among the motions that can be represented by an affine motion model is called a similarity (or simplified) affine motion model. When describing the embodiments of the present disclosure, for ease of description, the similarity (or simplified) affine motion model is mainly described, but the present disclosure is not limited thereto.

[0222] Figure 16 An example of control point motion vectors for affine motion prediction according to one embodiment of the present disclosure is shown.

[0223] like Figure 16 As shown in , affine motion prediction can use a pair of two control point motion vectors (CPMVs) v_0 and v_1 to determine the motion vector of a pixel position (or sub-block) included in a block. In this case, a set of motion vectors can be referred to as an affine motion vector field (MVF). In this case, the affine motion vector field can be determined using the following formula 1.

[0224] [Formula 1]

[0225]

[0226] In Formula 1, v_0 (v_0 = {v_0x, v_0y}) indicates the motion vector CPMV0 at the first control point at the upper left position of the current block 1300. v_1 (v_1 = {v_1x, v_1y}) indicates the motion vector CPMV1 at the second control point at the upper right position of the current block 1300. In addition, w indicates the width of the current block 1300. v (v = {v_x, v_y}) indicates the motion vector at the {x, y} position. The motion vector in the sub-block (or pixel) unit can be derived using Formula 1. In one embodiment, the motion vector accuracy can be rounded to 1 / 16 accuracy.

[0227] Figure 17 An example of a motion vector for each subblock of a block to which affine motion prediction according to one embodiment of the present disclosure has been applied is shown.

[0228] Reference Figure 17 During encoding or decoding, the affine motion vector field (MVF) can be determined in units of pixels or blocks. That is, in affine motion prediction, the motion vector of the current block can be derived in units of pixels or sub-blocks.

[0229] If the affine motion vector field is determined in units of pixels, a motion vector can be obtained based on each pixel value. If the affine motion vector field is determined in units of blocks, a motion vector of a block can be obtained based on the center pixel value of the corresponding block. Figure 17 As shown, it is assumed that the affine motion vector field (MVF) is determined in units of 4×4 blocks. However, this is for convenience of description and does not limit the embodiments of the present disclosure. Figure 17 An example is shown in which a coding block is constituted by 16×16 samples and an affine motion vector field (MVF) is determined in units of 4×4 sized blocks.

[0230] Affine motion prediction may include an affine merge mode (or AF_MERGE) and an affine inter mode (or AF_INTER). The AF_INTER mode may include an AF_4_INTER mode using a four-parameter motion model and an AF_6_INTER mode using a six-parameter motion model.

[0231] Affine merge mode

[0232] AF_MERGE determines the control point motion vector (CPMV) based on the affine motion model of neighboring blocks that are coded as affine motion prediction. Affine-coded neighboring blocks in search order can be used for AF_MERGE. When one or more neighboring blocks are coded as affine motion prediction, the current block can be coded as AF_MERGE.

[0233] That is, if the affine merge mode is applied, the CPMV of the neighboring blocks can be used to derive the CPMV of the current block. In this case, the CPMV of the neighboring blocks can be used as the CPMV of the current block without any changes. The CPMV of the neighboring blocks is modified based on the size of the neighboring blocks and the size of the current block, and can be used as the CPMV of the current block.

[0234] Figure 18 An example of neighboring blocks used in affine motion prediction in affine merge mode according to one embodiment of the present disclosure is shown.

[0235] In the affine merging (AF_MERGE) mode, the encoder may perform encoding as in the following process.

[0236] Step-1: Scan the neighboring blocks A 1810, B 1820, C 1830, D 1840 and E 1850 of the current coding block 1800 in alphabetical order. The block that is first encoded in the affine prediction mode according to the scanning order is determined as the candidate block for affine merging (AF_MERGE)

[0237] Step 2: Determine the affine motion model using the control point motion vector (CPMV) of the determined candidate block

[0238] Step 3: Determine the control point motion vector (CPMV) of the current block 1800 according to the affine motion model of the candidate block, and determine the MVF of the current block 1800.

[0239] Figure 19 An example of a block in which affine motion prediction is performed using a neighboring block to which affine motion prediction according to an embodiment of the present disclosure has been applied is shown.

[0240] For example, Figure 19 As shown in , if block A 1920 is encoded in an affine mode, after block A 1920 is determined as a candidate block, an affine motion model may be derived using control point motion vectors (CPMVs) (e.g., v2 and v3) of block A 1920, and control point motion vectors (CPMVs) v0 and v1 of the current block 1900 may be determined. An affine motion vector field (MVF) of the current block 1900 may be determined based on the control point motion vectors (CPMVs) of the current block 1900, and encoding may be performed.

[0241] Figure 20 is a diagram for describing a method for generating a merge candidate list using a peripheral affine coding block according to one embodiment of the present disclosure.

[0242] Reference Figure 20 If affine merge candidates are used to determine CPMV pairs, then one can use Figure 20 The candidate of the candidate shown. Figure 20 , it is assumed that the scanning order of the candidate list is set to A, B, C, D, and E. However, the present disclosure is not limited thereto, and the scanning order may be pre-set in various orders.

[0243] In one embodiment, if the number of candidates (hereinafter referred to as affine candidates) encoded in affine mode (or affine prediction) available in adjacent blocks (i.e., A, B, C, D, and E) is 0, the affine merge mode of the current block can be skipped. If the number of available affine candidates is one (e.g., A), the motion model of the corresponding candidate can be used to derive the control point motion vectors CPMV_0 and CPMV_1 of the current block. In this case, the index indicating the corresponding candidate may not be necessary (or encoded). If the number of available affine candidates is two or more, the two candidates can be configured as a candidate list for AF_MERGE in a scanning order. In this case, candidate selection information such as an index indicating the candidate selected within the candidate list can be signaled. The selection information can be a flag or index information and can be marked as AF_MERGE_flag and AF_merge_idx, etc.

[0244] In one embodiment of the present disclosure, motion compensation for the current block can be performed based on the size of the sub-block. In this case, the sub-block size of the affine block (current block) is derived. If both the width and height of the sub-block are greater than 4 luma samples, a motion vector is derived for each sub-block, and DCT-IF-based motion compensation can be invoked for that sub-block (1 / 16 pixel for luma and 1 / 32 for chroma). Otherwise, motion compensation based on an enhanced bilinear interpolation filter is invoked for the entire affine block.

[0245] In one embodiment of the present disclosure, when the merge / skip flag is true and the width and height of the CU are both greater than or equal to 8, the affine flag at the CU level is signaled in the bitstream to indicate whether the affine merge mode is used. And when the CU is encoded as AF_MERGE, a merge candidate index with a maximum value of 5 is signaled to specify which motion information candidate in the affine merge candidate list is used for the CUA.

[0246] Figure 21 and Figure 22 is a diagram for describing a method of configuring an affine merging candidate list using neighboring blocks encoded by affine prediction according to one embodiment of the present disclosure.

[0247] Reference Figure 21,The affine merge candidate list is constructed according to the following steps.

[0248] 1) Insert model-based affine candidates

[0249] Model-based affine candidates are candidates derived from valid neighboring reconstructed blocks coded in affine mode. Figure 21 As shown, the scanning order of candidate blocks is from left (A), top (b), top right (C), bottom left (D) to top left (E).

[0250] If the adjacent lower left block A is encoded in a 6-parameter affine mode, the motion vectors v_4, v_5, and v_6 of the upper left, upper right, and lower left corners of the CU containing block A are obtained. And the motion vectors v_0, v_1, and v_2 of the upper left corner of the current CU are calculated based on v_4, v_5, and v_6 through the six-parameter affine model.

[0251] If the adjacent lower left block A is encoded in a 4-parameter affine mode, the motion vectors v_4 and v_5 of the upper left and upper right corners of the CU containing block A are obtained. And the motion vectors v_0 and v_1 of the upper left corner of the current CU are calculated based on v_4 and v_5 through the 4-parameter affine model.

[0252] 2) Insert affine candidates based on control points

[0253] Reference Figure 21 ,Control point based candidates refer to constructing candidates by combining ,the adjacent motion information of each control point.

[0254] First, from Figure 21 The motion information for a control point is derived using the specified spatial and temporal neighbors shown in [1]. CP_k (k = 1, 2, 3, 4) represents the kth control point. A, B, C, D, E, F, and G are the spatial locations used to predict CP_k (k = 1, 2, 3); H is the temporal location used to predict CP4.

[0255] The coordinates of CP_1, CP_2, CP_3 and CP_4 are (0, 0), (W, 0), (H, 0) and (W, H), respectively, where W and H are the width and height of the current block.

[0256] The motion information of each control point is obtained according to the following priority order.

[0257] For CP_1, the priority is A→B→C. If available, use A. Otherwise, if B is available, use B. If neither A nor B is available, use C. If none of the three candidates are available, the motion information of CP1 cannot be obtained.

[0258] For CP_2, the check priority is E→D;

[0259] For CP_3, the check priority is G→F;

[0260] For CP_4, use H.

[0261] Second, a motion model is constructed using a combination of control points.

[0262] The motion vectors of two control points are required to compute the transformation parameters in a 4-parameter affine model. Two control points can be selected from one of the following six combinations: {CP_1, CP_4}, {CP_2, CP_3}, {CP_1, CP_2}, {CP_2, CP_4}, {CP_1, CP_3}, {CP_3, CP_4}. For example, a 4-parameter affine motion model constructed using control points CP_1 and CP_2 is labeled Affine(CP_1, CP_2).

[0263] The motion vectors of three control points are required to calculate the transformation parameters in the 6-parameter affine model. The three control points can be selected from one of the following four combinations ({CP_1, CP_2, CP_4}, {CP_1, CP_2, CP_3}, {CP_2, CP_3, CP_4}, {CP_1, CP_3, CP_4}). For example, the 6-parameter affine motion model constructed using the CP_1, CP_2, and CP_3 control points is labeled Affine(CP_1, CP_2, CP_3).

[0264] Furthermore, in one embodiment of the present disclosure, in the affine merge mode, if there is an affine merge candidate, it can always be regarded as a six-parameter affine mode.

[0265] Affine inter mode

[0266] Figure 23 An example of neighboring blocks used in affine motion prediction in affine inter mode according to one embodiment of the present disclosure is shown.

[0267] Reference Figure 23 Affine motion prediction can include affine merge mode (or AF_MERGE) and affine inter mode (or AF_INTER). In affine inter mode (AF_INTER), after determining two control point motion vector predictions (CPMVP) and CPMV, the control point motion vector difference CPMVD corresponding to the difference can be sent from the encoder to the decoder. The specific process of encoding the affine inter mode AF_INTER can be as follows.

[0268] Step 1: Identify two CPMVP candidates

[0269] Step 1.1: Determine the maximum value of the twelfth CPMVP candidate combination (refer to Equation 2 below)

[0270] [Formula 2]

[0271] {(v0,v1,v2)|v0={v A ,v B ,v C},v1={v D , v E}, v2={v F , v G}}

[0272] In Equation 2, v_0 indicates the motion vector CPMV0 at the upper left control point 2310 of the current block 2300. v_1 indicates the motion vector CPMV1 at the upper right control point 2311 of the current block 2300. v_2 indicates the motion vector CPMV2 at the lower left control point 2312 of the current block 2300. v_A represents the motion vector of the neighboring block A 2320 located to the upper left of the upper left control point 2310 of the current block 2300. v_B indicates the motion vector of the neighboring block B 2322 located above the upper left control point 2310 of the current block 2300. v_C indicates the motion vector of the neighboring block C 2324 located to the left of the upper left control point 2310 of the current block 2300. v_D indicates the motion vector of the neighboring block D 2326 located above the upper right control point 2311 of the current block 2300. v_E indicates the motion vector of the neighboring block E 2328 that is adjacent to the upper right of the upper right control point 2311 of the current block 2300. v_F indicates the motion vector of the neighboring block F 2330 that is adjacent to the left of the lower left control point 2312 of the current block 2300. v_G indicates the motion vector of the neighboring block G 2332 that is adjacent to the lower left of the lower left control point 2312 of the current block 2300.

[0273] Step 1.2: Use the first two candidates ranked based on the smaller difference value (DV) in the CPMVP candidate combination (refer to Equation 3 below)

[0274] [Formula 3]

[0275] DV=|(v 1x -v 0x )*h-(v 2y -v 0y )*w|+|(v 1y -v 0y )*h+(v 2x -v 0x )*w|

[0276] v_0x indicates the x-axis element of the motion vector V_0 or CPMV_0 at the upper left control point 2310 of the current block 2300. v_1x indicates the x-axis element of the motion vector V_1 or CPMV_1 at the upper right control point 2311 of the current block 2300. v_2x indicates the x-axis element of the motion vector V_2 or CPMV_2 at the lower left control point 2312 of the current block 2300. v_0y indicates the y-axis element of the motion vector V_0 or CPMV_0 at the upper left control point 2310 of the current block 2300. v_1y indicates the y-axis element of the motion vector V_1 or CPMV_1 at the upper right control point 2311 of the current block 2300. v_2y indicates the y-axis element of the motion vector V_2 or CPMV_2 at the lower left control point 2312 of the current block 2300. W indicates the width of the current block 2300. H indicates the height of the current block 2300.

[0277] Step 2: When the control point motion vector prediction value (CPMVP) pair candidate is less than 2, use the AMVP candidate list

[0278] Step 3: Determine the control point motion vector prediction value (CPMVP) of each of the two candidates and optimally select the candidate with the smaller value and CPMVP by comparing the RD cost

[0279] Step 4: Send the index and control point motion vector difference (CPMVD) corresponding to the best candidate

[0280] In one embodiment of the present disclosure, a process for constructing CPMVP candidates is provided in AF_INTER. Similar to AMVP, the number of candidates is 2, and an index indicating the position of the candidate list is signaled.

[0281] The construction process of the CPMVP candidate list is as follows:

[0282] 1) Scan the neighboring blocks to check whether they are coded as affine motion prediction. If the scanned block is coded as affine prediction, derive the motion vector pair of the current block from the affine motion model of the scanned neighboring blocks until the number of candidates is 2.

[0283] 2) If the number of candidates is less than two, the candidate construction process is performed. In addition, in one embodiment of the present disclosure, a four-parameter (two control points) affine inter-frame model is used to predict content using a motion model of magnification / reduction and rotation. Figure 16 As shown, the affine motion field of the block is described by two control point motion vectors.

[0284] The motion vector field (MVF) of a block is described by Equation 1 described above.

[0285] In the prior art, the advanced motion vector prediction (AMVP) mode requires signaling of the motion vector prediction (MVP) index and motion vector difference (MVD). When the AMVP mode is applied in the present disclosure, affine_flag is signaled to indicate whether affine prediction is used. If affine prediction is applied, the syntax of inter_dir, ref_idx, mvp_index and two MVDs (mvd_x and mvd_y) is signaled. An affine MVP pair candidate list containing two affine MVP pairs is generated. The signaled mvp_index is used to select one of them. The affine MVP pair is generated by two affine MVP candidates. One is a spatially inherited affine candidate and the other is a corner derived affine candidate. If the adjacent CU is encoded in affine mode, a spatially inherited affine candidate can be generated. The affine motion model of the adjacent affine coding block is used to generate the motion vector of the two control point MVP pairs. The MVs of the two control point MVP pairs of the spatially inherited affine candidate are derived by using the following formula.

[0286] [Formula 4]

[0287] V 0x =V B0x +(V B2_x -V B0x )*(posCurCU_Y-posRefCU_Y) / RefCU_height+(V B1x -

[0288] V B0x )*(posCurCU_X-posRefCU_X) / RefCU_width

[0289] [Formula 5]

[0290] V 0y =V B0y +(V B2_y -V B0y )*(posCurCU_Y-posRefCU_Y) / RefCU_height+(V B1y -

[0291] V B0y )*(posCurCU_X-posRefCU_X) / RefCU_width

[0292] Among them, V_B0, V_B1 and V_B2 can be replaced by the upper left MV, upper right MV and lower left MV of any reference / neighboring CU,

[0293] (posCurCU_X, posCurCU_Y) is the position of the top left sample of the current CU relative to the top left sample of the frame, and (posRefCU_X, posRefCU_Y) is the position of the top left sample of the reference / neighboring CU relative to the top left sample of the frame.

[0294] [Formula 6]

[0295] V 1x =V B0x +(V B1x -V B0x )*CU_width / RefCU_width

[0296] [Formula 7]

[0297] V 1y =V B0y +(V B1y -V B0y )*CU_width / RefCU_width

[0298] Figure 24 An example of neighboring blocks used for affine motion prediction in affine inter mode according to one embodiment of the present disclosure is shown.

[0299] Reference Figure 24 , if the number of MVP pairs is less than 2, then the affine candidates are derived using angles. Figure 24 As shown, adjacent motion vectors are used to derive affine MVP pairs. For the first angle-derived affine candidate, the first available MV in set A (A0, A1, and A2) and the first available MV in set B (B0 and B1) are used to construct the first MVP pair. For the second angle-derived affine candidate, the first available MV in set A and the first available MV in set C (C0 and C1) are used to calculate the MV of the upper right control point. The first available MV in set A and the calculated upper right control point MV are the second MVP pair.

[0300] In one embodiment of the present disclosure, two candidate sets with two (three) candidates {mv_0, mv_1} ({mv_0, mv_1, mv_2) are used to predict two (three) control points of the affine motion model. Given the motion vector difference vectors mvd_0, mvd_1, mvd_2, the control points can be calculated using the following formula.

[0301] [Formula 8]

[0302]

[0303] Figure 25 and Figure 26FIG. 1 is a diagram illustrating a method of deriving motion vector candidates using motion information of neighboring blocks in an affine inter mode according to an embodiment of the present disclosure.

[0304] The affine candidate list is sequentially added by extending the affine motion from spatially neighboring blocks (extrapolated affine candidates), the combination of motion vectors from spatially neighboring blocks (virtual affine candidates), and HEVC motion vector prediction (MVP) candidates until there are two affine MVPs in the candidate list. The candidate set is constructed as follows:

[0305] 1. Derived from the affine motion of neighboring blocks at most two different sets of affine MV prediction values. Check if Figure 25 Neighboring blocks A0, A1, B0, B1, and B2 are shown. If the neighboring block is encoded using an affine motion model and its reference frame is the same as the reference frame of the current block, the MVs at two (for a 4-parameter affine model) or three (for a 6-parameter affine model) control points of the current block are derived from the affine model of the neighboring item.

[0306] 2. Figure 29 The neighboring blocks used to generate the virtual affine candidate set are shown. The neighboring MVs are divided into three groups: S_0 = {mv_A, mv_B, mv_C}, S_1 = {mv_D, mv_E}, and S_2 = {mv_F, mv_G}. mv_0 is the first MV in S0 that references the same reference picture as the current block; mv_1 is the first MV in S1 that references the same reference picture as the current block; and mv_2 is the first MV in S2 that references the same reference picture as the current block.

[0307] If only mv_0 and mv_1 can be found, mv_2 can be derived by using the following formula.

[0308] [Formula 9]

[0309]

[0310] Referring to Equation 9, the current block size is W×H.

[0311] If only mv_0 and mv_2 can be found, mv_1 is derived by using the following formula.

[0312] [Equation 10]

[0313]

[0314] In one embodiment of the present disclosure, affine inter prediction may be performed according to the following sequence.

[0315] Input: affine motion parameters, reference image samples

[0316] Output: CU's prediction block

[0317] ·deal with

[0318] – Derivation of sub-block size for affine blocks

[0319] – If both the width and height of the sub-block are greater than 4 luma samples,

[0320] – For each sub-block

[0321] – Derivation of motion vectors for sub-blocks.

[0322] – Invoke DCT-IF based motion compensation for sub-blocks (1 / 16 pixel for luma and 1 / 32 for chroma)

[0323] Otherwise, invoke motion compensation based on enhanced bilinear interpolation filter for the entire affine block

[0324] In addition, in one embodiment of the present disclosure, when the merge / skip flag is false and the width and height of the CU are both greater than or equal to 8, the affine flag at the CU level is signaled in the bitstream to indicate whether the affine inter mode is used. And when the CU is encoded in affine inter mode, the model flag is signaled to specify whether a 4-parameter affine model or a 6-parameter affine model is used for the CU. If the model flag is true, the AF_6_INTER mode (6-parameter affine model) is applied and 3 MVDs will be parsed; otherwise, the AF_4_INTER mode (4-parameter affine model) will be applied and 2 MVDs will be parsed.

[0325] In AF_4_INTER mode, similar to the affine merge mode, affine motion vector pairs extrapolated from neighboring blocks encoded in affine mode are constructed and first inserted into the candidate list.

[0326] Afterwards, if the size of the candidate list is less than 4, a candidate with a motion vector pair {(v_0, v_1)|v0={v_A, v_B, v_c}, v_1={v_D, v_E}} is constructed using neighboring blocks. Figure 22 As shown, v_0 is selected from the motion vector of block A, B, or C. The motion vector from the neighboring block is scaled according to the reference list and the relationship between the POC of the reference for the neighboring block, the POC of the reference for the current CU, and the POC of the current CU. And the method of selecting v_1 from neighboring blocks D and E is similar. When the candidate list is greater than 4, the candidates are first sorted according to the consistency of the neighboring motion vectors (the similarity of the two motion vectors in a pair of candidates), and only the first four candidates are retained.

[0327] If the number of candidate lists is less than 4, the lists are padded with motion vector pairs composed by copying each of the AMVP candidates.

[0328] In AF_6_INTER mode, similar to affine merge mode, an affine motion vector triplet extrapolated from neighboring blocks encoded in affine mode is constructed and inserted into the candidate list first.

[0329] Afterwards, if the size of the candidate list is less than 4, the candidate with motion vector triple (((v_0, v_1, v_2)|v0={v_A, v_B, v_C}, v1={v_D, v_E}, v2={v_G, v_H}}) is constructed using neighboring blocks. Figure 22 As shown, v_0 is selected from the motion vector of block A, B, or C. The motion vector from the neighboring block is scaled according to the reference list and the relationship between the POC of the reference for the neighboring block, the POC of the reference for the current CU, and the POC of the current CU. And the method of selecting v_1 from neighboring blocks D and E and v_2 from F and G is similar. When the candidate list is greater than 4, the candidates are first sorted according to the consistency of the neighboring motion vectors (the similarity of the two motion vectors in the triplet candidate), and only the first four candidates are retained.

[0330] If the number of candidate lists is less than 4, the lists are populated with motion vector triplets composed by copying each of the AMVP candidates.

[0331] After deriving the CPMV of the current CU, according to the number of affine parameters, the MVF of the current CU is generated according to the following formula 11 for the 4-parameter affine model, and the MVF of the current CU is generated according to the following formula 12 for the 6-parameter affine model.

[0332] [Equation 11]

[0333]

[0334] [Equation 12]

[0335]

[0336] The sub-block size M×N is derived as shown in Equation 13 below, where MvPre is the motion vector fraction accuracy (1 / 16).

[0337] [Equation 13]

[0338]

[0339] After derivation from Equation 12, M and N should be adjusted downward if necessary to make them divisors of w and h, respectively. If M or N is less than 8, WIF is applied; otherwise, sub-block-based affine motion compensation is applied.

[0340] Figure 27 An example of a method of deriving an affine motion vector field in units of sub-blocks according to one embodiment of the present disclosure is shown.

[0341] Reference Figure 27 , in order to derive the motion vector of each M×N sub-block, as Figure 27 As shown, the motion vector of the center sample of each sub-block is calculated according to Equation 11 or Equation 12 and rounded to a fractional accuracy of 1 / 16. Then, an SHVC upsampling interpolation filter is applied to generate a prediction for each sub-block using the derived motion vector.

[0342] The SHVC upsampling interpolation filter with the same filter length and normalization factor as the HEVC motion compensation interpolation filter is used as the motion compensation interpolation filter for the additional fractional pixel positions. The chroma component motion vector precision is 1 / 32 sample, and the additional interpolation filter for the 1 / 32 pixel fractional position is derived by taking the average of two adjacent 1 / 16 pixel fractional position filters.

[0343] AF_MERGE mode is selected on the encoder side in a similar way to regular merge mode selection. First, a candidate list is constructed and the minimum RD cost of the candidate intra is compared with the RD costs of other inter modes. The comparison result determines whether AF_MERGE is applied.

[0344] For AF_4_INTER mode, an RD cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) for the current CU. After determining the CPMVP for the current affine CU, affine motion estimation is applied and the control point motion vector (CPMV) is found. The difference between the CPMV and the CPMVP is then determined.

[0345] On the encoder side, the AF_6_INTER mode is verified only if the AF_MERGE or AF_4_INTER mode was selected as the best mode in the previous mode selection stage.

[0346] In one embodiment of the present disclosure, the affine AMVP mode may be implemented as follows:

[0347] 1) AFFINE_MERGE_IMPROVE: This improvement attempts to find the neighboring block with the largest coding unit size as the affine merge candidate instead of finding the first neighboring block in affine mode.

[0348] 2) AFFINE_AMVP_IMPROVE: Similar to the regular AMVP process, the neighboring blocks in the affine mode are added to the affine AMVP candidate list.

[0349] The detailed construction process of the affine AMVP candidate list is as follows.

[0350] First, check whether the lower left neighboring block is using an affine motion model and has the same reference index as the current reference index. If not, then check the left neighboring block in the same manner. If not, then check whether the lower left neighboring block is using an affine motion model and has a different reference index. If so, add the scaled affine motion vector to the reference picture list. If not, then check the left neighboring block in the same manner.

[0351] Next, the upper right neighboring block, the upper neighboring block, and the upper left neighboring block will then be checked in the same manner.

[0352] If we find two candidates after the above process, we will complete the construction of the affine AMVP candidate list. If we do not find two candidates, we will perform the original processing in the JEM software to construct the affine AMVP candidate list.

[0353] 3) AFFINE_SIX_PARAM: In addition to the four-parameter affine motion model, a six-parameter affine motion model is added as an additional model.

[0354] The six-parameter affine motion model is derived by using the following formula.

[0355] [Equation 14]

[0356]

[0357] Since the above motion model has six parameters, three motion vectors are needed to determine the model: the upper left position MV_0, the upper right position MV_1, and the lower left position MV_2. These three motion vectors are determined in a similar manner to the two motion vectors in the four-parameter affine motion model. Note that the affine model merging is always set to a six-parameter affine motion model.

[0358] 4) AFFINE_CLIP_REMOVE: Removes motion vector constraints for all affine motion vectors, allowing the motion compensation process to handle motion vector constraints itself.

[0359] Affine motion model

[0360] As mentioned above, various affine motion models can be used or considered in affine inter prediction. For example, an affine motion model can be represented as Figure 15 An affine motion model that represents three motions (translation, scaling, and rotation) among the motions that can represent the affine motion model may be referred to as a similar (or simplified) affine motion model. Depending on which affine motion model is used, the number of CPMVs derived and / or the method of deriving the sample / sub-block unit MV of the current block may differ.

[0361] A motion model can be used. In AF_INTER, in addition to the existing four-parameter motion model in JEM, a six-parameter motion model is proposed. The following equation 15 describes the six-parameter affine motion model.

[0362] [Equation 15]

[0363] x′=a*x+b*y+c

[0364] y′=d*x+e*y+f

[0365] Here, coefficients a, b, c, d, e, and f are affine motion parameters, and (x, y) and (x', y') are the pixel position coordinates before and after the transformation of the affine motion model. In order to use the affine motion model in video coding, if CPMV0, CPMV1, and CPMV2 are the MVs of CP0 (upper left), CP1 (upper right), and CP2 (lower left), then Equation 16 can be described as:

[0366] [Equation 16]

[0367]

[0368] Where CPMV_0 = {v_0x, v_0y}, CPMV_1 = {v_1x, v_1y}, CPMV_2 = {v_2x, v_2y}, w and h are the width and height of the coding block respectively. Equation 16 describes the motion vector field (MVF) of the block.

[0369] A flag is parsed at the CU level to indicate whether a four-parameter or six-parameter affine motion model is used when the neighboring block is coded as affine prediction. If the neighboring block is not coded as affine prediction, the flag is skipped and the four-parameter model is used for affine prediction. In other words, the six-parameter model is considered if one or more neighboring blocks are coded as affine motion models. When it comes to the number of CPMVDs, two and three CPMVDs are signaled for the four-parameter and six-parameter affine motion models, respectively.

[0370] In addition, in one embodiment of the present disclosure, pattern-matched motion vector refinement can be used. In JEM's pattern matching motion vector derivation (named PMMVD in the JEM encoder description and referred to as PMVD in this document), the decoder needs to evaluate multiple motion vector (MV) candidates to determine the starting MV candidate for CU-level search. In the sub-CU level search, several MV candidates are added in addition to the best CU-level MV. The decoder needs to evaluate these MV candidates to find the best MV, which requires a lot of memory bandwidth. In the proposed pattern matching motion vector refinement (PMVR), the concepts of template matching and bilateral matching in PMVD in JEM are adopted. A PMVR_flag is signaled when skip mode or merge mode is selected to indicate whether PMVR is enabled. In order to significantly reduce the memory bandwidth requirement compared to PMVD, an MV candidate list is generated, and if PMVR is applied, the starting MV candidate index is explicitly signaled.

[0371] The candidate list is generated by using the merge candidate list generation process, but sub-CU merge candidates, such as affine candidates and ATMVP candidates, are excluded. For bilateral matching, only unilaterally predicted MV candidates are included. Bilaterally predicted MV candidates are split into two unilaterally predicted MV candidates. In addition, similar MV candidates (MV difference less than a predefined threshold) are also removed. For CU-level search, diamond search MV refinement is performed starting from the signaled MV candidate.

[0372] Sub-CU level search is enabled only for bilateral match merge mode. To reduce memory bandwidth, only MVs determined from CU-level search are evaluated. The search window for sub-CU level search for all sub-CUs is the same as that for CU-level search. Therefore, no additional bandwidth is required for sub-CU level search.

[0373] Template matching is also used to refine the MVP in AMVP mode. In AMVP mode, two MVPs are generated using the HEVC MVP generation process, and an MVP index is signaled to select one of them. The selected MVP is refined by using template matching in PMVR. If adaptive motion vector resolution (AMVR) is applied, the MVP is rounded to the corresponding precision before template matching refinement. This refinement process is called pattern matching motion vector predictor refinement (PMVPR). In the rest of this document, unless otherwise specified, PMVR includes template matching PMVR, bilateral matching PMVR, and PMVPR.

[0374] To reduce memory bandwidth requirements, PMVR is disabled for 4×4, 4×8, and 8×4 CUs. To further reduce memory bandwidth requirements, the search range for {template matching, bilateral matching} is reduced to {±2, ±4} for CU regions equal to 64, and the search range for {template matching, bilateral matching} is reduced to {±6, ±8} for CU regions greater than 64. Compared to the worst case in HEVC, by using all the above methods described in this PMVR section, the required memory bandwidth is reduced from 45.9x in PMVD of JEM-7.0 to 3.1x in PMVR.

[0375] Application techniques when using affine in non-QT blocks

[0376] Figure 28 A method of generating a prediction block and a motion vector in inter prediction to which the affine motion model according to an embodiment of the present disclosure has been applied is shown.

[0377] Figure 28 The relationship equation for deriving a motion vector when the affine motion model is applied is shown. The motion vector can be derived based on the following equation 17.

[0378] [Equation 17]

[0379]

[0380] In this case, v_x indicates the x component of the sample-unit motion vector of the (x, y) coordinate sample within the current block. v_y indicates the y component of the sample-unit motion vector of the (x, y) coordinate sample within the current block. That is, (v_x, v_y) becomes the sample-unit motion vector of the (x, y) coordinate sample. In this case, a, b, c, d, e, and f indicate the parameters of the relationship equation for deriving the sample-unit motion vector of the (x, y) coordinate from the control point (CP) of the current block. The CP can be represented as a control pixel. The parameters can be derived from the motion information of the CP of each PU sent in units of PUs. The relationship equation for deriving the sample-unit motion vector derived from the motion information of the CP can be applied to each sample of the block, and can be derived as the position of the sample within the reference image based on the relative position of the x-axis and y-axis of each sample. The sample-unit motion vector can be derived differently according to the size, asymmetry or symmetry, and block position of the block of the QTBT (TT) block partition structure, and its detailed implementation method is achieved by Figures 29 to 38 Shown.

[0381] Figure 29 is a diagram illustrating a method of performing motion compensation based on a motion vector of a control point according to one embodiment of the present disclosure.

[0382] Reference Figure 29 , assuming and describing the case where the current block is a 2N×2N block. For example, the motion vector of the upper left sample within the current block can be considered to be v_0. In addition, using the samples of the adjacent blocks adjacent to the current block as CPs, the motion vectors of the CPs can be considered to be v_1 and v_2. That is, assuming that the width and height of the current block are S and the coordinates at the upper left sample position of the current block are (xp, yp), the coordinates of CP0 among the CPs can be considered to be (xp, yp), the coordinates of CP1 can be considered to be (xp+S, yp), and the coordinates of CP2 can be considered to be (xp, yp+S). The motion vector of CP0 can be considered to be v_0, the motion vector of CP1 can be considered to be v_1, and the motion vector of CP2 can be considered to be v_2. The motion vector of the CP can be used to derive the sample unit motion vector. The sample unit motion vector can be derived based on the following formula 18.

[0383] [Equation 18]

[0384]

[0385] In this case, v_x and v_y indicate the x component and y component, respectively, of the motion vector for the sample at the (x, y) coordinate within the current block. v_x0 and v_y0 indicate the x component and y component, respectively, of the motion vector v_0 for CP0. v_x1 and v_y1 indicate the x component and y component, respectively, of the motion vector v_1 for CP1. v_x2 and v_y2 indicate the x component and y component, respectively, of the motion vector v_2 for CP2. The motion vector of the sample within the current block can be derived based on the relative position within the current block according to a relationship for deriving a sample-unit motion vector, such as Equation 18.

[0386] Figure 30 is a diagram illustrating a method of performing motion compensation based on motion vectors of control points in an irregular block according to one embodiment of the present disclosure.

[0387] Figure 30 The CP of a block partitioned into N×2N is shown. The same method as for the partition type 2N×2N can be used to drive the relationship for deriving the sample unit motion vector within the current block. In the process of deriving the relationship, a width value suitable for the shape of the current block can be used. In order to derive the sample unit motion vector, three CPs can be derived. The position of the CP can be as follows Figure 30 That is, assuming that the width and height of the current block are S / 2 and S and the coordinates of the current block at the upper left sample position are (xp, yp), the coordinates of CP0 in the CP may be (xp, yp), the coordinates of CP1 thereof may be (xp+S / 2, yp), and the coordinates of CP2 may be (xp, yp+S). The sample-unit motion vector can be derived based on the following equation 19.

[0388] [Equation 19]

[0389]

[0390] In this case, vx and vy indicate the x component and y component of the motion vector for the sample at the (x, y) coordinates within the current block, respectively. v_x0 and v_y0 indicate the x component and y component of the motion vector v_0 for CP0, respectively. v_x1 and v_y1 indicate the x component and y component of the motion vector v_1 for CP1, respectively. v_x2 and v_y2 indicate the x component and y component of the motion vector v_2 for CP2, respectively. Equation 3 indicates a relationship for deriving a sample-unit motion vector in which the width of the current block is considered to be S / 2. The motion vector of a sample within the current block partitioned from the CU based on the partition type N×2N can be derived based on the relative position within the current block according to a relationship such as Equation 19.

[0391] Figure 31is a diagram illustrating a method of performing motion compensation based on motion vectors of control points in an irregular block according to one embodiment of the present disclosure.

[0392] Figure 31 The block partitioned based on the partition type 2N×N is shown. In order to derive the sample unit motion vector, three CPs can be derived. Figure 31 Adjust the position of CP as shown, based on Figure 31 The shape of the current block shown adjusts the height of the current block to S / 2. That is, assuming that the width and height of the current block are S and S / 2 and the coordinates of the current block at the upper left sample position are (xp, yp), the coordinates of CP0 among the CPs may be (xp, yp), the coordinates of CP1 may be (xp+S, yp), and the coordinates of CP2 may be (xp, yp+S / 2). The sample-unit motion vector can be derived based on the following equation 20.

[0393] [Equation 20]

[0394]

[0395] In this case, v_x and v_y indicate the x component and y component of the motion vector for the sample at the (x, y) coordinate within the current block, respectively. v_x0 and v_y0 indicate the x component and y component of the motion vector v_0 for CP0, respectively. v_x1 and v_y1 indicate the x component and y component of the motion vector v_1 for CP1, respectively. v_x2 and v_y2 indicate the x component and y component of the motion vector v_2 for CP2, respectively. Equation 4 indicates a relationship for deriving a sample-unit motion vector in which the height of the current block is considered to be S / 2. The motion vector of each sample within the current block partitioned from the CU based on the partition type 2N×N can be derived based on the relative position within the current block according to a relationship for deriving a sample-unit motion vector such as Equation 4.18.

[0396] Figures 32 to 38 is a diagram illustrating a method of performing motion compensation based on a motion vector of a control point in an irregular block according to one embodiment of the present disclosure.

[0397] Figure 32 The CP of the asymmetric current block is shown. Figure 32 As shown, the width and height of the asymmetric current block can be considered as W and H. In order to derive the sample unit motion vector, three CPs of each current block can be derived. Figure 32As shown, the coordinates of the CP can be adjusted according to the width and height of the shape of the current block. That is, assuming that the width and height of the current block are W and H and the coordinates of the upper left sample position of each current block are (xp, yp), the coordinates of CP0 among the CPs can be set to (xp, yp), the coordinates of CP1 can be set to (xp+W, yp), and the coordinates of CP2 can be set to (xp, yp+H). In this case, the sample-unit motion vector within the current block can be derived based on the following formula 21.

[0398] [Equation 21]

[0399]

[0400] In this case, v_x and v_y indicate the x component and y component of the motion vector for the sample at the (x, y) coordinate within the current block, respectively. v_x0 and v_y0 indicate the x component and y component of the motion vector v_0 for CP0, respectively. v_x1 and v_y1 indicate the x component and y component of the motion vector v_1 for CP1, respectively. v_x2 and v_y2 indicate the x component and y component of the motion vector v_2 for CP2. Equation 21 indicates a relationship for deriving a sample-unit motion vector, in which the width and height of the asymmetric current block are taken into account.

[0401] In addition, according to the present disclosure, in order to reduce the amount of data of motion information for CP indicated in units of blocks, a motion information prediction candidate for at least one CP may be selected based on motion information of neighboring blocks or neighboring samples of the current block. The motion information prediction candidate may be referred to as an affine motion information candidate or an affine motion vector candidate. The affine motion information candidate may include, for example, a reference to Figures 33 to 38 Public content.

[0402] MVD encoding

[0403] Current state-of-the-art video coding standards use motion vectors and their motion vector predictors to generate a motion vector difference (MVD). MVD can be more formally defined as the difference between a motion vector and its motion vector predictor. Similar to a motion vector, an MVD has x and y components corresponding to motion in the x (horizontal) and y (vertical) directions. MVD is a property that is only available when a coding unit is encoded using the (Advanced) Motion Vector Prediction ((A)MVP) mode.

[0404] Once the MVD is determined, it is then encoded using entropy techniques. Video standards rely on the MVD as one of their possible methods to exploit redundancy in motion vectors and achieve compression. At the decoder, the motion vector difference (MVD) is decoded before the motion vector of the coding unit is decoded. By encoding the MVD instead of the actual motion vector, the redundancy between the motion vector and its prediction can be exploited, thereby improving compression efficiency.

[0405] The input to the MVD coding stage at the decoder is simply the encoded MVD binary number that has been parsed for decoding. The input to the MVD coding stage at the encoder is the actual MVD value and, in addition, a flag indicating the resolution used for MVD encoding (the "imv" flag). This flag is used to determine whether the MVD should be represented as 1 pixel, 4 pixels, or a quarter pixel.

[0406] Figure 39 An overall coding structure for deriving motion vectors according to one embodiment of the present disclosure is shown.

[0407] Reference Figure 39 , first check whether the coding unit is in merge mode (S3901).

[0408] If the coding unit is in merge mode, the affine flag and merge index are parsed for decoding (S3902).

[0409] If the coding unit is not in merge mode, it will be in AMVP mode. In AMVP mode, first parse list information (ie, whether to use list 0, list 1, or both lists) (S3903).

[0410] Then, the affine flag is parsed (S3904). After that, it is checked whether the parsed affine flag is true or false (S3905).

[0411] If true, then process parse_MVD_LT and parse_MVD_RT corresponding to the left (LT) and right (RT) MVDs (S3906). If the affine flag is false, then process the MVDs (S3907). The affine motion model in the special case of AMVP will be described in detail below.

[0412] Figure 40 An example of an MVD coding structure according to one embodiment of the present disclosure is shown.

[0413] Reference Figure 40 First, the MVD greater than zero flag for the horizontal component (MVDxGT0) and the vertical component (MVDyGT0) is parsed (S4001).

[0414] Next, a check is made to see if the parsed data for the horizontal component is greater than zero (i.e., MVDxGT0) (S4002). If the MVDxGT0 flag is true (i.e., MVDxGT0 is equal to "1"), the parsed horizontal component is greater than one (i.e., MVDxGT1) (S4002). If MVDxGT0 is not true (i.e., MVDxGT0 is equal to "0"), the MVDxGT1 data is not parsed.

[0415] Then, similar steps are performed for the vertical component (S4003 and S4004).

[0416] Thereafter, the parsed MVD data may be further processed in blocks labeled MVDx_Rem_Level and MVDy_Rem_Level to obtain a reconstructed MVD (S4005 and S4006).

[0417] Figure 41 An example of an MVD coding structure according to one embodiment of the present disclosure is shown.

[0418] Figure 41 shows how the decoder further processes Figure 40 The data in the block MVDx_Rem_Level in the decoder is used to decode the MVDx component. If the decoding flag indicating that the parsed data will be greater than zero (i.e., MVDxGT0) is true (S4101) and the decoding flag indicating that the parsed data will be greater than one (i.e., MVDxGT1) is true (S4102), then a first-order exponential Golomb (EG) code is used to decode the binary number corresponding to the parsed MVDx component (S4103). The input to EG will be a binary number containing the absolute min two (i.e., Abs-2) MVD value and a Golomb of order 1.

[0419] The sign information is then parsed by decoding the bypass binary containing this information (S4104). If the value of the decoded bypass binary is 1, a negative sign is appended to the decoded MVDx. However, if the value of the decoded bypass binary is 0, the decoded MVD is indicated as a positive value. If MVDxGR0 is true but MVDxGR1 is not true, this indicates that the absolute value of the MVDx being decoded is 1. The sign information is then parsed and updated. However, if MVDxGR0 is false, the reconstructed MVDx is 0.

[0420] In the following Figure 42A similar process for decoding MVDy (ie, MVDy_Rem_Level) at the decoder is shown in .

[0421] Figure 42 An example of an MVD coding structure according to one embodiment of the present disclosure is shown.

[0422] Reference Figure 42 , if the decoding flag indicating that the parsed MVDy is greater than zero (ie, MVDyGT0) is true (S4201), the flag MVDyGR1 is checked (S4202).

[0423] If both MVDyGR0 and MVDyGR1 are true, the parsed MVD data is decoded using an EG code whose input is a binary number containing the absolute value minus 2 (Abs-2) MVD and order 1 (S4203). Subsequently, the sign information is parsed and decoded to obtain decoded MVDy (S4204). If MVDyGR0 is true but MVDyGR1 is false, the absolute vertical value is considered to be +1 or -1. The sign information is then parsed and decoded in a manner similar to that described above to obtain decoded MVDy. If the MVDyGR0 flag is false, MVDy is zero.

[0424] Figure 43 An example of an MVD coding structure according to one embodiment of the present disclosure is shown.

[0425] Reference Figure 43 , at the encoder, the signed MVD value will be encoded. Figure 41 Similarly, by checking the absolute values ​​of the horizontal and vertical components, a binary number greater than zero (i.e., MVDxGR0 and MVDyGR0) is encoded for the x component and the y component (S4301, S4311). Then, a greater than one flag (i.e., MVDxGR1 and MVDyGR1) is encoded for the horizontal component and the vertical component (S4302, S4312). Thereafter, similar to the decoder, the absolute MVD value is encoded, and then the horizontal component and the vertical component are encoded in sequence.

[0426] For horizontal MVD encoding, if the absolute horizontal MVD component is greater than zero (i.e., MVDxGR0) and greater than 1 (i.e., MVDxGR1), then (absolute value -2) is encoded using a first-order EG code (S4303). Thereafter, the sign information is encoded using a bypass binary (S4304). If MVDxGR0 is true and MVDxGR1 is not true, only the sign information is encoded. If MVDxGR0 is not true, MVDx is zero. The same process is repeated to encode MVDy (S4313, S4314).

[0427] Affine encoding

[0428] Previous video coding standards only considered translational motion models. However, potential motion can include effects such as scaling, rotation, panning, and other irregular motions. To capture the characteristics of such motion, the latest video coding standards have introduced affine motion coding, which uses a 4-parameter or 6-parameter affine motion model to capture the irregular nature of motion information.

[0429] If a 4-parameter model is used, 2 control points are generated, and if a 6-parameter model is used, 3 control points are used. Figure 16 The concept of affine motion is shown more clearly. Using a 4-parameter model, the current block is encoded using two control point motion vectors given by v_0 (cpmv_0) and v1 (cpmv_1).

[0430] Once these control points are derived, the MVF for each of the 4×4 sub-blocks is described by Equation 22 below.

[0431] [Equation 22]

[0432]

[0433] Where (v_0x, v_0y) is the motion vector of the upper left control point, and (v_1x, v_1y) is the motion vector of the upper right control point. Figure 27 As shown, the motion vector of each 4×4 sub-block is calculated by deriving the motion vector of the center sample of each sub-block.

[0434] Implementation 1: MVD Precision

[0435] Affine coding can be used in both merge mode and (A)MVP mode. As described above, in AMVP mode, affine coding can use two control points or three control points depending on the motion model used. Therefore, there can be two or three motion vector differences (MVDs). In other words, if two control points are used according to the motion model, the MVD of at least one of the top left (LT) and / or top right (RT) control points can be encoded. If three control points are used, the MVD of at least one of the top left (LT), top right (RT) and / or bottom right (LB) control points can be encoded.

[0436] In the decoder, the MVD is decoded before the motion vector in the coding unit is finally determined. In this case, the accuracy of the affine prediction (or affine motion prediction) may depend on the accuracy of the control point motion vector. As a result, the accuracy of the affine prediction may depend on the accuracy of the MVD encoding.

[0437] However, in conventional image compression techniques, if affine prediction is applied, the MVD is encoded only with 1 / 4 pixel (or fractional) precision (or accuracy, resolution).

[0438] In other words, the efficiency of affine coding can be greatly dependent on the high precision of the control point motion vectors and, subsequently, the high precision of the motion vectors for the center samples of each sub-block. Furthermore, the aforementioned relationships for deriving motion vectors (e.g., 1, 11, 12, 16, and 22) can provide precision far greater than 1 / 16 pixel. For example, if 1 / 16 pixel precision is used, the values ​​calculated in the aforementioned relationships can be rounded to 1 / 16 pixel precision. This is useful because a motion-compensated interpolation filter operating at 1 / 16 pixel precision can be applied to easily generate prediction samples for each sub-block using the derived motion vectors.

[0439] After motion compensation, the high-precision motion vector of each sub-block can be rounded and stored with the same precision as a normal motion vector. The MVD is calculated based on the difference between the predicted value and the actual motion vector, and the initial calculation can be maintained with 1 / 16 pixel accuracy. However, in conventional image compression technology, if affine prediction is applied, the MVD accuracy is reduced to 1 / 4 pixel and encoded. In order to decode the motion vector more accurately, if high precision is maintained despite the application of affine prediction, the accuracy of affine prediction can be increased, and compression efficiency can be improved.

[0440] In conventional compression techniques, ordinary MVDs (i.e., MVDs not derived from affine prediction) are processed (encoded or transmitted) with 1 / 4 pixel, 1 pixel (i.e., integer pixel), or 4 pixel precision. Furthermore, the encoder / decoder uses a precision flag (or syntax element) to control this precision. However, as described above, in affine prediction, the MVD is stored only with 1 / 4 pixel precision. Therefore, the present disclosure proposes a method for increasing the precision of the MVD in order to improve the accuracy of affine prediction.

[0441] In the present disclosure, for convenience of description, if affine prediction is applied, MVD may be expressed as affine MVD.

[0442] Figure 44 is a diagram illustrating a method of deriving affine motion vector difference information according to one embodiment to which the present disclosure is applied.

[0443] Reference Figure 44 For the sake of convenience, the decoder is mainly described, but the present disclosure is not limited thereto. The method of signaling affine motion vector difference information can be applied to the encoder in substantially the same manner. Figure 44In the present invention, it is assumed and described that two control points at the upper left and upper right positions are used for affine prediction, but the present disclosure is not limited to this and can be basically applied in the same way to the case where three control points at the lower left, upper left and upper right positions are used for affine prediction.

[0444] The decoder checks whether merge mode is applied to the current block (S4401). If merge mode is applied to the current block, the decoder parses an affine flag indicating whether affine prediction is applied to the current block and / or a merge index indicating a candidate in a merge candidate list that is applied to the current block (S4402).

[0445] The decoder parses a reference list index (or prediction list index) indicating a reference direction (or prediction direction, reference list) of the current block (S4403).

[0446] The decoder parses an affine flag indicating whether affine prediction is applied to the current block (S4404).

[0447] The decoder checks whether affine prediction is applied to the current block based on the affine flag value (S4405).

[0448] If affine prediction is not applied to the current block, the decoder parses the MVD of the current block (S4406).

[0449] In one embodiment of the present disclosure, if affine prediction is applied to the current block, the decoder may parse the precision flag (or precision index) and perform a checking process on the precision.

[0450] Specifically, if affine prediction is applied to the current block, the decoder parses the MVD precision flag (S4407). In this case, the MVD precision flag (or affine MVD precision flag) indicates whether the adaptive affine MVD precision mode is applied. In one embodiment, if the adaptive affine MVD precision mode is applied, the affine MVD may be derived with another precision other than the predefined default (or basic) precision. If the adaptive affine MVD precision mode is applied, the affine MVD may be derived with the predefined default precision. In one embodiment, the predetermined default precision may be 1 / 4 pixel precision. The other precision other than the predefined default precision may include at least one of integer pixel, 4 pixel, 1 / 8 pixel, and / or 1 / 16 pixel precision.

[0451] The decoder checks whether the adaptive affine MVD precision mode is applied based on the MVD precision flag value (S4408). If the adaptive affine MVD precision mode is applied, the decoder derives the MVD for the two control points with another precision other than the default precision (S4409). In one embodiment, if the adaptive affine MVD precision mode is applied, that is, if another precision other than the default precision is applied, the encoder may send a syntax element indicating a specific precision among the preset precisions to the decoder.

[0452] If the adaptive affine MVD precision mode is not applied, the decoder derives the MVD for the two control points with default precision ( S4410 ).

[0453] In one embodiment, the precision of the affine MVD can be signaled through the bitstream. To this end, the encoder can signal a higher-level syntax element to the decoder. For example, the higher-level syntax element can be signaled through a sequence parameter set, a picture parameter set, a slice header (or a tile group header), etc. In addition, for example, the encoder can generate set_affine_MVD_precision_flag and can signal set_affine_MVD_precision_flag to the decoder. In this case, set_affine_MVD_precision_flag represents a higher-level syntax element indicating the precision of the affine MVD.

[0454] For example, set_affine_MVD_precision_flag may indicate whether the precision of the affine MVD is a predefined default (or basic) precision (e.g., 1 / 4 pixel precision). If the predefined default precision is not applied, set_affine_MVD_precision_flag may include other items of precision information. Additional precision information may be signaled from the encoder to the decoder. That is, the encoder may send a syntax element indicating whether the precision of the affine MVD is a predefined default precision (e.g., 1 / 4 pixel precision) to the decoder. If the precision of the affine MVD is not the predefined default precision, the encoder may send a syntax element indicating the detailed precision of the affine MVD to the decoder. For example, the detailed precision of the affine MVD may include at least one of integer pixel, 4 pixel, 1 / 8 pixel, or 1 / 16 pixel precision.

[0455] Alternatively, for example, a syntax element may indicate whether to send the affine MVD with higher precision.

[0456] In one embodiment, the location of the syntax header may be summarized as high_level_parameter_set() according to Table 2. Furthermore, in one embodiment, an auxiliary syntax element may be used as a syntax element (index or flag) for indicating detailed precision.

[0457] [Table 2]

[0458]

[0459] In Table 2, when set_affine_MVD_precision_flag is 1, this may indicate that set_affine_MVD_precision_flag exists in the slice header of the non-IDR picture of the coded video sequence (CVS). In addition, when set_affine_MVD_precision_flag is 0, this may indicate that set_affine_precision_flag does not exist in the slice header and the adaptive affine MVD according to the present embodiment is not used in the CVS.

[0460] In addition, in one embodiment, a syntax element for indicating detailed precision information may be additionally signaled. For example, a syntax structure according to Table 3 may be defined.

[0461] [Table 3]

[0462]

[0463] In Table 3, slice_affine_mvd_precision_idx represents a syntax element indicating the specific (detailed) precision of the affine MVD. In the present disclosure, the name of slice_affine_mvd_precision_idx is not limited, and the syntax element for indicating the specific precision of the affine MVD can be represented as a flag. In addition, in Table 3, it is assumed that the syntax element indicating the specific precision of the affine MVD is included in the slice segment header, but the present disclosure is not limited thereto. Syntax having various levels can be included in the slice segment header. For example, the syntax element indicating the specific (detailed) precision of the affine MVD can be included in the coding tree unit syntax or the coding unit syntax.

[0464] In one embodiment, when slice_affine_mvd_precision_idx is 0, this may indicate a default MVD precision of 1 / 4 pixel. Similarly, an index value of 1 may indicate an MVD precision of 1 / 8 pixel, and an index value of 2 may indicate an MVD precision of 1 / 16 pixel.

[0465] Implementation 2: Entropy and GOLOMB Parameters

[0466] One embodiment of the present disclosure proposes a method using a feature in which the MVD statistic is changed. Specifically, the MVD statistic of an inter-frame coded block depends on a motion model for translational motion only. However, the statistics of the affine MVD are different from those of a normal AMVP block because the prediction unit (or coding block or coding unit) coded in affine mode uses an affine motion model that takes into account various other motions. This means that the same entropy coding method and / or parameters should not be used to encode the MVD of all blocks in general.

[0467] As mentioned above Figures 41 to 43 As described above, in conventional image compression technology, when the absolute values ​​of the MVD in the horizontal and vertical directions are greater than 1, decoding is performed using an exponential Golomb code with an order of 1. The exponential Golomb code can be very efficient when expressing the number or a group of similar patterns without limiting the maximum number that can be expressed.

[0468] The degree of the exponential Golomb code (hereinafter referred to as the Golomb degree) contains the probability that a symbol may appear. In conventional image compression technology, regardless of the distribution of MVD values, a degree of 1 is used. However, in the case of affine motion, it is not necessary to maintain the same method. Therefore, the present disclosure proposes an exponential Golomb code having a degree that depends on the range of the affine MVD value. In one embodiment, the encoder / decoder can use the same Figure 45 The MVD range can be selected using the same method as described above, but the present disclosure is not limited thereto. Histogram analysis can be very useful in determining the range of MVD absolute values. The most frequent values ​​can be grouped, and each sub-region (or range) of the MVD can be encoded using another Golomb series.

[0469] This can be very efficient because similar MVD values ​​can be encoded using the same series. Specifically, in affine motion, the control points on the left and right can have a very close relationship. The encoder / decoder can use the statistics of any one control point to determine the most likely region (or range) of the other control point, and can select various Golomb series based on the most likely region (or range).

[0470] One embodiment of the present disclosure proposes an entropy coding method that relies on unique statistics of a motion model when performing entropy coding for MVD, rather than a constant entropy coding method, as described with reference to the accompanying drawings.

[0471] Figure 45 2 is a diagram illustrating a coding structure of a motion vector difference according to an embodiment to which the present disclosure is applied.

[0472] Reference Figure 45 For convenience of description, the decoder is mainly described, but the present disclosure is not limited thereto. The method of signaling affine motion vector difference information can be applied to the encoder in substantially the same manner.

[0473] In one embodiment of the present disclosure, when the MVD value is greater than 0, the decoder can divide the MVD greater than 0 without restriction, such as conventional MVDxGR1 and MVDyGR1, based on a given integer N value. In addition, N can be determined based on the distribution of MVD values.

[0474] Specifically, the decoder checks syntax elements (flags) MVDxGR_0 and MVDyGR_0 indicating whether the MVD value is greater than 0 (S4501, S4511). When the MVDxGR_0 and / or MVDyGR_0 values ​​are 0, the MVD value in each direction (horizontally or vertically) is considered to be 0.

[0475] When the values ​​of MVDxGR_0 and MVDyGR_0 are 1, the decoder checks the MVDxGR_N and MVDyGR_N syntax elements (flags) (S4502, S4512). When the values ​​of MVDxGR_N and / or MVDyGR_N are 1, the decoder decodes (or parses) the MVD value in each direction using the Golomb level k1 (i.e., level 1) based on the exponential Golomb code with an input of absolute value -N-1 (Abs-N-1) (S4503, S4513).

[0476] When the MVDxGR_N and / or MVDyGR_N values ​​are 0, the decoder decodes (or parses) the MVD values ​​in each direction using an exponential Golomb code based on another Golomb level other than level k1 (S4504, S4514). In one embodiment, exponential Golomb binarization of Golomb level k2 (i.e., level 2) can be used to encode / decode corresponding absolute values ​​greater than 0 and less than or equal to N.

[0477] The decoder decodes (or parses) the symbols of the MVD in each direction ( S4505 , S4515 ).

[0478] Furthermore, in one embodiment, the encoder / decoder may apply different binarizations to each of the parts divided into 0 and N. For example, the encoder / decoder may use an exponential Golomb code to encode absolute values ​​greater than 0 and less than N, and may use truncated binary (TB) (or truncated unary binarization) to encode absolute values ​​greater than N.

[0479] Implementation 3: MVD Precision Control with Entropy and GOLOMB Parameters

[0480] One embodiment of the present disclosure proposes a method that combines two embodiments (embodiments 1 and 2). In other words, one embodiment of the present disclosure may include features that combine the two aforementioned embodiments. Specifically, one embodiment of the present disclosure proposes a method that integrates precision information for MVD and entropy coding.

[0481] Figure 46 is a diagram illustrating a method of deriving an affine motion vector based on accuracy information according to one embodiment of the present disclosure.

[0482] Reference Figure 46 For convenience of description, the decoder is mainly described, but the present disclosure is not limited thereto. The method of signaling affine motion vector difference information can be applied to the encoder in substantially the same manner.

[0483] If the precision control function has been activated, the decoder parses the syntax element indicating a specific precision (S4601). Figure 46 In , syntax elements are represented as precision indices, but are not limited to such names.

[0484] The decoder parses the MVD value in the horizontal / vertical direction based on the accuracy checked in step S4601 (S4602).

[0485] In one embodiment, the precision index may indicate high precision such as 1 / 16 pixel or 1 / 8 pixel, and may indicate low precision such as integer pixel or 4 pixel. For example, if the syntax element indicating whether to apply the adaptive affine precision mode (e.g., set_affine_MVD_precision_flag) is true, the decoder may additionally check the syntax element indicating a specific precision (e.g., slice_affine_mvd_precision_idx). The decoder may determine the precision of the MVD encoded based on the syntax element indicating the specific precision. In addition, the decoder may parse the MVD information in the horizontal / vertical direction based on the determined precision.

[0486] In one embodiment, when decoding the MVD based on the determined precision, the method described in embodiment 2 may be applied. If high precision is applied, the decoder may parse MVDx_GR_N and / or MVDy_GR_N when the MVD value in the horizontal and / or vertical direction is greater than 0. As described above, the decoder may apply a first binarization when the absolute value is greater than N, and may apply a second binarization (or binarization method) when the absolute value is less than or equal to N. For example, the decoder may use an exponential Golomb code with a level of 1 as the first binarization, and may use truncated binary (TB) (or truncated unary binarization) as the second binarization. If low precision is applied (e.g., 1 / 4, 1, or 4 pixel precision), the decoder may use a third binarization to perform MVD decoding. For example, the decoder may use truncated unary binarization as the third binarization.

[0487] Figure 47 2 is a diagram illustrating a coding structure of a motion vector difference according to an embodiment to which the present disclosure is applied.

[0488] Reference Figure 47 For convenience of description, the decoder is mainly described, but the present disclosure is not limited thereto. The method of signaling affine motion vector difference information can be applied to the encoder in substantially the same manner.

[0489] In one embodiment of the present disclosure, when the MVD value is greater than 0, the decoder can divide the MVD value greater than 0 based on a given integer N value without being restricted as in conventional MVDxGR1 and MVDyGR1. In addition, N can be determined based on the distribution of MVD values.

[0490] Specifically, the decoder checks syntax elements (flags) MVDxGR_0 and MVDyGR_0 indicating whether the MVD value is greater than 0 (S4701, S4711). When the MVDxGR_0 and / or MVDyGR_0 values ​​are 0, the MVD value in each direction (horizontally and / or vertically) is considered to be 0.

[0491] When the values ​​of MVDxGR_0 and MVDyGR_0 are 1, the decoder checks whether the MVD precision of the current block is higher than a predefined precision ( S4702 , S4711 ). For example, the predefined precision may be 1 pixel, 1 / 4 pixel, or 1 / 8 pixel precision.

[0492] When the current MVD precision is higher than the predefined precision, the decoder checks the MVDxGR_N and MVDyGR_N syntax elements (flags) (S4703, S4713).

[0493] When the MVDxGR_N and / or MVDyGR_N values ​​are 1, the decoder decodes (or parses) the MVD value in each direction using a first binarization (or binarization method) (S4704, S4714). For example, the first binarization may be an exponential Golomb code method with a Golomb level of k1 (i.e., level 1). That is, the decoder may decode (or parse) the MVD value in each direction using a Golomb level of k1 based on an exponential Golomb code with an absolute value of -N (Abs-N) as input.

[0494] When the MVDxGR_N and / or MVDyGR_N values ​​are 0, the decoder decodes (or parses) the MVD value in each direction using the second binarization (S4705, S4715). For example, the second binarization may be an exponential Golomb code using another level other than the Golomb level k1, and may be truncated binary (TB) (or truncated unary binarization).

[0495] When the current MVD precision is less than or equal to the predefined precision, the decoder decodes (or parses) the MVD value in each direction using a third binarization (S4706, S4716). For example, the third binarization may be an exponential Golomb code using another Golomb series other than k1, and may be truncated binary (TB) (or truncated unary binarization).

[0496] The decoder decodes (or parses) the symbols of the MVD in each direction ( S4707 , S4717 ).

[0497] For the convenience of description, the aforementioned embodiments of the present disclosure have been divided and described, but the present disclosure is not limited thereto. That is, the described embodiments 1 to 3 can be performed independently, and one or more embodiments can be combined and performed.

[0498] Figure 48 is a flowchart illustrating a method of generating an inter-frame prediction block based on affine prediction according to one embodiment to which the present disclosure is applied.

[0499] Reference Figure 48 For the convenience of description, the decoder is mainly described, but the present disclosure is not limited thereto. The method of generating an inter-frame prediction block according to one embodiment of the present disclosure can be performed identically in an encoder and a decoder.

[0500] The decoder checks whether affine prediction (or affine motion prediction) is applied to the current block (S4801).

[0501] If, as a result of the check, affine prediction is applied, the decoder obtains at least one syntax element indicating the resolution (or accuracy or precision) of the motion vector difference used for affine prediction (S4802).

[0502] The decoder derives a control point motion vector of the current block based on the at least one syntax element (S4803).

[0503] The decoder derives a motion vector for each of a plurality of subblocks included in the current block based on the control point motion vector (S4804).

[0504] The decoder generates a prediction sample of the current block using the motion vector of each sub-block (S4805).

[0505] As described above, step S4802 may include the step of obtaining a first syntax element indicating whether the resolution of the motion vector difference is a preset default resolution, and if the resolution of the motion vector difference is not the default resolution, obtaining a second syntax element indicating the resolution of the motion vector difference among the remaining resolutions other than the default resolution.

[0506] Furthermore, as described above, the default resolution may be pre-set to 1 / 4 pixel accuracy.

[0507] Furthermore, as described above, each item of residual resolution may include at least one of integer pixel precision, 4-pixel precision, 1 / 8 pixel precision, or 1 / 16 pixel precision.

[0508] In addition, as described above, step S4804 may further include the following steps: a step of determining the resolution of the motion vector difference using at least one syntax element; and a step of obtaining the motion vector difference based on the resolution of the motion vector difference.

[0509] In addition, as described above, the step of obtaining the motion vector difference may further include the step of obtaining a flag indicating whether the motion vector difference is greater than 0, and when the motion vector difference is greater than 0, the step of obtaining a flag indicating whether the motion vector difference is greater than a predefined specific value.

[0510] Furthermore, as described above, when the motion vector difference is greater than 0 and less than or equal to a predefined specific value, the motion vector difference may be binarized using an exponential Golomb code with an order of 1. When the motion vector difference is greater than the predefined specific value, the motion vector difference may be binarized using a truncated binarization method.

[0511] Figure 49 is a diagram illustrating an inter-frame prediction apparatus based on affine prediction according to one embodiment to which the present disclosure is applied.

[0512] exist Figure 49 , for convenience of description, the inter-frame prediction unit is shown as one block, but the inter-frame prediction unit may be implemented as a structure included in an encoder and / or a decoder.

[0513] Reference Figure 49 , inter-frame prediction unit implementation Figures 8 to 48 Specifically, the inter prediction unit may be configured to include an affine prediction mode identification unit 4901, a syntax element acquisition unit 4902, a control point motion vector derivation unit 4903, a sub-block motion vector derivation unit 4904, and a prediction sample generation unit 4905.

[0514] The affine prediction mode identification unit 4901 checks whether affine prediction is applied to the current block.

[0515] If, as a result of the check, affine prediction is applied, the syntax element acquisition unit 4902 obtains at least one syntax element indicating the resolution of the motion vector difference used for affine prediction.

[0516] The control point motion vector derivation unit 4903 derives a control point motion vector of the current block based on the at least one syntax element.

[0517] The sub-block motion vector derivation unit 4904 derivates a motion vector of each of a plurality of sub-blocks included in the current block based on the control point motion vector.

[0518] The prediction sample generation unit 4905 generates a prediction sample of the current block using the motion vector of each sub-block.

[0519] As described above, the syntax element acquisition unit 4902 can obtain a first syntax element indicating whether the resolution of the motion vector difference is a preset default resolution, and if the resolution of the motion vector difference is not the default resolution, it can obtain a second syntax element indicating the resolution of the motion vector difference in each remaining resolution other than the default resolution.

[0520] Furthermore, as described above, the default resolution is preset to 1 / 4 pixel accuracy.

[0521] Furthermore, as described above, each item of residual resolution may include at least one of integer pixel precision, 4-pixel precision, 1 / 8 pixel precision, or 1 / 16 pixel precision.

[0522] In addition, as described above, the control point motion vector derivation unit 4903 may determine the resolution of the motion vector difference using the at least one syntax element, and may obtain the motion vector difference based on the resolution of the motion vector difference.

[0523] In addition, as described above, the control point motion vector derivation unit 4903 can obtain a flag indicating whether the motion vector difference is greater than 0, and when the motion vector difference is greater than 0, it can obtain a flag indicating whether the motion vector difference is greater than a predefined specific value.

[0524] In addition, as described above, when the motion vector difference is greater than 0 and less than or equal to a predefined specific value, the motion vector difference may be binarized using an exponential Golomb code with an order of 1. When the motion vector difference is greater than the predefined specific value, the motion vector difference may be binarized using a truncated binarization method.

[0525] Figure 50 A video encoding system to which the present disclosure is applied is shown.

[0526] The video coding system may include a source device and a sink device. The source device may forward the coded video / image information or data to the sink device in a file or stream format via a digital storage medium or a network.

[0527] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a standalone device or an external component.

[0528] The video source can obtain the video / image through a process such as capture, synthesis, or generation. The video source may include a video / image capture device and / or a video / image generation device. For example, the video / image capture device may include one or more cameras and a video / image archive including previously captured videos / images. The video / image generation device may include, for example, a computer, a tablet computer, and a smartphone, and may, for example, (electrically) generate the video / image. For example, a virtual video / image may be generated by a computer, and in this case, the video / image capture process may be replaced by a process for generating relevant data.

[0529] The encoding device can encode input video / images and perform a series of processes including prediction, transformation, and quantization for compression and encoding efficiency.

[0530] The transmitter can forward the encoded video / image information or data output in a bitstream format to a receiver of a receiving device in a file or stream format via a digital storage medium or network. Digital storage media can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter can include components for generating a media file in a predetermined file format and can also include components for transmitting via a broadcast / communication network. The receiver can extract the bitstream and forward it to a decoding device.

[0531] The decoding device may perform a series of processes including dequantization, inverse transformation, prediction, and the like corresponding to the operations of the encoding device and decode the video / image.

[0532] The renderer may render the decoded video / image, and the rendered video / image may be displayed by a display unit.

[0533] Figure 51 It is a configuration diagram of a content streaming system as one embodiment to which the present disclosure is applied.

[0534] Reference Figure 51 The content streaming transmission system to which the present disclosure is applied may include an encoding server, a streaming server, a Web server, a media storage device, a user device, and a multimedia input device.

[0535] The encoding server is used to compress content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream and send the bitstream to the streaming server. As another example, when multimedia input devices such as smartphones, cameras, and camcorders directly generate the bitstream, the encoding server can be omitted.

[0536] A bitstream may be generated by applying the encoding method or the bitstream generating method of the present disclosure, and a streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0537] The streaming server transmits multimedia data to user devices based on user requests via a web server, and the web server serves as a medium for notifying users of services. When a user sends a request for a desired service to the web server, the web server passes the request to the streaming server, and the streaming server transmits the multimedia data to the user. The content streaming system may include an additional control server, which in this case controls commands and responses between devices in the content streaming system.

[0538] The streaming server can receive content from a media storage device and / or an encoding server. For example, when receiving content from an encoding server, the streaming server can receive the content in real time. In this case, the streaming server can store the bitstream for a predetermined time in order to provide a smooth streaming service.

[0539] Examples of user devices may include cellular phones, smart phones, laptop computers, digital broadcast terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation devices, tablet computers, tablet computers, ultrabooks, wearable devices (e.g., smart watches, smart glasses, and HMDs (head-mounted displays)), digital televisions, desktop computers, and digital signage, etc.

[0540] Each server in the content streaming system may operate as a distributed server, and in this case, data received by each server may be processed in a distributed manner.

[0541] The embodiments described in this disclosure may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in the drawings may be implemented and executed on a computer, processor, microprocessor, controller, or chip.

[0542] In addition, the decoder and encoder to which the present disclosure is applied can be included in multimedia broadcast transmission and reception devices, mobile communication terminals, home theater video devices, digital movie video devices, cameras for monitoring, video conversation devices, real-time communication devices such as video communication, mobile streaming devices, storage media, portable cameras, video on demand (VoD) service providing devices, Internet (OTT) video devices, Internet streaming service providing devices, three-dimensional (3D) video devices, video phone devices, and medical video devices, and can be used to process video signals or data signals. For example, OTT video devices may include game consoles, Blu-ray players, Internet access TVs, home theater systems, smartphones, tablet computers, and digital video recorders (DVRs).

[0543] In addition, the processing method to which the present disclosure is applied can be generated in the form of a program executed by a computer and can be stored in a computer-readable recording medium. Multimedia data having a data structure according to the present disclosure can also be stored in a computer-readable recording medium. Computer-readable recording media include all types of storage devices in which computer-readable data is stored. For example, computer-readable recording media may include Blu-ray discs (BDs), universal serial buses (USBs), ROMs, PROMs, EPROMs, EEPROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. In addition, computer-readable recording media include media implemented in the form of carriers (e.g., transmission over the Internet). In addition, the bit stream generated using the encoding method can be stored in a computer-readable recording medium or can be transmitted over wired and wireless communication networks.

[0544] In addition, one embodiment of the present disclosure may be implemented as a computer program product using program code. The program code according to one embodiment of the present disclosure may be executed by a computer. The program code may be stored on a computer-readable carrier.

[0545] In the aforementioned embodiments, the elements and features of the present disclosure have been combined in a specific form. Unless otherwise clearly described, each element or feature may be considered optional. Each element or feature may be implemented in a form not combined with other elements or features. In addition, some elements and / or features may be combined to form an embodiment of the present disclosure. The order of operations described in the embodiments of the present disclosure may be changed. Some elements or features of an embodiment may be included in another embodiment, or may be replaced by corresponding elements or features of another embodiment. Obviously, an embodiment may be constructed by combining claims that do not have a clear reference relationship in the claims, or may be included as a new claim by amendment after submitting the application.

[0546] According to the embodiments of the present disclosure, various methods such as hardware, firmware, software or a combination thereof may be used to implement the embodiments of the present disclosure. In the case of hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers and microprocessors may be used to implement the embodiments of the present disclosure.

[0547] In the case of implementation by firmware or software, the embodiments of the present disclosure may be implemented in the form of modules, processes, or functions for performing the above-mentioned functions or operations. The software code may be stored in a memory and driven by a processor. The memory may be located inside or outside the processor and may exchange data with the processor in various known ways.

[0548] It is obvious to those skilled in the art that the present disclosure may be implemented in other specific forms without departing from the essential features of the present disclosure. Therefore, the detailed description should not be interpreted as limiting, but rather as illustrative in all aspects. The scope of the present disclosure should be determined by a reasonable analysis of the appended claims, and all changes within the equivalent range of the present disclosure are included within the scope of the present disclosure.

[0549] [Industrial Applicability]

[0550] The above-described preferred embodiments of the present disclosure have been disclosed for illustrative purposes, and those skilled in the art may improve, change, replace or add various other embodiments without departing from the technical spirit and scope disclosed in the appended claims.

Claims

1. A method for decoding a video signal using affine prediction, the method comprising the following steps: Obtaining a first syntax element indicating whether the affine prediction is applied to the current block; obtaining, based on the first syntax element indicating that the affine prediction is applied to the current block, a second syntax element including information on a resolution of a motion vector difference of a control point motion vector used in the affine prediction; obtaining, based on the second syntax element, a third syntax element of the resolution including the motion vector difference; deriving a control point motion vector of the current block based on at least one of the second syntax element and the third syntax element; deriving a motion vector for each of a plurality of sub-blocks included in the current block based on the control point motion vector; as well as generating a prediction sample of the current block based on the motion vector of each of the sub-blocks, wherein the second syntax element indicates whether the resolution of the motion vector difference of the control point motion vector used in the affine prediction is a predefined resolution, wherein the third syntax element indicates at least one resolution among a plurality of different resolutions of the motion vector difference, and Here, at least one of the resolutions of the motion vector differences of the control point motion vectors used in the affine prediction includes 1 / 16 pixel accuracy.

2. A method for encoding a video signal using affine prediction, the method comprising the following steps: determining whether the affine prediction is applied to the current block; determining, based on a determination that the affine prediction is applied to the current block, whether a resolution of a motion vector difference of a control point motion vector used in the affine prediction is a predefined resolution; determining at least one resolution among a plurality of different resolutions of the motion vector difference based on information indicating whether the resolution of the motion vector difference is the predefined resolution; deriving the control point motion vector of the current block based on at least one of information indicating whether the resolution of the motion vector difference is the predefined resolution and information about the at least one resolution; deriving a motion vector for each of a plurality of sub-blocks included in the current block based on the control point motion vector; generating a prediction sample of the current block based on the motion vector of each of the sub-blocks; generating a first syntax element indicating whether the affine prediction is applied to the current block; generating, based on the first syntax element indicating that the affine prediction is applied to the current block, a second syntax element including information on a resolution of a motion vector difference of a control point motion vector used in the affine prediction; as well as generating, based on the second syntax element, a third syntax element indicating at least one of the resolutions of the motion vector difference, wherein the second syntax element indicates whether the resolution of the motion vector difference of the control point motion vector used in the affine prediction is a predefined resolution, wherein the third syntax element indicates at least one resolution among a plurality of different resolutions of the motion vector difference, and Here, at least one of the resolutions of the motion vector differences of the control point motion vectors used in the affine prediction includes 1 / 16 pixel accuracy.

3. A non-transitory computer-readable storage medium storing encoded picture data, wherein the encoded picture data is generated by performing the following steps: determining whether affine prediction is applied to the current block; determining, based on a determination that the affine prediction is applied to the current block, whether a resolution of a motion vector difference of a control point motion vector used in the affine prediction is a predefined resolution; determining at least one resolution among a plurality of different resolutions of the motion vector difference based on information indicating whether the resolution of the motion vector difference is the predefined resolution; deriving the control point motion vector of the current block based on at least one of information indicating whether the resolution of the motion vector difference is the predefined resolution and information about the at least one resolution; deriving a motion vector for each of a plurality of sub-blocks included in the current block based on the control point motion vector; generating a prediction sample of the current block based on the motion vector of each of the sub-blocks; generating a first syntax element indicating whether the affine prediction is applied to the current block; generating, based on the first syntax element indicating that the affine prediction is applied to the current block, a second syntax element including information on a resolution of a motion vector difference of a control point motion vector used in the affine prediction; as well as generating, based on the second syntax element, a third syntax element indicating at least one of the resolutions of the motion vector difference, wherein the second syntax element indicates whether the resolution of the motion vector difference of the control point motion vector used in the affine prediction is a predefined resolution, wherein the third syntax element indicates at least one resolution among a plurality of different resolutions of the motion vector difference, and Here, at least one of the resolutions of the motion vector differences of the control point motion vectors used in the affine prediction includes 1 / 16 pixel accuracy.

Images