Method and apparatus for processing video signal based on inter prediction

Abstract

To provide a method and apparatus for processing video signals to improve the complexity of redundancy checks and increase efficiency.SOLUTION: A method includes constructing a first merge candidate list of a current block using a spatial merge candidate and temporal merge candidate of the current block, constructing a second merge candidate list by adding a history-based merge candidate indicating motion information of a block coded prior to the current block to the first merge candidate list, obtaining a merge index indicating a merge candidate applied to the current block within the second merge candidate list, and generating a prediction block of the current block using motion information of a merge candidate indicated by the merge index. The history-based merge candidate may be added to the first merge candidate list if the history-based merge candidate has motion information not overlapping motion information of a pre-defined merge candidate among merge candidates included in the first merge candidate list.SELECTED DRAWING: None

Description

[Technical field]

[0001] Embodiments of the present specification relate to methods and apparatus for processing a video signal using inter prediction, and in particular, to methods and apparatus for performing inter prediction using history-based motion vector prediction. [Background technology]

[0002] Compression coding refers to a series of signal processing techniques for transmitting digitalized information over communication lines or storing it in a form suitable for storage media. Media such as video, images, and audio can be subject to compression coding, and the technology that performs compression coding on video in particular is called video compression.

[0003] Next-generation video content will have characteristics called high spatial resolution, high frame rate, and high dimensionality of scene representation. Processing such content will bring about a significant increase in memory storage, memory access rate, and processing power.

[0004] Therefore, there is a need to design coding tools to more efficiently process the next generation of video content. Summary of the Invention [Problem to be solved by the invention]

[0005] The purpose of the embodiments herein is to propose a restriction of the redundancy check for adding HMVP candidates to the merge list (or AMVP list).

[0006] The technical problems to be solved in the examples of this specification are not limited to the technical problems mentioned above, and further different technical problems not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention belongs from the description below. [Means for solving the problem]

[0007] In one aspect of an embodiment of the present specification, in a method for processing a video signal based on inter prediction, the method includes the steps of: constructing a first merge candidate list for a current block using spatial merge candidates and temporal merge candidates for the current block; constructing a second merge candidate list by adding a history-based merge candidate indicating motion information of a block previously coded for the current block to the first merge candidate list; obtaining a merge index indicating a merge candidate to be applied to the current block within the second merge candidate list; and generating a predictive block for the current block using motion information of the merge candidate indicated by the merge index, wherein the history-based merge candidate may be added to the first merge candidate list if it has motion information that does not overlap with a predefined merge candidate among the merge candidates included in the first merge candidate list.

[0008] Preferably, the history-based merging candidate may be added to the first merging candidate list if it has motion information that does not overlap with a predefined specific number of merging candidates from among the merging candidates included in the first merging candidate list.

[0009] Preferably, the history-based merge candidate may be added to the first merge candidate list if it has non-overlapping motion information with a particular spatial merge candidate included in the first merge candidate list.

[0010] Preferably, the history-based merging candidates may be derived from a predefined number of candidates within a history-based buffer that stores the history-based merging candidates.

[0011] Preferably, the step of constructing the second merge candidate list further includes the step of adding a history-based spatio-temporal merge candidate, and the history-based spatio-temporal candidate may be derived using the two history-based merge candidates and the temporal merge candidate.

[0012] Preferably, the motion vector of the history-based spatio-temporal candidate may be derived to be an average value of the motion vectors of the two history-based merge candidates and an average value of the motion vector of the temporal merge candidate.

[0013] In another aspect of the embodiment of the present specification, an apparatus for processing a video signal based on inter prediction includes a memory for storing the video signal, and a processor coupled to the memory, the processor configuring a first merge candidate list for a current block using spatial merge candidates and temporal merge candidates for the current block, configuring a second merge candidate list by adding a history-based merge candidate indicating motion information of a previously coded block of the current block to the first merge candidate list, obtaining a merge index in the second merge candidate list indicating a merge candidate to be applied to the current block, and generating a predictive block for the current block using motion information of the merge candidate indicated by the merge index, wherein the history-based merge candidate may be added to the first merge candidate list if it has motion information that does not overlap with a predefined merge candidate from among the merge candidates included in the first merge candidate list. Effect of the Invention

[0014] According to embodiments of the present disclosure, by limiting the redundancy checks for additions to the merge list (or AMVP list), the complexity associated with the redundancy checks can be improved and efficiency can be increased.

[0015] The effects obtained by the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from the following description. [Brief description of the drawings]

[0016] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide examples of the present invention and, together with the detailed description, explain the technical features of the present invention.

[0017] [Figure 1] As an example of a video signal processing device according to an embodiment of the present invention, an example of the functional configuration of an encoder will be described. [Diagram 2] 1 shows a schematic block diagram of a decoding device for decoding a video / image signal as an embodiment to which the present invention is applied. [Diagram 3] FIG. 2 is a diagram showing an example of a multi-type tree structure as an embodiment to which the present invention can be applied. [Figure 4] 1 is a diagram illustrating a signaling mechanism for partition information of a quadtree with nested multi-type tree structure as an embodiment to which the present invention can be applied. [Diagram 5] 1 is a diagram illustrating a method of dividing a CTU into multiple CUs based on a quadtree and nested multi-type tree structure as an embodiment to which the present invention can be applied. [Figure 6] FIG. 2 illustrates a method for restricting ternary-tree splitting as an embodiment to which the present invention can be applied. [Figure 7]1 is a diagram illustrating redundant partitioning patterns that can occur in partitioning a binary tree and a ternary tree as an embodiment to which the present invention can be applied. FIG. [Figure 8] 1 is a diagram illustrating an inter-prediction based video / image encoding method according to an embodiment of the present invention and an inter-prediction unit in an encoding device according to an embodiment of the present invention. [Figure 9] 1 is a diagram illustrating an inter-prediction based video / image encoding method according to an embodiment of the present invention and an inter-prediction unit in an encoding device according to an embodiment of the present invention. [Figure 10] 1 is a diagram illustrating a method for decoding inter-prediction based video / image according to an embodiment of the present invention and an inter-prediction unit in a decoding device according to an embodiment of the present invention. [Figure 11] 1 is a diagram illustrating a method for decoding inter-prediction based video / image according to an embodiment of the present invention and an inter-prediction unit in a decoding device according to an embodiment of the present invention. [Figure 12] FIG. 1 is a diagram for explaining peripheral blocks used in a merge mode or a skip mode as an embodiment to which the present invention is applied. [Figure 13] 1 is a flowchart illustrating a method for constructing a merge candidate list according to an embodiment of the present invention; [Figure 14] 1 is a flowchart illustrating a method for constructing a merge candidate list according to an embodiment of the present invention; [Figure 15] 1 shows examples of motion models according to an embodiment of the present invention. [Figure 16] 4 illustrates example motion vectors of control points for affine motion prediction according to an embodiment of the present invention. [Figure 17] 1 illustrates an example of motion vectors for each sub-block of a block to which affine motion prediction is applied, according to an embodiment of the present invention. [Figure 18] 4 illustrates an example of neighboring blocks used for affine motion prediction in an affine merge mode according to an embodiment of the present invention. [Figure 19] 4 illustrates an example of a block for which affine motion prediction is performed using neighboring blocks to which affine motion prediction is applied according to an embodiment of the present invention. [Figure 20] 11 is a diagram illustrating a method for generating a merge candidate list using neighboring affine coding blocks according to an embodiment of the present invention. [Figure 21] 11 is a diagram illustrating a method for constructing an affine merge candidate list using neighboring blocks coded by affine prediction according to an embodiment of the present invention. [Figure 22] 11 is a diagram illustrating a method for constructing an affine merge candidate list using neighboring blocks coded by affine prediction according to an embodiment of the present invention. [Figure 23] 13 illustrates an example of neighboring blocks used for affine motion prediction in an affine inter mode according to an embodiment of the present invention. [Figure 24] 13 illustrates an example of neighboring blocks used for affine motion prediction in an affine inter mode according to an embodiment of the present invention. [Diagram 25] 11 is a diagram illustrating a method for deriving motion vector candidates using motion information of neighboring blocks in an affine inter mode according to an embodiment of the present invention; [Figure 26] 11 is a diagram illustrating a method for deriving motion vector candidates using motion information of neighboring blocks in an affine inter mode according to an embodiment of the present invention; [Figure 27] 1 illustrates an example of a method for deriving a sub-block-based affine motion vector field according to an embodiment of the present invention. [Figure 28] 13 illustrates an exemplary method of generating a prediction block and a motion vector in inter prediction using an affine motion model according to an embodiment of the present invention. [Figure 29]4 illustrates a method for performing motion compensation based on motion vectors of control points according to an embodiment of the present invention; [Diagram 30] 4 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention; [Diagram 31] 4 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention; [Diagram 32] 4 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention; [Diagram 33] 4 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention; [Diagram 34] 4 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention; [Diagram 35] 4 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention; [Diagram 36] 4 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention; [Figure 37] 4 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention; [Figure 38] 4 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention; [Figure 39] 1 is a flow chart illustrating a method for storing HMVP according to an embodiment of the present specification. [Diagram 40] FIG. 2 is a diagram for explaining an HMVP table that operates in a non-restrictive FIFO manner according to an embodiment of the present specification. [Diagram 41]FIG. 2 is a diagram for explaining an HMVP table that operates in a restrictive FIFO manner according to an embodiment of the present specification. [Diagram 42] FIG. 2 is a diagram illustrating an HMVP LUT and a long-term HMVP LUT according to an embodiment of the present specification. [Diagram 43] FIG. 2 illustrates an example of a method for updating an HMVP LUT in accordance with an embodiment of the present specification. [Diagram 44] FIG. 1 illustrates a method for limiting the number of HMVP candidates subject to pruning checks according to an embodiment of the present specification. [Diagram 45] FIG. 1 illustrates an example of a method for performing a pruning check according to an embodiment of the present specification. [Diagram 46] 1 is a diagram for explaining a method of deriving H-STMVP candidates using motion vectors that refer to different reference pictures according to an embodiment of the present specification. [Figure 47] 2 is a flowchart illustrating a method for processing a video signal based on inter prediction according to an embodiment of the present invention; [Figure 48] 1 shows an example of a block diagram of an apparatus for processing a video signal according to an embodiment of the present specification. [Figure 49] 1 shows a video coding system to which the present invention can be applied; [Figure 50] As an embodiment to which the present invention is applied, a structural diagram of a content streaming system is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below in conjunction with the accompanying drawings is intended to describe exemplary embodiments of the present invention, and is not intended to show the only embodiments in which the present invention can be practiced. The following detailed description includes specific details to provide a complete understanding of the present invention. However, those skilled in the art will appreciate that the present invention can be practiced without such specific details.

[0019] In some cases, in order to avoid obscuring the concept of the present invention, well-known structures and devices may be omitted or shown in the form of a block diagram focusing on the core functions of each structure and device.

[0020] Furthermore, the terms used in this invention are currently common terms that are widely used as much as possible, but in certain cases, the applicant will use terms arbitrarily selected by the applicant to explain the invention. In such cases, the meanings will be clearly described in the detailed explanation of the relevant parts, so it is important to clarify that the terms should not be interpreted simply based on the names of the terms used in the description of this invention, but should be interpreted while also understanding the meanings of the relevant terms.

[0021] Specific terms used in the following description are provided to aid in understanding the present invention, and the use of such specific terms may be changed to other forms without departing from the technical spirit of the present invention. For example, signals, data, samples, pictures, frames, blocks, etc. may be appropriately substituted and interpreted in each encoding process.

[0022] Hereinafter, in this specification, a "processing unit" refers to a unit in which encoding / decoding processes such as prediction, transformation, and / or quantization are performed. Hereinafter, for convenience of explanation, a processing unit may be referred to as a "processing block" or a "block."

[0023] The processing unit may be interpreted as including a unit of a luma component and a unit of a chroma component. For example, the processing unit may correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0024] Also, the processing unit may be interpreted as including a unit of a luma component or a unit of a chroma component. For example, the processing unit may correspond to a coding tree block (CTB), a coding block (CB), a prediction block (PU), or a transform block (TB) of the luma component. Or, the processing unit may correspond to a coding tree block (CTB), a coding block (CB), a prediction block (PU), or a transform block (TB) of the chroma component. Also, without being limited thereto, the processing unit may be interpreted as including a unit of a luma component and a unit of a chroma component.

[0025] Furthermore, the processing units are not necessarily limited to square blocks, but may also be configured in the form of polygons having three or more vertices.

[0026] In the following description, pixels or picture elements are generally referred to as samples. Using a sample can mean using a pixel value or picture element value.

[0027] FIG. 1 shows a schematic block diagram of an encoding device in which encoding of a video / image signal is performed as an embodiment to which the present invention is applied.

[0028] Referring to FIG. 1, the encoding device 100 may include an image division unit 110, a subtraction unit 115, a transformation unit 120, a quantization unit 130, an inverse quantization unit 140, an inverse transformation unit 150, an addition unit 155, a filtering unit 160, a memory 170, an inter prediction unit 180, an intra prediction unit 185, and an entropy encoding unit 190. The inter prediction unit 180 and the intra prediction unit 185 may be commonly referred to as a prediction unit. That is, the prediction unit may include the inter prediction unit 180 and the intra prediction unit 185. The transformation unit 120, the quantization unit 130, the inverse quantization unit 140, and the inverse transformation unit 150 may be included in a residual processing unit. The residual processing unit may further include a subtraction unit 115. As an embodiment, the image division unit 110, subtraction unit 115, transformation unit 120, quantization unit 130, inverse quantization unit 140, inverse transformation unit 150, addition unit 155, filtering unit 160, inter prediction unit 180, intra prediction unit 185, and entropy encoding unit 190 may be configured as one hardware component (e.g., an encoder or processor). Also, the memory 170 may include a decoded picture buffer (DPB) and may be configured as a digital storage medium.

[0029] The image division unit 110 may divide an input image (or picture, frame) input to the encoding device 100 into one or more processing units. As an example, the processing units may be referred to as coding units (CUs). In this case, the coding units may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) based on a quad-tree binary-tree (QTBT) structure. For example, one coding unit may be divided into a plurality of coding units of a deeper depth based on a quad-tree structure and / or a binary tree structure. In this case, for example, the quad-tree structure may be applied first and the binary tree structure may be applied later. Alternatively, the binary tree structure may be applied first. The coding procedure according to the present invention may be performed based on a final coding unit that is not further divided. In this case, based on coding efficiency according to image characteristics, the largest coding unit is immediately used as the final coding unit, or the coding unit is recursively divided into coding units of lower depths as necessary, and a coding unit of an optimal size is used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be divided or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0030] The term unit is sometimes used interchangeably with terms such as block or area. In the general case, an MxN block can refer to a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally refer to a pixel or a pixel value, or it can refer to only a pixel / pixel value of the luma component or only a pixel / pixel value of the chroma component. A sample is used as a term that corresponds to a pixel or pel of one picture (or image).

[0031] The encoding device 100 may subtract a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 180 or the intra prediction unit 185 from an input video signal (original block, original sample array) to generate a residual signal (residual signal, remaining block, remaining sample array), and the generated residual signal is transmitted to the conversion unit 120. In this case, as shown in the figure, a unit in the encoder 100 that subtracts a prediction signal (predicted block, prediction sample array) from an input video signal (original block, original sample array) may be referred to as a subtraction unit 115. The prediction unit may predict a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including a prediction sample of the current block.

[0032] The prediction unit may determine whether intra prediction or inter prediction is applied in units of a current block or CU. The prediction unit may generate various information related to prediction, such as prediction mode information, as described below in the description of each prediction mode, and transmit the information to the entropy encoding unit 190. The prediction information may be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

[0033] The intra prediction unit 185 may predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from the current block depending on the prediction mode. In the intra prediction, the prediction mode may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the fineness of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the settings. The intra prediction unit 185 may also determine the prediction mode to be applied to the current block using the prediction modes applied to the neighboring blocks.

[0034] The inter prediction unit 180 may derive a predicted block of the current block based on a reference block (an array of reference samples) identified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring blocks may be called collocated reference blocks, collocated CUs, etc., and the reference picture including the temporal neighboring blocks may be called collocated pictures (colPic). For example, the inter prediction unit 180 may generate information indicating which candidates are used to derive a motion vector and / or a reference picture index of the current block by forming a motion information candidate list based on neighboring blocks. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit 180 may use motion information of neighboring blocks as motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the case of motion vector prediction (MVP) mode, the motion vector of the surrounding blocks can be used in the motion vector predictor to indicate the motion vector of the current block by signaling the motion vector difference.

[0035] The prediction signal generated by the inter prediction unit 180 or the intra prediction unit 185 is used to generate a reconstructed signal or a residual signal.

[0036] The transform unit 120 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, the GBT refers to a transform obtained from a graph when the relationship information between pixels is represented by a graph. The CNT refers to a transform obtained based on a predicted signal generated using all previously reconstructed pixels. In addition, the transform process may be applied to pixel blocks having the same square size, or may be applied to variable-sized blocks that are not square.

[0037] The quantizer 130 quantizes the transform coefficients and transmits the quantized transform coefficients to the entropy encoder 190, which may encode the quantized signal (information on the quantized transform coefficients) and output the quantized signal as a bitstream. The information on the quantized transform coefficients may be referred to as residual information. The quantizer 130 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and may generate information on the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoder 190 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 190 may encode information required for video / image restoration (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in NAL (network abstraction layer) unit units. The bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitting unit (not shown) for transmitting the signal output from the entropy encoding unit 190 and / or a storing unit (not shown) for storing the signal may be configured as an element inside / outside the encoding device 100, or the transmitting unit may be a component of the entropy encoding unit 190.

[0038] The quantized transform coefficients output from the quantizer 130 are used to generate a prediction signal. For example, the quantized transform coefficients can be inversely quantized and inversely transformed through the inverse quantizer 140 and the inverse transformer 150 in the loop to restore a residual signal. The adder 155 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit 180 or the intra prediction unit 185. When there is no residual of the block to be processed, as in the case where the skip mode is applied, the predicted block is used as a reconstructed block. The adder 155 can be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal is used for intra prediction of the next block to be processed in the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0039] The filtering unit 160 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 160 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may store the modified reconstructed picture in the memory 170, specifically, in the DPB of the memory 170. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, and the like. The filtering unit 160 may generate various information related to filtering, as will be described later in the description of each filtering method, and transmit the information to the entropy encoding unit 190. The information related to filtering may be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

[0040] The modified reconstructed picture transmitted to the memory 170 is used as a reference picture by the inter prediction unit 180. In this way, when inter prediction is applied, the encoding apparatus can avoid a prediction mismatch between the encoding apparatus 100 and the decoding apparatus, and can also improve coding efficiency.

[0041] The memory 170DPB may store the modified reconstructed picture for use as a reference picture from the inter prediction unit 180. The memory 170 may store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter prediction unit 180 to be used as motion information of a spatially neighboring block or motion information of a temporally neighboring block. The memory 170 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 185.

[0042] FIG. 2 is a schematic block diagram of a decoding device for decoding a video / image signal as an embodiment to which the present invention is applied.

[0043] Referring to FIG. 2, the decoding device 200 may include an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an adder 235, a filtering unit 240, a memory 250, an inter prediction unit 260, and an intra prediction unit 265. The inter prediction unit 260 and the intra prediction unit 265 may be collectively referred to as a prediction unit. That is, the prediction unit may include an inter prediction unit 180 and an intra prediction unit 185. The inverse quantization unit 220 and the inverse transform unit 230 may be collectively referred to as a residual processing unit. That is, the residual processing unit may include an inverse quantization unit 220 and an inverse transform unit 230. The entropy decoding unit 210, the inverse quantization unit 220, the inverse transform unit 230, the adder 235, the filtering unit 240, the inter prediction unit 260, and the intra prediction unit 265 may be configured by one hardware component (e.g., a decoder or a processor) according to an embodiment. The memory 170 may also include a decoded picture buffer (DPB) and may be implemented by a digital storage medium.

[0044] When a bitstream including video / image information is input, the decoding device 200 can restore an image corresponding to the process in which the video / image information from the encoding device of Fig. 1 was processed. For example, the decoding device 200 can perform decoding using a processing unit applied in the encoding device. Thus, the processing unit for decoding can be, for example, a coding unit, which can thus separate a quad tree structure and / or a binary tree structure from a coding tree unit or a maximal coding unit. Then, the restored image signal decoded and output through the decoding device 200 can be reproduced by a reproduction device.

[0045] The decoding device 200 may receive a signal output from the encoding device of FIG. 1 in the form of a bitstream, and may decode the received signal through the entropy decoding unit 210. For example, the entropy decoding unit 210 may derive information (ex. video / image information) required for image restoration (or picture restoration) by analyzing the bitstream. For example, the entropy decoding unit 210 may decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients related to residuals. More specifically, the CABAC entropy decoding method may receive nulls corresponding to each syntax element from the bitstream, determine a context model using information of the syntax element to be decoded and decoded information of the surrounding and blocks to be decoded, or symbol / null information decoded in a previous step, predict the occurrence probability of a bin based on the determined context model, and perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method may determine a context model and then update the context model using the decoded symbol / empty information for the context model of the next symbol / empty. Prediction information from the information decoded by the entropy decoding unit 2110 is provided to a prediction unit (inter prediction unit 260 and intra prediction unit 265), and residual values ​​entropy decoded from the entropy decoding unit 210, i.e., quantized transform coefficients and related parameter information, may be input to the inverse quantization unit 220. In addition, filtering information from the information decoded by the entropy decoding unit 210 may be provided to the filtering unit 240. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding device may be configured as an internal / external element of the decoding device 200, or the receiving unit may be a component of the entropy decoding unit 210.

[0046] The inverse quantization unit 220 may inverse quantize the quantized transform coefficients to output transform coefficients. The inverse quantization unit 220 may rearrange the quantized transform coefficients in the form of a two-dimensional block. In this case, the rearrangement may be performed based on a coefficient scan order performed in the encoding device. The inverse quantization unit 220 may perform inverse quantization of the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0047] In the inverse transform unit 230, the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0048] The prediction unit may perform prediction of a current block and generate a predicted block including a prediction sample of the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoding unit 210, and may determine a specific intra / inter prediction mode.

[0049] The intra prediction unit 265 may predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located far away depending on the prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 265 may also determine the prediction mode to be applied to the current block using the prediction modes applied to the neighboring blocks.

[0050] The inter prediction unit 260 may derive a predicted block of the current block based on a reference block (an array of reference samples) identified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on an inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter prediction unit 260 may configure a motion information candidate list based on the neighboring blocks, and derive a motion vector and / or a reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the prediction information may include information indicating the inter prediction mode of the current block.

[0051] The adder 235 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to a prediction signal (predicted block, predicted sample array) output from the inter prediction unit 260 or the intra prediction unit 265. When there is no residual for the current block, such as when the skip mode is applied, the predicted block is used as the reconstructed block.

[0052] The adder 235 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of a next block to be processed in the current picture, and may be used for inter prediction of a next picture after filtering, as described below.

[0053] The filtering unit 240 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 240 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may transmit the modified reconstructed picture to the memory 250, specifically, to the DPB of the memory 250. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc.

[0054] The (modified) reconstructed picture stored in the DPB of the memory 250 is used as a reference picture by the inter prediction unit 260. The memory 250 may store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter prediction unit 260 to be used as motion information of a spatially neighboring block or motion information of a temporally neighboring block. The memory 170 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 265.

[0055] In this specification, the embodiments described for the filtering unit 160, inter prediction unit 180 and intra prediction unit 185 of the encoding device 100 can also be applied identically or correspondingly to the filtering unit 240, inter prediction unit 260 and intra prediction unit 265 of each decoding device 200.

[0056] Block Partitioning

[0057] The video / image coding method according to the present document may be performed based on various detailed techniques, each of which will be briefly described as follows: It will be apparent to those skilled in the art that the techniques described below may be related to related procedures such as prediction, residual processing ((inverse) transformation, (inverse) quantization, etc.), coding of syntax elements, filtering, partitioning / division, etc., in the video / image encoding / decoding procedures described above and / or below.

[0058] The block partitioning procedure according to this document is performed by the image partitioning unit 110 of the encoding device described above, and partitioning-related information is processed (encoded) by the entropy encoding unit 190 and can be transmitted to the decoding device in the form of a bitstream. The entropy decoding unit 210 of the decoding device can derive a block partitioning structure of the current picture based on the partitioning-related information acquired from the bitstream, and perform a series of procedures for image decoding based on the block partitioning structure (e.g., prediction, residual processing, block reconstruction, in-loop filtering, etc.).

[0059] Partitioning of picture into CTUs

[0060] A picture can be divided into a sequence of coding tree units (CTUs). A CTU can correspond to a coding tree block (CTB). Alternatively, a CTU can contain a coding tree block of luma samples and two coding tree blocks of corresponding chroma samples. That is, for a picture containing three sample arrays, a CTU contains an NxN block of luma samples and two corresponding blocks of chroma samples.

[0061] The maximum allowed size of a CTU for coding, prediction, etc. may be different from the maximum allowed size of a CTU for transforms, for example, the maximum allowed size of a luma block in a CTU may be 128x128.

[0062] Partitioning of the CTUs using a tree structure

[0063] A CTU may be divided into CUs based on a quad-tree (QT) structure. A quad-tree structure may be referred to as a quaternary tree structure because it reflects various local characteristics. Meanwhile, in this document, a CTU may be divided based on division of a multi-type tree structure including not only a quad-tree but also a binary-tree (BT) and a ternary-tree (TT). Hereinafter, a QTBT structure may include a quad-tree and a binary-tree based division structure, and a QTBTTT may include a quad-tree, a binary-tree and a ternary-tree based division structure. Alternatively, a QTBT structure may include a quad-tree, a binary-tree and a ternary-tree based division structure. In a coding tree structure, a CU may have a square or rectangular shape. A CTU may first be divided into a quad-tree structure. Thereafter, a leaf node of the quad-tree structure may be further divided by a multi-type tree structure.

[0064] FIG. 3 is a diagram showing an example of a multi-type tree structure as an embodiment to which the present invention can be applied.

[0065] In one embodiment of the present specification, the multi-type tree structure may include four splitting types as shown in Fig. 3. The four splitting types may include vertical binary splitting (SPLIT_BT_VER), horizontal binary splitting (SPLIT_BT_HOR), vertical ternary splitting (SPLIT_TT_VER), and horizontal ternary splitting (SPLIT_TT_HOR).

[0066] The leaf nodes of the multi-type tree structure may be called CUs. Such CUs are used for prediction and transformation procedures. In general, CUs, PUs, and TUs may have the same block size in this document. However, CUs and TUs may have different block sizes if the maximum supported transform length is smaller than the width or height of a color component of the CU.

[0067] FIG. 4 is a diagram illustrating a signaling mechanism for partition division information of a quadtree with nested multi-type tree structure as an embodiment to which the present invention can be applied.

[0068] Here, the CTU is treated as the root of the quadtree and is first partitioned in the quadtree structure. Each quadtree leaf node can then be further partitioned in the multitype tree structure. In the multitype tree structure, a first flag (ex. mtt_split_cu_flag) is signaled to indicate whether the node is to be additionally partitioned. If the node is to be additionally partitioned, a second flag (ex. mtt_split_cu_vertical_flag) can be signaled to indicate the splitting direction. Then, a third flag (ex. mtt_split_cu_binary_flag) can be signaled to indicate whether the split type is binary or turnary. For example, based on the mtt_split_cu_vertical_flag and the mtt_split_cu_binary_flag, a multi-type tree splitting mode (MttSplitMode) of a CU can be derived as shown in Table 1 below.

[0069] [Table 1]

[0070] FIG. 5 is a diagram illustrating a method of dividing a CTU into multiple CUs based on a quadtree and nested multi-type tree structure as an embodiment to which the present invention can be applied.

[0071] Here, the bold block edges indicate quad-tree partitioning and the remaining edges indicate multi-type tree partitioning. The partitioning of the quad-tree with the multi-type tree can provide a content-adapted coding tree structure. A CU can correspond to a coding block (CB). Alternatively, a CU can contain a coding block of luma samples and two coding blocks of corresponding chroma samples. The size of a CU can be as large as a CTU or as small as 4x4 in luma sample units. For example, in a 4:2:0 color format (or chroma format), the maximum chroma CB size can be 64x64 and the minimum chroma CB size can be 2x2.

[0072] In this document, for example, the maximum allowed luma TB size may be 64x64 and the maximum allowed chroma TB size may be 32x32. If the width or height of a CB split along the tree structure is greater than the maximum transform width or height, the CB may be automatically (or implicitly) split until the horizontal and vertical TB size restrictions are satisfied.

[0073] Meanwhile, for a quad-tree coding tree scheme with a multi-type tree, the following parameters can be defined and identified in the sps syntax element:

[0074] - CTU size: the root node size of a quaternary tree

[0075] - MinQTSize: the minimum allowed quaternary tree leaf node size

[0076] - MaxBtSize: (the maximum allowed binary tree root node size)

[0077] - MaxTtSize: (the maximum allowed ternary tree root node size)

[0078] - MaxMttDepth: the maximum allowed hierarchy depth of multi-type tree splitting from a quadtree leaf

[0079] - MinBtSize: the minimum allowed binary tree leaf node size

[0080] - MinTtSize: the minimum allowed ternary tree leaf node size

[0081] As an example of a quad-tree coding tree structure with a multi-type tree, the CTU size can be set to 64x64 blocks of 128x128 luma samples and two corresponding chroma samples (in 4:2:0 chroma format). In this case, MinOTSize can be set to 16x16, MaxBtSize can be set to 128x128, MaxTtSzie can be set to 64x64, MinBtSize and MinTtSize (for both width and height) can be 4x4, and MaxMttDepth can be set to 4. Quote tree partitioning can be applied to the CTU to generate quad-tree leaf nodes. The quad-tree leaf nodes can be referred to as leaf QT nodes. The quad-tree leaf nodes can have a size from 16x16 (i.e. the MinOTSize) to 128x128 (i.e. the CTU size). If the leaf QT node is 128x128, it may not be further split into a binary tree / ternary tree, because in this case the split would exceed MaxBtsize and MaxTtszie (i.e. 64x64). Otherwise, the leaf QT node may be further split into a multitype tree. Thus, the leaf QT node is the root node of the multitype tree, and the leaf QT node may have a multitype tree depth (mttDepth) value of 0. If the multitype tree depth reaches MaxMttdepth (ex. 4), no further splits may be considered. If the width of the multitype tree node is equal to MinBtSize and less than or equal to 2xMinTtSize, no further horizontal splits may be considered. If the height of the multitype tree node is equal to MinBtSize and less than or equal to 2xMinTtSize, no further vertical splits may be considered.

[0082] FIG. 6 is a diagram illustrating a method for restricting ternary-tree splitting as an embodiment to which the present invention can be applied.

[0083] Referring to FIG 6, in order to allow the design of a 64x64 luma block and a 32x32 chroma pipeline in a hardware decoder, the TT partitioning may be restricted in certain cases. For example, if the width or height of a luma coding block is greater than a preset specific value (e.g., 32, 64), the TT partitioning may be restricted as shown in FIG 6.

[0084] In this document, the coding tree scheme can support luma and chroma blocks having a tree structure of separate blocks.

[0085] For P and B slices, the luma and chroma CTBs in one CTU may be restricted to have the same coding tree structure. However, for an I slice, the luma and chroma blocks may have separate block tree structures. If an individual block tree mode is applied, the luma CTB may be divided into CUs based on a specific coding tree structure, and the chroma CTB may be divided into chroma CUs based on another coding tree structure. This may mean that a CU in an I slice may consist of a coding block of a luma component or a coding block of two chroma components, and a CU in a P or B slice may consist of blocks of three color components.

[0086] Although the quad tree coding tree structure with a multi-type tree was described in the above "Partitioning of the CTUs using a tree structure", the structure in which the CU is divided is not limited to this. For example, the BT structure and the TT structure can be interpreted as concepts included in the structure of a multiple partitioning tree (MPT), and the CU can be interpreted as being divided through the QT structure and the MPT structure. In one example of dividing a CU through the QT structure and the MPT structure, a syntax element (e.g., MPT_split_type) including information on whether the leaf node of the QT structure is divided into several blocks and a syntax element (e.g., MPT_split_mode) including information on whether the leaf node of the QT structure is divided vertically or horizontally are signaled, so that the division structure can be determined.

[0087] In addition, in different examples, the CUs may be divided in a different manner, such as the QT structure, the BT structure, or the TT structure, that is, unlike the lower depth CUs being divided into 1 / 4 the size of the higher depth CUs based on the QT structure, the lower depth CUs being divided into 1 / 2 the size of the higher depth CUs based on the BT structure, and the lower depth CUs being divided into 1 / 4 or 1 / 2 the size of the higher depth CUs based on the TT structure, the lower depth CUs may be divided into 1 / 5, 1 / 3, 3 / 8, 3 / 5, 2 / 3, or 5 / 8 the size of the higher depth CUs, as the case may be, and the manner in which the CUs are divided is not limited thereto.

[0088] If a portion of a tree node block exceeds a bottom or right picture boundary, the tree node block may be constrained such that all samples of all coded CUs are located within the picture boundary. In this case, for example, the following splitting rules may be applied:

[0089] - If a portion of a tree node block exceeds both the bottom and the right picture boundaries,

[0090] - If the block is a QT node and the size of the block is larger than the minimum QT size, the block is forced to be split with QT split mode.

[0091] - Otherwise, the block is forced to be split with SPLIT_BT_HOR mode

[0092] - Otherwise if a portion of a tree node block exceeds the bottom picture boundaries,

[0093] - If the block is a QT node, and the size of the block is larger than the minimum QT size, and the size of the block is larger than the maximum BT size, the block is forced to be split with QT split mode.

[0094] - Otherwise, if the block is a QT node, and the size of the block is larger than the minimum QT size and the size of the block is smaller than or equal to the maximum BT size, the block is forced to be split with QT split mode or SPLIT_BT_HOR mode.

[0095] - Otherwise(the block is a BTT node or the size of the block is smaller than or equal to the minimum QT size), the block is forced to be split with SPLIT_BT_HOR mode.

[0096] - Otherwise if a portion of a tree node block exceeds the right picture boundaries,

[0097] - If the block is a QT node, and the size of the block is larger than the minimum QT size, and the size of the block is larger than the maximum BT size, the block is forced to be split with QT split mode.

[0098] - Otherwise, if the block is a QT node, and the size of the block is larger than the minimum QT size and the size of the block is smaller than or equal to the maximum BT size, the block is forced to be split with QT split mode or SPLIT_BT_VER mode.

[0099] - Otherwise(the block is a BTT node or the size of the block is smaller than or equal to the minimum QT size), the block is forced to be split with SPLIT_BT_VER mode.

[0100] Meanwhile, the quad-tree coding block structure with the multi-type tree described above can provide a very flexible block partitioning structure. Because of the partitioning types supported by the multi-type tree, other partitioning patterns can potentially result in the same coding block structure in some cases. By limiting the occurrence of such redundant partitioning patterns, the amount of data of partitioning information can be reduced. This will be described with reference to the following figure.

[0101] FIG. 7 is a diagram illustrating a redundant division pattern that can occur in the division of a binary tree and a ternary tree as an embodiment to which the present invention can be applied.

[0102] As shown in FIG. 7, two levels of consecutive binary splits in one direction have the same coding block structure as the binary split of the central partition after the Turner-Lee partition. In this case, the split of the binary tree of the central partition of the Turner-Lee tree in the given direction can be restricted. Such a restriction can be applied to the CU of all pictures. When such a specific split is restricted, the signaling of the corresponding syntax element can be modified to reflect such a restricted case, thereby reducing the number of bits signaled for partitioning. For example, as in the example shown in FIG. 7, when the split of the binary tree of the central partition of the CU is restricted, the mtt_split_cu_binary_flag syntax element indicating whether the split is a binary split or a Tenor-Lee split is not signaled and can be inferred by the decoder as a value of 0.

[0103] Prediction

[0104] To reconstruct the current processing unit for which decoding is performed, decoded portions of the current picture or other pictures in which the current processing unit is included can be used.

[0105] A picture (slice) that uses only the current picture for reconstruction, i.e., performs only intra-screen prediction, can be called an intra-picture or I-picture (slice); a picture (slice) that uses up to one motion vector and reference index to predict each unit can be called a predictive picture or P-picture (slice); and a picture (slice) that uses up to two motion vectors and reference indexes can be called a bi-predictive picture or B-picture (slice).

[0106] Intra prediction refers to a prediction method that derives a current processing block from data elements (e.g., sample values, etc.) of the same decoded picture (or slice), i.e., a method of predicting pixel values ​​of a current processing block by referring to a reconstructed region in the current picture.

[0107] Inter prediction will be explained in more detail below.

[0108] Inter prediction (or inter prediction)

[0109] Inter prediction refers to a prediction method that derives a current processing block based on data elements (e.g., sample values ​​or motion vectors, etc.) of pictures other than the current picture, that is, a method that predicts pixel values ​​of the current processing block by referring to reconstructed areas in other reconstructed pictures other than the current picture.

[0110] Inter prediction (or inter-picture prediction) is a technique that removes redundancy that exists between pictures, mostly through motion estimation and motion compensation.

[0111] In the embodiments of the present specification, detailed techniques of the inter prediction method described above in Figures 1 and 2 are described, and in the case of a decoder, it can be represented by an inter prediction based video / image decoding method of Figure 10 and an inter prediction unit in a decoding device of Figure 11, which will be described later. In addition, in the case of an encoder, it can be represented by an inter prediction based video / image encoding method of Figure 8 and an inter prediction unit in an encoding device of Figure 9, which will be described later. Furthermore, data encoded according to Figures 8 and 9 can be stored in the form of a bitstream.

[0112] The prediction unit of the encoding device / decoding device may perform inter prediction on a block-by-block basis to derive a prediction sample. Inter prediction may refer to a prediction derived in a manner dependent on data elements (e.g., sample values, motion information, etc.) of pictures other than the current picture. When inter prediction is applied to the current block, a predicted block (prediction sample array) of the current block may be derived based on a reference block (array of reference samples) identified by a motion vector on a reference picture pointed to by a reference picture index.

[0113] In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information of the current block can be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.).

[0114] When inter prediction is applied, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and the reference picture including the temporal neighboring blocks may be called collocated pictures (colPic).

[0115] For example, a motion information candidate list may be constructed based on neighboring blocks of the current block, and flag or index information may be signaled indicating which candidate is selected (used) to derive a motion vector and / or a reference picture index for the current block.

[0116] Inter prediction may be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the motion information of the current block may be the same as the motion information of the selected neighboring block. In the case of skip mode, unlike the merge mode, the residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the selected neighboring block may be used as a motion vector predictor, and a motion vector difference may be signaled. In this case, the motion vector of the current block may be derived using the combination of the motion vector predictor and the motion vector difference.

[0117] 8 and 9 are diagrams illustrating an inter-prediction based video / image encoding method according to an embodiment of the present invention and an inter-prediction unit in an encoding device according to an embodiment of the present invention.

[0118] 8 and 9, S801 may be performed by the inter prediction unit 180 of the encoding device, and S802 may be performed by the residual processing unit of the encoding device. Specifically, S802 may be performed by the subtraction unit 115 of the encoding device. In S803, prediction information may be derived by the inter prediction unit 180 and encoded by the entropy encoding unit 190. In S803, residual information may be derived by the residual processing unit and encoded by the entropy encoding unit 190. The residual information is information about the residual sample. The residual information may include information about quantized transform coefficients for the residual sample.

[0119] As described above, the residual samples are derived into transform coefficients through the transform unit 120 of the encoding device, and the transform coefficients may be derived as quantized transform coefficients through the quantization unit 130. Information about the quantized transform coefficients may be encoded in the entropy encoding unit 190 through a residual coding procedure.

[0120] The encoding apparatus performs inter prediction of a current block (S801). The encoding apparatus may derive an inter prediction mode and motion information of the current block and generate a prediction sample of the current block. Here, the steps of determining an inter prediction mode, deriving motion information, and generating a prediction sample may be performed simultaneously, or one step may be performed prior to the other steps. For example, the inter prediction unit 180 of the encoding apparatus may include a prediction mode determination unit 181, a motion information derivation unit 182, and a prediction sample derivation unit 183, and the prediction mode determination unit 181 may determine a prediction mode of the current block, the motion information derivation unit 182 may derive motion information of the current block, and the prediction sample derivation unit 183 may derive a motion sample of the current block.

[0121] For example, the inter prediction unit 180 of the encoding device may search for a block similar to the current block within a certain region (search area) of a reference picture through motion estimation, and derive a reference block whose difference from the current block is minimum or equal to or less than a certain criterion. Based on this, an index of a reference picture indicating a reference picture in which the reference block is located may be derived, and a motion vector may be derived based on the difference between the positions of the reference block and the current block. The encoding device may determine a mode to be applied to the current block among various prediction modes. The encoding device may compare RD costs for the various prediction modes to determine an optimal prediction mode for the current block.

[0122] For example, when a skip mode or a merge mode is applied to the current block, the encoding apparatus may construct a merge candidate list (to be described later) and derive a reference block, among reference blocks indicated by merge candidates included in the merge candidate list, whose difference between the current block and the current block is minimum or equal to or less than a certain criterion. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to a decoding apparatus.

[0123] Motion information of the current block may be derived using motion information of the selected merging candidates.

[0124] As another example, when the (A)MVP mode is applied to the current block, the encoding apparatus may construct an (A)MVP candidate list (to be described later) and use a motion vector of a selected MVP candidate among the MVP (motion vector predictor) candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, a motion vector indicating a reference block derived by the above-mentioned motion estimation is used as the motion vector of the current block, and among the MVP candidates, an MVP candidate having a motion vector with the smallest difference from the motion vector of the current block may become the selected MVP candidate. A motion vector difference (MVD), which is a difference obtained by subtracting the MVP from the motion vector of the current block, may be derived. In this case, information regarding the MVD may be signaled to the decoding apparatus. In addition, when the (A)MVP mode is applied, a value of an index of the reference picture may be configured as reference picture index information and may be separately signaled to the decoding apparatus.

[0125] The encoding apparatus may derive a residual sample based on the predicted sample (S802). The encoding apparatus may derive the residual sample through a comparison between an original sample of the current block and the predicted sample.

[0126] The encoding apparatus encodes video information including prediction information and residual information (S803). The encoding apparatus may output the encoded video information in the form of a bitstream. The prediction information may include prediction mode information (ex. skip flag, merge flag or mode index, etc.) and information regarding motion information, which is information related to the prediction procedure. The information regarding the motion information may include candidate selection information (ex. merge index, MVP flag or MVP index), which is information for deriving a motion vector. In addition, the information regarding the motion information may include the above-mentioned information regarding MVD and / or index information of a reference picture.

[0127] In addition, the information regarding the motion information may include information indicating whether L0 prediction, L1 prediction, or pair (bi) prediction is applied. The residual information is information regarding the residual sample. The residual information may include information regarding quantized transform coefficients for the residual sample.

[0128] The output bitstream can be stored in a (digital) storage medium and then transmitted to the decoding device, or can be transmitted to the decoding device via a network.

[0129] Meanwhile, as described above, the encoding apparatus can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is because the encoding apparatus derives the same prediction result as that performed by the decoding apparatus, thereby improving coding efficiency. Therefore, the encoding apparatus can store the reconstructed picture (or reconstructed samples, reconstructed blocks) in a memory and use it as a reference picture for inter prediction. As described above, an in-loop filter procedure can be further applied to the reconstructed picture.

[0130] 10 and 11 are diagrams illustrating an inter-prediction based video / image decoding method according to an embodiment of the present invention and an inter-prediction unit in a decoding device according to an embodiment of the present invention.

[0131] 10 and 11, the decoding device may perform actions corresponding to those performed by the encoding device. The decoding device may perform prediction on the current block based on the received prediction information and derive a prediction sample.

[0132] S1001 to S1003 may be performed by the inter prediction unit 260 of the decoding device, and the residual information of S1004 may be acquired from a bitstream by the entropy decoding unit 210 of the decoding device. The residual processing unit of the decoding device may derive a residual sample of a current block based on the residual information. Specifically, the inverse quantization unit 220 of the residual processing unit may perform inverse quantization based on the quantized transform coefficient derived based on the residual information to derive a transform coefficient, and the inverse transform unit 230 of the residual processing unit may perform inverse transform of the transform coefficient to derive a residual sample of the current block. S1005 may be performed by the addition unit 235 or a reconstruction unit of the decoding device.

[0133] Specifically, the decoding device may determine a prediction mode of the current block based on received prediction information (S1001). The decoding device may determine which inter prediction mode is applied to the current block based on prediction mode information in the prediction information.

[0134] For example, it may be determined whether the merge mode is applied to the current block or whether the (A)MVP mode is determined based on the merge flag, or one of various inter prediction mode candidates may be selected based on the mode index. The inter prediction mode candidates may include skip mode, merge mode, and / or (A)MVP mode, or may include various inter prediction modes described below.

[0135] The decoding apparatus derives motion information of the current block based on the determined inter prediction mode (S1002). For example, when a skip mode or a merge mode is applied to the current block, the decoding apparatus may form a merge candidate list (to be described later) and select one of the merge candidates included in the merge candidate list. The selection may be performed based on the selection information (merge index) described above. Motion information of the current block may be derived using motion information of the selected merge candidate. The motion information of the selected merge candidate is used as motion information of the current block.

[0136] As another example, when the (A)MVP mode is applied to the current block, the decoding apparatus may construct an (A)MVP candidate list described later, and may use a motion vector of a selected MVP candidate among the MVP (motion vector predictor) candidates included in the (A)MVP candidate list for the MVP of the current block. The selection may be performed based on the selection information (mvp flag or mvp index) described above. In this case, the MVD of the current block may be derived based on information on the MVD, and the motion vector of the current block may be derived based on the MVP of the current block and the MVD. Also, the reference picture index of the current block may be derived based on index information of the reference picture. In the reference picture list for the current block, a picture pointed to by the reference picture index may be derived as a reference picture referenced for inter prediction of the current block.

[0137] On the other hand, the motion information of the current block may be derived without constructing a candidate list, as described below, in which case the motion information of the current block may be derived according to a procedure disclosed in a prediction mode, as described below, in which case the construction of the candidate list as described above may be omitted.

[0138] The decoding device may generate a prediction sample of the current block based on the motion information of the current block (S1003). In this case, the reference picture may be derived based on the reference picture index of the current block, and the prediction sample of the current block may be derived using a sample of a reference block to which the motion vector of the current block points on the reference picture. In this case, as described below, a filtering procedure may be performed on all or a part of the prediction samples of the current block, depending on the circumstances.

[0139] For example, the inter prediction unit 260 of the decoding device may include a prediction mode determination unit 261, a motion information derivation unit 262, and a prediction sample derivation unit 263, and may determine a prediction mode of the current block based on prediction mode information received by the prediction mode determination unit 261, derive motion information of the current block (such as a motion vector and / or a reference picture index) based on information regarding the motion information received from the motion information derivation unit 262, and derive a prediction sample of the current block in the prediction sample derivation unit 263.

[0140] The decoding apparatus generates a residual sample of the current block based on the received residual information (S1004). The decoding apparatus generates a reconstructed sample of the current block based on the prediction sample and the residual sample, and may generate a reconstructed picture based on the reconstructed sample (S1005). Thereafter, as described above, an in-loop filter procedure may be further applied to the reconstructed picture.

[0141] As described above, the inter prediction procedure may include an inter prediction mode determination step, a motion information derivation step according to the determined prediction mode, and a prediction execution step (creating a prediction sample) based on the derived motion information.

[0142] Determination of inter prediction mode

[0143] For prediction of the current block in a picture, various inter prediction modes can be used. For example, in merge mode and skip mode, various modes such as MVP mode and affine mode can be used. Decoder side motion vector refinement mode (DMVR) and adaptive motion vector resolution mode (AMVR) are additionally used as auxiliary modes. Affine mode can also be called affine motion prediction mode. MVP mode can also be called advanced motion vector prediction mode.

[0144] Prediction mode information indicating an inter prediction mode of a current block may be signaled from an encoding apparatus to a decoding apparatus. The prediction mode information may be included in a bitstream and received by the decoding apparatus. The prediction mode information may include index information indicating one of a number of candidate modes. Alternatively, the inter prediction mode may be indicated through hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags.

[0145] For example, a skip flag may be signaled to indicate whether or not a skip mode is applied, and if the skip mode is not applicable, a merge flag may be signaled to indicate whether or not a merge mode is applied, and if the merge mode is not applicable, an MVP mode may be applied, or a flag for additional division may be further signaled. The affine mode may be signaled in an independent mode, or in a mode dependent on the merge mode or MVP mode, etc. For example, the affine mode may be configured as one candidate in a merge candidate list or an MVP candidate list, as described below.

[0146] Derivation of motion information according to inter prediction mode

[0147] Inter prediction may be performed using motion information of the current block. The encoding apparatus may derive optimal motion information of the current block through a motion estimation procedure. For example, the encoding apparatus may use an original block in an original picture of the current block to search for a similar reference block having a high correlation in a fractional pixel unit within a determined search range in the reference picture, and may derive motion information through this. The similarity of the block may be derived based on a difference in phase-based sample values. For example, the similarity of the block may be calculated based on the SAD between the current block (or the template of the current block) and the reference block (or the template of the reference block). In this case, the motion information may be derived based on the reference block having the smallest SAD in the search area. The derived motion information may be signaled to the decoding apparatus based on several methods depending on the inter prediction mode.

[0148] Merge and Skip Modes

[0149] FIG. 12 is a diagram for explaining peripheral blocks used in the merge mode or skip mode as an embodiment to which the present invention is applied.

[0150] When a merge mode is applied, the motion information of the current prediction block is not directly transmitted, but the motion information of the current prediction block is derived using the motion information of the surrounding prediction block. Therefore, the motion information of the current prediction block can be indicated by transmitting flag information indicating that the merge mode is used and a merge index indicating which surrounding prediction block is used.

[0151] In one embodiment of the present invention,

[0152] In one embodiment of the present specification, an encoder may search for merge candidate blocks used to derive motion information of a current prediction block to perform a merge mode. For example, up to five merge candidate blocks may be used, but the present invention is not limited thereto. The maximum number of merge candidate blocks may be transmitted in a slice header (or a tile group header), but the present invention is not limited thereto. After finding the merge candidate blocks, the encoder may generate a merge candidate list and select a merge candidate block having the smallest cost among the merge candidate blocks as a final merge candidate block.

[0153] An embodiment of the present specification provides various embodiments of merging candidate blocks that constitute the merging candidate list.

[0154] The merge candidate list may include, for example, five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate may be used. As a specific example, in the case of spatial merge candidates, the blocks shown in FIG. 12 may be used as spatial merge candidates.

[0155] FIG. 13 is a flowchart illustrating a method for constructing a merge candidate list according to an embodiment of the present invention.

[0156] Referring to FIG. 13, a coding apparatus (encoder / decoder) searches for spatially neighboring blocks of a current block and inserts derived spatial merge candidates into a merge candidate list (S1301). For example, the spatial neighboring blocks may include a lower left corner neighboring block, a left neighboring block, a right upper corner neighboring block, an upper neighboring block, and an upper left corner neighboring block of the current block. However, this is merely an example, and in addition to the above-mentioned spatial neighboring blocks, additional neighboring blocks such as a right neighboring block, a lower neighboring block, and a lower right neighboring block may be further used as the spatial neighboring blocks. The coding apparatus may search the spatial neighboring blocks based on a priority order to detect available blocks, and derive motion information of the detected blocks as the spatial merge candidates. For example, the encoder and decoder may search the five blocks shown in FIG. 12 in the order of A1, B1, B0, A0, and B2, and sequentially index available candidates to form a merge candidate list.

[0157] The coding apparatus searches for temporal neighboring blocks of the current block and inserts derived temporal merging candidates into the merging candidate list (S1302). The temporal neighboring blocks may be located on a reference picture that is a picture different from the current picture in which the current block is located. The reference picture in which the temporal neighboring blocks are located may be referred to as a collocated picture or a col picture. The temporal neighboring blocks may be searched for in the order of a lower right corner neighboring block and a lower right center block of a co-located block of the current block on the col picture.

[0158] Meanwhile, when motion data compression is applied, specific motion information can be stored as representative motion information for each fixed storage unit in the col picture. In this case, it is not necessary to store motion information of all blocks in the fixed storage unit, and thus a motion data compression effect can be obtained. In this case, the fixed storage unit may be predetermined, for example, in units of 16x16 samples or 8x8 samples, or size information of the fixed storage unit may be signaled from the encoder to the decoder. When the motion data compression is applied, the motion information of the temporally neighboring blocks may be replaced with representative motion information of the fixed storage unit in which the temporally neighboring blocks are located.

[0159] That is, in this case, in terms of implementation, the temporal merge candidate may be derived based on motion information of a prediction block that is arithmetically shifted right by a certain value based on the coordinates (upper left sample position) of the temporal neighboring block, rather than the prediction block located at the coordinates of the temporal neighboring block, and then arithmetically shifted left. For example, when the certain storage unit is a 2nx2 sample unit, if the coordinates of the temporal neighboring block are (xTnb, yTnb), the motion information of a prediction block located at the modified position ((xTnb >> n)<< n), (yTnb >> n)<< n)) is used for the temporal merge candidate.

[0160] Specifically, for example, when the constant storage unit is a 16x16 sample unit, and the coordinates of the temporal neighboring blocks are (xTnb, yTnb), the motion information of the predicted block located at the modified position ((xTnb >> 4) << 4), (yTnb >> 4) << 4) is used for the temporal merge candidate. Or, for example, when the constant storage unit is an 8x8 sample unit, and the coordinates of the temporal neighboring blocks are (xTnb, yTnb), the motion information of the predicted block located at the modified position ((xTnb >> 3) << 3), (yTnb >> 3) << 3) is used for the temporal merge candidate.

[0161] The coding apparatus may check whether the number of current merging candidates is smaller than the maximum number of merging candidates (S1303). The maximum number of merging candidates may be predefined or may be signaled from the encoder to the decoder. For example, the encoder may generate information regarding the maximum number of merging candidates, encode the information, and transmit the information to the decoder in the form of a bitstream. When the maximum number of merging candidates is full, the process of adding subsequent candidates may not proceed.

[0162] If the number of the current merge candidates is less than the maximum number of merge candidates as a result of the check, the coding apparatus inserts additional merge candidates into the merge candidate list (S1304). The additional merge candidates may include, for example, ATMVP, combined bi-predictive merge candidates (if the slice type of the current slice is type B), and / or zero vector merge candidates.

[0163] If the check results in that the number of the current merging candidates is not less than the maximum number of merging candidates, the coding apparatus may terminate construction of the merging candidate list. In this case, the encoder may select an optimal merging candidate from among the merging candidates constituting the merging candidate list based on a rate-distortion (RD) cost, and may signal selection information (e.g., merge index) indicating the selected merging candidate to a decoder. The decoder may select the optimal merging candidate based on the merging candidate list and the selection information.

[0164] As described above, the motion information of the selected merging candidate is used as the motion information of the current block, and the predictive sample of the current block may be derived based on the motion information of the current block. The encoder may derive a residual sample of the current block based on the predictive sample, and may signal residual information related to the residual sample to a decoder. As described above, the decoder may generate a reconstructed sample based on the residual sample derived based on the residual information and the predictive sample, and generate a reconstructed picture based on the reconstructed sample.

[0165] When skip mode is applied, the motion information of the current block can be derived in the same way as when merge mode is applied previously, except that when skip mode is applied, the residual signal of the block is omitted, and therefore the predicted samples are immediately used for the reconstructed samples.

[0166] MVP Mode

[0167] FIG. 14 is a flow chart illustrating a method for constructing a merge candidate list according to one embodiment of the present invention.

[0168] When the MVP (Motion Vector Prediction) mode is applied, a motion vector predictor (mvp) candidate list can be generated using the motion vectors of the restored spatially surrounding blocks (which may be the surrounding blocks described in 12 above) and / or the motion vectors corresponding to the temporally surrounding blocks (or Col blocks). That is, the motion vectors of the restored spatially surrounding blocks and / or the motion vectors corresponding to the temporally surrounding blocks are used as motion vector predictor candidates.

[0169] The prediction information may include selection information (e.g., MVP flag or MVP index) indicating an optimal motion vector predictor candidate selected from the motion vector predictor candidates included in the list. In this case, the prediction unit may select a motion vector predictor of a current block from the motion vector predictor candidates included in the motion vector candidate list using the selection information. The prediction unit of the encoding device may obtain a motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, encode the motion vector difference, and output the motion vector predictor in the form of a bitstream. That is, the MVD may be obtained by subtracting the motion vector predictor from the motion vector of the current block. In this case, the prediction unit of the decoding device may obtain the motion vector difference included in the prediction information, and derive the motion vector of the current block by adding the motion vector difference and the motion vector predictor. The prediction unit of the decoding device may obtain or induce a reference picture index indicating a reference picture from the prediction information. For example, the motion vector predictor candidate list may be configured as shown in FIG. 14.

[0170] Affine motion prediction

[0171] FIG. 15 shows examples of motion models according to an embodiment of the present invention.

[0172] Conventional video compression technologies (e.g., high efficiency video coding (HEVC)) use one motion vector to represent the motion of a coding block. Although a method using one motion vector for each block can represent the optimal motion for each block, it may not be the optimal motion for each pixel. Therefore, if the optimal motion vector can be determined for each pixel, coding efficiency can be improved. Then, an embodiment of the present invention describes a motion prediction method for encoding or decoding a video signal using a multi-motion model. In particular, a motion vector can be represented for each pixel or sub-block of a block using motion vectors of two to four control points, and such a prediction method using the motion vectors of multiple control points can be referred to as affine motion prediction, affine prediction, etc.

[0173] The affine motion model according to the embodiment of the present invention can express four motion models as shown in Fig. 15. The affine motion model expressing three motion (translation, scale, rotate) DMFs among the motions that the affine motion model can express is referred to as a similarity (or simplified) affine motion model, and in describing the embodiment of the present invention, a description will be given based on the similarity (or simplified) affine motion model for convenience of description, but the present invention is not limited thereto.

[0174] FIG. 16 shows example motion vectors of control points for affine motion prediction according to an embodiment of the present invention.

[0175] As shown in FIG. 16, affine motion prediction can determine a motion vector for a pixel position (or sub-block) including a block using two control point motion vector (CPMV) pairs, v_0 and v_1. At this time, a set of motion vectors can be referred to as an affine motion vector field (MVF). At this time, the affine motion vector field can be determined using the following Equation 1.

[0176]

number

[0177] In Equation 1, v_0 (v_0={v_0x, v_0y}) indicates the motion vector (CPMV0) of the first control point at the upper left position of the current block 1300, and v_1 (v_1={v_1x, v_1y}) indicates the motion vector (CPMV1) of the second control point at the upper right position of the current block 1300. In addition, w indicates the width of the current block 1300. v (v={v_x, v_y}) indicates the motion vector at the {x, y} position. A motion vector in sub-block (or pixel) unit can be derived using Equation 1. In one embodiment, the accuracy of the motion vector can be rounded to 1 / 16 accuracy.

[0178] FIG. 17 illustrates an example of motion vectors for each sub-block of a block to which affine motion prediction is applied according to an embodiment of the present invention.

[0179] 17, during encoding or decoding, an affine motion vector field (MVF) may be determined on a pixel-by-pixel or block-by-block basis, i.e., in affine motion prediction, a motion vector of a current block may be derived on a pixel-by-pixel or sub-block-by-sub-block basis.

[0180] When the affine motion vector field is determined in units of pixels, the motion vector is obtained based on the value of each pixel, and when it is determined in units of blocks, the motion vector of the block can be obtained based on the value of the center pixel of the block. In this document, it is assumed that the affine motion vector field (MVF) is determined in units of 4*4 blocks as shown in FIG. 17. However, this is for convenience of explanation and is not limited to the embodiment of the present invention. FIG. 17 shows an example in which a coding block is composed of 16*16 samples and the affine motion vector field (MVF) is determined in units of blocks of 4*4 size.

[0181] Affine motion prediction can include an affine merge mode (AF_MERGE) and an affine inter mode (AF_INTER). The AF_INTER mode can include an AF_4_INTER mode that uses a four parameter-based motion model and an AF_6_INTER mode that uses a six parameter-based motion model.

[0182] Affine merge mode

[0183] AF_MERGE determines a control point motion vector (CPMV) based on the affine motion model of neighboring blocks coded as affine motion prediction. Affine coded neighboring blocks in search order are used for AF_MERGE. The current block can be coded as AF_MERGE when one or more neighboring blocks are coded as affine motion prediction.

[0184] That is, when the affine merge mode is applied, the CPMV of the current block can be derived using the CPMV of the neighboring blocks. In this case, the CPMV of the neighboring blocks can be used as the CPMV of the current block as is, or the CPMV of the neighboring blocks can be modified based on the size of the neighboring blocks and the size of the current block, and then used as the CPMV of the current block.

[0185] FIG. 18 illustrates an example of neighboring blocks used for affine motion prediction in affine merge mode according to an embodiment of the present invention.

[0186] In the affine merge (AF_MERGE) mode, the encoder can perform the encoding as follows:

[0187] Step-1: Scan the neighboring blocks A to E (1810, 1820, 1830, 1840, 1850) of the current coding block 1800 in alphabetical order, and determine the block coded in the first affine prediction mode based on the scanning order as a candidate block for the affine merge (AF_MERGE).

[0188] Step 2: Determine an affine motion model using the control point motion vectors (CPMVs) of the determined candidate blocks.

[0189] Step-3: Based on the affine motion model of the candidate block, the control point motion vector (CPMV) of the current block 1800 is determined, and the MVF of the current block 1800 is determined.

[0190] FIG. 19 illustrates an example of a block on which affine motion prediction is performed using neighboring blocks to which affine motion prediction is applied according to an embodiment of the present invention.

[0191] For example, as shown in FIG. 19, when block A 1920 is coded in affine mode, after determining block A 1920 as a candidate block, an affine motion model is derived using the control point motion vectors (CPMVs) (e.g., v2, v3) of block A 1920, and then control point motion vectors (CPMVs) v0 and v1 of a current block 1900 can be determined. Based on the control point motion vectors (CPMVs) of the current block 1900, an affine motion vector field (MVF) of the current block 1900 can be determined and coded.

[0192] FIG. 20 is a diagram illustrating a method for generating a merge candidate list using neighboring affine-coded blocks according to an embodiment of the present invention.

[0193] 20, when affine merge candidates are used to determine a CPMV pair, candidates as shown in Fig. 20 are used. In Fig. 20, it is assumed that the scan order of the candidate list is set to A, B, C, D, E. However, the present invention is not limited to this, and various orders can be set in advance.

[0194] As an embodiment, if the number of candidates (hereinafter referred to as affine candidates) coded in affine mode (or affine prediction) used in the neighboring blocks (i.e., A, B, C, D, E) is 0, the affine merge mode of the current block may be skipped. If the number of affine candidates used is one (e.g., A), the motion model of the candidate is used to derive the motion vectors (CPMV_0 and CPMV_1) of the control points of the current block. In this case, an index indicating the candidate may not be required (or coded). If the number of affine candidates used is multiple, the two candidates in the scan order may be configured in a candidate list of AF_MERGE. In this case, candidate selection information such as an index indicating the selected candidate in the candidate list may be signaled. The selection information may be flag or index information, and may be designated as AF_MERGE_flag, AF_merge_idx, etc.

[0195] In the embodiment of the present specification, the motion compensation of the current block can be performed based on the size of the sub-blocks. In this case, the sub-block size of the affine block (i.e., the current block) is derived. If the width and height of the sub-blocks are all larger than four luma samples, the motion vector of each sub-block can be derived and DCT-IF based motion compensation (1 / 16 fel for luma and 1 / 32 for chroma) can be performed on the sub-blocks. Otherwise, enhanced bi-linear interpolation filter based motion compensation can be performed on the entire affine block.

[0196] In the embodiment of the present specification, when the merge / skip flag is true and the width and height of the CU are all greater than or equal to 8, an affine flag at the CU level is signaled via the bitstream indicating that the affine merge mode is to be used. When a CU is coded as AF_MERGE, a merge candidate index with a maximum value of '5' is signaled to specify which motion information candidate from the affine merge candidate list to use for the CU.

[0197] 21 and 22 are diagrams illustrating a method of constructing an affine merge candidate list using neighboring blocks coded using affine prediction according to an embodiment of the present invention.

[0198] Referring to FIG. 21, the affine merge candidate list is constructed as the next step.

[0199] 1) Insert model-based affine candidates

[0200] Model-based affine candidates means that the candidates are derived from valid surrounding reconstructed blocks coded in affine mode. As shown in Fig. 21, the scanning order of the candidate blocks is from left (A), top (b), top right (C) and bottom left (D) to top left (E).

[0201] When the neighboring lower-left block (A) is coded in the 6-parameter affine mode, it will get the motion vectors (v_4, v_5, v_6) for the upper-left corner, the upper-right corner, and the lower-left corner of the CU containing block (A). The motion vector (v_0, v_1, v_2) for the upper-left corner of the current block is calculated according to the motion vectors (v_4, v_5, and v_6) according to the 6-parameter affine model.

[0202] If the neighboring lower-left block (A) is coded in 4-parameter affine mode, it will get motion vectors (v_4, v_5) for the upper-left and upper-right corners of the CU containing block (A). The motion vector (v_0, v_1) for the upper-left corner of the current block is calculated according to the motion vectors (v_4, v_5) according to the 4-parameter affine model.

[0203] 2) Insert control point-based affine candidates

[0204] Referring to FIG. 21, the control point-based candidate means that the candidate is constructed by combining the motion information around each control point.

[0205] The motion information for a control point is first derived from the designated spatial and temporal neighboring blocks shown in Figure 21. CP_k (k = 1, 2, 3, 4) denotes the kth control point, and A, B, C, D, E, F, and G are spatial positions for predicting CP_k (k = 1, 2, 3), and H is the temporal position for predicting CP4.

[0206] The coordinates of CP_1, CP_2, CP_3 and CP_4 are (0,0), (W,0), (H,0) and (W,H), respectively, where W and H are the width and height of the current block.

[0207] The motion information for each control point is obtained according to the following priority:

[0208] For CP_1, the checking priority is A->B->C, if A is used, then A is used. Otherwise, if B is used, then B is used. If both A and B are not used, then C is used. If all three candidates are not used, the motion information of CP1 cannot be obtained.

[0209] For CP_2, the checking priority is E->D.

[0210] For CP_3, the checking priority is G->F.

[0211] For CP_4, H is used.

[0212] Second, a combination of the control points is used to construct a motion model.

[0213] The motion vectors of two control points are required to calculate the transformation parameters in the four-parameter affine model. The two control points can be selected from one of the following six combinations: {CP_1, CP_4}, {CP_2, CP_3}, {CP_1, CP_2}, {CP_2, CP_4}, {CP_1, CP_3}, {CP_3, CP_4}. For example, using CP_1 and CP_2 control points to construct a four-parameter affine motion model is denoted as "Affine(CP_1, CP_2)".

[0214] The motion vectors of three control points are required to calculate the transformation parameters in a six-parameter affine model. The three control points can be selected from one of the following four combinations: {CP_1, CP_2, CP_4}, {CP_1, CP_2, CP_3}, {CP_2, CP_3, CP_4}, {CP_1, CP_3, CP_4}. For example, using CP_1, CP_2 and CPv3 control points to construct a six-parameter affine motion model is denoted as "Affine(CP_1, CP_2, CP_3)".

[0215] Also, in the embodiment of the present specification, in the affine merge mode, if an affine merge candidate exists, it may always be considered as a six-parameter affine mode.

[0216] Affine inter mode

[0217] FIG. 23 shows an example of neighboring blocks used for affine motion prediction in an affine inter mode according to an embodiment of the present invention.

[0218] Referring to FIG. 23, the affine motion prediction may include an affine merge mode (AF_MERGE) and an affine inter mode (AF_INTER). In the affine inter mode (AF_INTER), after determining the motion vector prediction (CPMVP) and CPMV of two control points, a control point motion vector difference (CPMVD) corresponding to the difference may be transmitted from the encoder to the decoder. A specific encoding process of the affine inter mode (AF_INTER) may be the same as the following.

[0219] Step 1: Determine two CPMVP pair candidates

[0220] Step-1.1: Determine combinations of up to 12 CPMVP candidates (see Equation 2 below)

[0221]

number

[0222] In Equation 2, v_0 is the motion vector (CPMV0) at the upper left control point 2310 of the current block 2300, v_1 ​​is the motion vector (CPMV1) at the upper right control point 2311 of the current block 2300, v_2 is the motion vector (CPMV2) at the lower left control point 2312 of the current block 2300, v_A is the motion vector of the neighboring block A 2320 adjacent to the upper left side of the upper left control point 2310 of the current block 2300, v_B is the motion vector of the neighboring block B 2322 adjacent to the upper side of the upper left control point 2310 of the current block 2300, and vC is the motion vector of the neighboring block B 2322 adjacent to the upper side of the upper left control point 2310 of the current block 2300. v_F indicates the motion vector of peripheral block F 2330 adjacent to the left of the lower left control point 2312 of the current block 2300; and v_G indicates the motion vector of peripheral block G 2332 adjacent to the left of the lower left control point 2312 of the current block 2300.

[0223] Step-1.2: Among the combinations of CPMVP candidates, sort by the smallest difference value (DV) and use the top two candidates (see Equation 3 below).

[0224]

number

[0225] v_0x is the x-axis element of the motion vector (V0 or CPMV0) of the top-left control point 2310 of the current block 2300, v_1x is the x-axis element of the motion vector (V1 or CPMV1) of the top-right control point 2311 of the current block 2300, v_2x is the x-axis element of the motion vector (V_2 or CPMV_2) of the bottom-left control point 2312 of the current block 2300, and v_0y is the x-axis element of the motion vector (V_1 or CPMV_2) of the top-left control point 2310 of the current block 2300. v_1y is the y-axis element of the motion vector (V_0 or CPMV_0), v_1y is the y-axis element of the motion vector (V_1 or CPMV_1) of the upper right control point 2311 of the current block 2300, v_2y is the y-axis element of the motion vector (V_2 or CPMV_2) of the lower left control point 2312 of the current block 2300, w indicates the width of the current block 2300, and h indicates the height of the current block 2300.

[0226] Step-2: If the number of control point motion vector predictor (CPMVP) pair candidates is less than 2, use the AMVP candidate list.

[0227] Step-3: For each of the two candidates, determine the control point motion vector predictor (CPMVP), compare the RD cost, and optimally select the candidate with the smaller CPMVP.

[0228] Step-4: Transfer the index corresponding to the best candidate and the control point motion vector difference (CPMVD)

[0229] In the embodiment of this specification, AF_INTER provides a configuration process for CPMVP candidates. Similar to AMVP, the number of candidates is two, and an index indicating the position of the candidate list is signaled.

[0230] The procedure for setting the CPMVP candidate list is as follows:

[0231] 1) Scan the surrounding blocks to check if it can be coded as affine motion prediction. If the scanned block is coded as affine prediction, derive the motion vector pair for the current block from the affine motion models of the scanned surrounding blocks until the number of candidates is 2.

[0232] 2) If the number of candidates is less than 2, perform the candidate construction process. Also, in this embodiment of the present invention, a 4-parameter (2-control point) affine inter mode is used to predict the zoom-in / out and rotation motion models and content. As shown in Figure 16, the affine motion field of a block is described by the motion vectors of two control points.

[0233] The motion vector field (MVF) of a block is described by Equation 1 above.

[0234] In the prior art, advanced motion vector prediction (AMVP) mode is required to signal motion vector prediction (MVP) index and motion vector differences (MVDs). When AMVP mode is applied to the present invention, affine_flag is signaled to indicate whether affine prediction is used. If affine prediction is applied, the syntax of inter_dir, ref_idx, mvp_index, and two MVDs (mvd_x and mvd_y) is signaled. An affine MVP pair candidate list is generated, which includes two affine MVP pairs. The signaled mvp_index is used to select one of them. The affine MVP pair is generated by two types of affine MVP candidates. One is a spatial inherited affine candidate, and the other is a corner derived affine candidate. If the surrounding CU is coded in affine mode, a spatial inherited affine candidate can be generated. The affine motion model of the peripherally affine coded block is used to generate a two-control-point MVP pair of motion vectors. The MV of the two-control-point MVP pair of the spatial inheritance affine candidate is derived by using the following formula:

[0235]

number

[0236]

number

[0237] If V_B0, V_B1, and V_B2 can be replaced by the upper left MV, upper right MV, and lower left MV of any reference / surrounding CU, (posCurCU_X, posCurCU_Y) is the position of the upper left sample of the current CU for the upper left sample of the frame, and (posRefCU_X, posRefCU_Y) is the position of the upper left sample of the reference / surrounding CU for the upper left sample of the frame.

[0238]

number

[0239]

number

[0240] FIG. 24 illustrates an example of neighboring blocks used for affine motion prediction in affine inter mode according to an embodiment of the present invention.

[0241] Referring to FIG. 24, if the number of MVP pairs is less than two, corner derived affine candidates are used. The surrounding motion vectors are used to derive affine MVP pairs as shown in FIG. 24. For the first corner derived affine candidate, the first available MV in set A (A0, A1 and A2) and the first available MV in set B (B0 and B1) are used to construct the first MVP pair. For the second corner derived affine candidate, the first available MV in set A and the first available MV in set C (C0 and C1) are used to calculate the MV of the top right side control point. The first available MV in set A and the calculated top right side control point MV are the second MVP pair.

[0242] In an embodiment of the present invention, two candidate sets including two (three) candidates {mv_0, mv_1} ({mv_0, mv_1, mv_2) are used to predict two (three) control points of an affine motion model. Given the motion vector difference (mvd_0, mvd_1, mvd_2) and the control points, they are calculated by using the following formula:

[0243]

number

[0244] 25 and 26 are diagrams illustrating a method of deriving motion vector candidates using motion information of neighboring blocks in an affine inter mode according to an embodiment of the present invention.

[0245] The affine candidate list extends affine motion from spatially neighboring blocks (extrapolated affine candidates) and is appended by a combination of motion vectors from spatially neighboring blocks (virtual affine candidates). The candidate set is set as follows:

[0246] 1. Up to two different affine MV predictor sets are derived from the affine motion of neighboring blocks. Neighboring blocks A0, A1, B0, B1, and B2 are identified as shown in Figure 25. If the neighboring block is coded by an affine motion model and its reference frame is the same as that of the current block, two (for a four-parameter affine model) or three (for a six-parameter affine model) control points of the current block are derived from the affine model of the neighboring blocks.

[0247] 2. Figure 29 shows the neighboring blocks used to generate the virtual affine candidate set. The neighboring MVs are divided into three groups: S_0 = {mv_A, mv_B, mv_C}, S_1 = {mv_D, mv_E}, S_2 = {mv_F, mv_G}. mv_0 is the first MV in S0 that references the same reference picture as the current block. mv_2 is the first MV in S1 that references the same reference picture as the current block.

[0248] If mv_0 and mv_1 are given, mv_2 can be derived by the following equation 9.

[0249]

number

[0250] In formula 9, the current block size is WxH.

[0251] If only mv_0 and mv_2 are given, mv_1 can be derived by Equation 10 below.

[0252]

number

[0253] In one embodiment of the present invention, affine inter prediction may be performed based on the following sequence:

[0254] Input: Affine motion parameters, reference picture samples

[0255] Output: Predicted block for CU

[0256] process

[0257] - Induce the size of sub-blocks of affine blocks

[0258] - If the sub-block width and width mode are greater than 4 luma samples,

[0259] - For each sub-block

[0260] - Induce sub-block motion vectors

[0261] - DCT-IF based motion compensation (1 / 16 pel for luma, 1 / 32 pel for chroma) is invoked for sub-blocks

[0262] - Otherwise, an enhanced bi-linear interpolation filter based compensation is invoked for the entire affine block.

[0263] Also, in one embodiment of the present specification, if the merge / skip flag is false and the width and width of the CU are 8 or more, an affine flag is signaled at the CU level to indicate whether the affine inter mode is used. If a CU is coded as an affine inter mode, a model flag is signaled to indicate whether a 4-parameter or 6-parameter affine model is applied to the CU. If the model flag is true, then the AF_6_INTER mode (6-parameter affine model) is applied and three MVDs are parsed, otherwise the AF_4_INTER mode (4-parameter affine model) is applied and two MVDs are parsed.

[0264] In AF_4_INTER mode, similar to the affine merge mode, extrapolated motion vector pairs from neighboring blocks coded in affine mode are generated and inserted into the candidate list first.

[0265] Then, if the size of the candidate list is less than 4, candidates with motion vector pairs {(v_0, v_1)| v0 = {v_A, v_B, v_c}, v_1 ​​= {v_D, v_E}} are generated by using neighboring blocks. As shown in FIG. 22, v_0 is selected from the motion vectors of blocks A, B, and C. The motion vectors from the neighboring blocks are scaled based on the relationship between the reference list and the POC of the reference of the neighboring block, the POC of the reference of the current CU, and the current CU. Then, the approach of selecting v_1 from the neighboring blocks D and E is similar. If the candidate list is greater than 4, the candidates are preferentially ordered based on the consistency of the neighboring motion vectors (similar to two motion vectors in a candidate pair) and the first four candidates are saved.

[0266] If the number of candidates in the list is less than four, the list is padded with motion vector pairs by duplicating each AMVP candidate.

[0267] In AF_6_INTER mode, similar to the affine merge mode, extrapolated motion vector triples from neighboring blocks coded in affine mode are generated and preferentially inserted into the candidate list.

[0268] Hereafter, if the size of the candidate list is smaller than 4, candidates including motion vector triples {(v_0, v_1, v_2)| v0 = {v_A, v_B, v_c}, v1 = {v_D, v_E}, v2 = {v_G, v_H}} are generated using the neighboring blocks. As shown in FIG. 22, v_0 is selected from the motion vectors of blocks A, B, or C. The motion vectors from the neighboring blocks are scaled based on the relationship between the reference list and the POC of the reference of the neighboring block, the POC of the reference of the current CU, and the POC of the current CU. And the approach for selecting v_1 from the neighboring blocks D and E and selecting v_2 from F and G is similar. If the candidate list is larger than 4, the candidates are ordered based on the consistency of the neighboring motion vectors (similar to 2 motion vectors from 3 candidates) and the first 4 candidates are kept.

[0269] If the number of candidate lists is less than four, the list can be padded with motion vector triples constructed by duplicating each AMVP candidate.

[0270] After the CPMV of the current CU is derived, the MVF of the current CU is generated based on Equation 11 under a 4-parameter affine model or based on Equation 12 under a 6-parameter affine model, depending on the number of affine parameters.

[0271]

number

[0272]

number

[0273] Here, the sub-block size MxN is derived from Equation 13 below, and MvPre is the partial precision (1 / 16) of the motion vector.

[0274]

number

[0275] After being derived by Equation 12, M and N should be adjusted downwards if necessary to make them divisors of w and h. If M or N is less than 8, WIF is applied, otherwise sub-block based affine motion compensation is applied.

[0276] FIG. 27 illustrates an example of a method for deriving a sub-block-based affine motion vector field according to an embodiment of the present invention.

[0277] Referring to Figure 27, to derive a motion vector for each MxN sub-block, the motion vector of the center sample of each sub-block as shown in Figure 27 is calculated based on Equation 11 or 12, and rounded to 1 / 16 fractional accuracy. A SHVC upsampling interpolation filter is applied to generate a prediction for each sub-block using the derived motion vector.

[0278] An SHVC upsampling interpolation filter with the same filter length and normalization factor as the HEVC motion compensated interpolation filter is used as the motion compensated interpolation filter for additional fractional pel positions. The accuracy of the motion vectors for the chroma components is 1 / 32 sample, and the additional interpolation filter for the 1 / 32 pel fraction positions is derived by using the average of the filters of the two adjacent 1 / 16 pel fraction positions.

[0279] The AF_MERGE mode can be selected on the encoder side in the same way that a regular merge mode selection is performed: a candidate list is generated in priority order, and the candidate with the smallest RD-cost is selected for comparison with the RD-costs of the different inter modes. The result of the comparison is a decision whether AF_MERGE is applied or not.

[0280] For AF_4_INTER mode, the RD cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current affine CU. After the CPMVP of the current affine CU is determined, affine motion estimation is applied to obtain the control point motion vector (CPMV). Then, the difference between the CPMV and the CPMVP is determined.

[0281] On the encoder side, the AF_6_INTER mode is only checked if the AF_MERGE or AF_4_INTER modes are determined as the optimal mode in the previous mode selection stage.

[0282] In one embodiment of the present invention, the Affine Inter (Affine AMVP) mode can be implemented as follows:

[0283] 1) AFFINE_MERGE_IMPROVE: Instead of searching the first neighboring block that is in affine mode, the improvement tries to search the neighboring block with the largest coding unit size as an affine merge candidate.

[0284] 2) AFFINE_AMVL_IMPROVE: Add neighboring blocks that are in affine mode to the affine AMVP candidate list similar to the regular AMVP procedure.

[0285] The detailed process of generating an affine AMVP candidate list is as follows.

[0286] First, it is checked whether the bottom left neighboring block has the same reference index as the current reference index using an affine motion model. If not, the left neighboring block is checked in the same manner. If not, it is checked whether the bottom left neighboring block has a different reference index using an affine motion model. If it does, the scaled affine motion vector is added to the reference picture list. If not, the left neighboring block is checked in the same manner.

[0287] Second, the right-side upper neighbor, the top neighbor, and the left-side upper neighbor are identified in the same manner.

[0288] After the above process, if two candidates are found, the operation of generating the affine AMVP candidate list is completed. If two candidates are not found, the original operation in the JEM software is executed to generate the affine AMVP candidate list.

[0289] 3) AFFINE_SIX_PARAM: Besides the 4-parameter affine motion model, a 6-parameter affine motion model is added as an additional model.

[0290] A six-parameter affine motion model is derived via Equation 14 below.

[0291]

number

[0292] Since there are six parameters in the motion model described above, three motion vectors at the top left position MV_0, the top right position MV_1, and the bottom left position MV_2 are required to determine the model. The three motion vectors can be determined in a similar manner to the two motion vectors in the four parameter affine motion model. The affine model merge is always set as a six parameter affine motion model.

[0293] 4) AFFINE_CLIP_REMOVE: Removes the motion vector constraints for all affine motion vectors, letting the motion compensation process control the motion vector constraints itself.

[0294] Affine motion model

[0295] As described above, various affine motion models can be used or considered in Affine inter prediction. For example, the affine motion model can express four motions as shown in FIG. 15. An affine motion model expressing three motions (translation, scale, and rotation) among the motions that the affine motion model can express can be called a similarity (or simplified) affine motion model. Depending on which of the affine motion models is used, the number of CPMVs derived and / or a sample / sub-block unit MV derivation method of the current block may be different.

[0296] In one embodiment of the present invention, adaptive four and six parameter motion models are used. In AF_INTER, a six-parameter motion model is provided in addition to the four-parameter motion model present in JEM. The six-parameter affine motion model is described as follows in Equation 15:

[0297]

number

[0298] where coefficients a, b, c, d, e, and f are affine motion parameters, and (x, y) and (x', y') are coordinates of pixel positions before and after transformation of the affine motion model. To use the affine motion model of video coding, if CPMV0, CPMV1, and CPMV2 are MVs of CP0 (upper left), CP1 (upper right), and CP2 (lower left), Equation 16 can be expressed as follows:

[0299]

number

[0300] where CPMV_0 = {v_0x, v_0y}, CPMV_1 = {v_1x, v_1y}, CPMV_2 = {v_2x, v_2y}, and w and h are the width and height of the coding block, respectively. Equation 16 is the motion vector field (MVF) of the block.

[0301] A flag is parsed at the CU level to indicate whether a 4-parameter or 6-parameter affine motion model is used when a neighboring block is coded with affine prediction. If there are no neighboring blocks coded with affine prediction, the flag is omitted and the 4-parameter model is used for affine prediction. That is, the 6-parameter model is considered in the condition that one or more neighboring blocks are coded with an affine motion model. In relation to the number of CPMVDs, two and three CPMVDs are signaled for the 4-parameter and 6-parameter affine motion models, respectively.

[0302] Also, in one embodiment of the present invention, pattern-matched motion vector refinement can be used. In the pattern-matched motion vector derivation of JEM (PMMVD in the description of the JEM encoder, hereinafter abbreviated as PMVD), the decoder needs to evaluate several motion vectors (MVs) to determine the starting MV candidate for the CU-level search. In the sub-CU search, in addition to the optimal CU-level MV, several MV candidates are added. The decoder needs to evaluate such MV candidates to search for the optimal MV, which requires a lot of memory bandwidth. In the proposed pattern-matched motion vector refinement (PMVR), the concept of template matching and bilateral matching in PMVD in JEM is adopted. To indicate whether PMVR is available, one PMVR_flag is signaled when skip mode or merge mode is selected. To significantly reduce memory bandwidth requirements compared to PMVD, a candidate list of MVs is generated and the starting MV candidate index is explicitly signaled if PMVR is applied.

[0303] Using a merge candidate list generation process, a candidate list is generated, but excluding sub-CU merge candidates, e.g., affine candidates and ATMVP candidates.

[0304] For bilateral matching, only one uni-prediction MV candidate is included. A bi-prediction MV candidate is split into two uni-prediction MV candidates. Similar MV candidates (whose MV difference is less than a predefined threshold) are also removed. For CU-level search, diamond search MV refinement is performed starting from the signaled MV candidate.

[0305] Sub-CU-level searching is only available in bilateral matching merge mode. The search window of the sub-CU-level searching for all sub-CUs is the same as the search window of the CU-level searching. Therefore, no additional bandwidth is required in the sub-CU-level searching.

[0306] Template matching is also used to refine the MVP in the AMVP mode. In the AMVP mode, two MVPs are generated by using the HEVC MVP generation process, and one MVP index is signaled to select one of them. The selected MVP is further refined by using template matching with the PMVR. When adaptive motion vector resolution (AMVR) is applied, the MVP is rounded to a suitable accuracy before template matching refinement. Such a refinement process is named pattern-matched motion vector predictor refinement (PMVPR). In the remainder of this document, PMVR includes template matching PMVR, bidirectional matching PMVR, and PMVPR, unless otherwise defined.

[0307] To reduce memory bandwidth requirements, PMVR is disabled for 4x4, 4x8, and 8x4 CUs. Due to the additional memory bandwidth requirement reduction, the search range for {template matching, bidirectional matching} for CU regions equal to 64 can be reduced to {±2, ±4}, and the search range for {template matching, bidirectional matching} for CU regions greater than 64 can be reduced to {±6, ±8}. By using all the above methods described in the PMVR section of this document, the memory bandwidth required compared to the worst case in HEVC is reduced from 45.9x with PMVD in JEM-7.0 to 3.1x with PMVR.

[0308] Affine used in non-QT block

[0309] FIG. 28 exemplarily illustrates a method of generating a prediction block and a motion vector in inter prediction using an affine motion model according to an embodiment of the present invention.

[0310] 28, a formula for deriving a motion vector when an affine motion model is applied can be seen. The motion vector can be derived based on the following Formula 17.

[0311]

number

[0312] Here, v_x denotes the x component of the sample-by-sample motion vector of the (x, y) coordinate sample in the current block, and v_y denotes the y component of the sample-by-sample motion vector of the (x, y) coordinate sample in the current block.

[0313] That is, (v_x, v_y) is the sample-based motion vector for the sample of the (x, y) coordinate. Here, a, b, c, d, e, and f indicate parameters of a formula for deriving a sample-based motion vector (motion information) of the (x, y) coordinate from a control point (CP) of the current block. The CP may also be expressed as a steering pixel. The parameters may be derived from the motion information of the CP of each PU transmitted in units of a PU. The formula for deriving the sample-based motion vector derived from the motion information of the CP described above may be applied to each sample of a block, and may be derived to the position of a sample in a reference image based on the relative positions of the x and y axes of each sample. The sample-based motion vector may be derived differently depending on the size of the block according to the QTBT (TT) block division structure, asymmetric or symmetric type, block position, etc. A specific embodiment of this will be described with reference to FIG. 29 to FIG. 38.

[0314] FIG. 29 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points according to an embodiment of the present invention.

[0315] Referring to FIG. 29, the description will be given assuming that the current block is a 2Nx2N block. For example, the motion vector of the top left sample in the current block may be v_0. In addition, the motion vectors of each CP may be v_1 and v_2 by using samples of the current block and the neighboring blocks as CPs. That is, when the width and height of the current block are S and the coordinates of the top-left sample position of the current block are (xp, yp), the coordinates of CP0 among the CPs may be (xp, yp), the coordinates of CP1 may be (xp + S, yp), and the coordinates of CP2 may be (xp, yp + S). The motion vector of CP0 may be v_0, the motion vector of CP1 may be v_1, and the motion vector of CP2 may be v_2. The motion vector of the CP may be used to derive the sample-based motion vector. The sample-based motion vector may be derived according to Equation 18 as follows.

[0316]

number

[0317] Here, v_x and v_y respectively denote the x and y components of the motion vector of the sample at (x, y) coordinate in the current block, v_x0 and v_y0 respectively denote the x and y components of the motion vector v_0 of the CP0, v_x1 and v_y1 respectively denote the x and y components of the motion vector v_1 of the CP1, and v_x2 and v_y2 respectively denote the x and y components of the motion vector v_2 of the CP2. A motion vector can be derived for each sample in the current block based on its relative position in the current block by the equation for deriving the sample-based motion vector, such as Equation 18, described above.

[0318] FIG. 30 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention.

[0319] FIG. 30 exemplarily illustrates CPs of a block divided into Nx2N. A formula for deriving a motion vector in a sample unit in a current block can be derived through the same method as in the case of the partition type 2Nx2N described above. In the process of deriving the formula, a width value that fits the shape of the current block can be used. In order to derive the motion vector in a sample unit, three CPs can be derived, and the positions of the CPs can be adjusted as shown in FIG. 30. That is, when the width and height of the current block are S / 2 and S, respectively, and the coordinates of the top-left sample position of the current block are (xp, yp), the coordinates of CP0 among the CPs can be (xp, yp), the coordinates of CP1 can be (xp +S / 2, yp), and the coordinates of CP2 can be (xp, yp+S). The motion vector in a sample unit can be derived based on the following Formula 19.

[0320]

number

[0321] Here, vx and vy respectively indicate the x and y components of the motion vector of the sample of (x, y) coordinates in the current block, v_x0 and v_y0 respectively indicate the x and y components of the motion vector v_0 of the CP0, v_x1 and v_y1 respectively indicate the x and y components of the motion vector v_1 of the CP1, and v_x2 and v_y2 respectively indicate the x and y components of the motion vector v_2 of the CP2. Equation 3 shows an equation for deriving a motion vector in a sample unit considering that the width of the current block is S / 2. A motion vector can be derived for each sample in the current block partitioned from a CU based on the partition type Nx2N according to the equation for deriving the motion vector in a sample unit as shown in Equation 19 above, based on the relative position in the current block.

[0322] FIG. 31 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention.

[0323] FIG. 31 exemplarily illustrates a block divided based on a partitioning type 2NxN. In order to derive the sample-based motion vector, three CPs can be derived, and the positions of the CPs can be adjusted as shown in FIG. 31 to adjust the height to S / 2 according to the shape of the current block shown in FIG. 31. That is, when the width and height of the current block are S and S / 2, respectively, and the coordinates of the top-left sample position of the current block are (xp, yp), the coordinates of CP0 among the CPs can be (xp, yp), the coordinates of CP1 can be (xp+S, yp), and the coordinates of CP2 can be (xp, yp+S / 2). The sample-based motion vector can be derived based on Equation 20 as follows.

[0324]

number

[0325] Here, v_x, v_y respectively indicate the x and y components of the motion vector of the sample of (x, y) coordinates in the current block, v_x0, v_y0 respectively indicate the x and y components of the motion vector v_0 of the CP0, v_x1, v_y1 respectively indicate the x and y components of the motion vector v_1 of the CP1, and v_x2, v_y2 respectively indicate the x and y components of the motion vector v_2 of the CP2. Equation 4 shows a formula for deriving a motion vector in a sample unit considering that the height of the current block is S / 2. For each sample in the current block partitioned from a CU based on the partitioning type 2NxN according to the formula for deriving the motion vector in a sample unit, such as Equation 4.18, a motion vector can be derived based on the relative position in the current block.

[0326] 32 to 38 are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention.

[0327] FIG. 32 exemplarily illustrates an asymmetric CP of a current block. As shown in FIG. 32, the width and height of the asymmetric current block may be W and H, respectively. To derive the sample-based motion vector, three CPs of each current block may be derived, and the coordinates of the CPs may be adjusted based on the width and height along the shape of the current block, as shown in FIG. 32. That is, when the width and height of the current block are W and H, and the coordinates of the top-left sample position of each current block are (xp, yp), the coordinates of CP0 among the CPs may be set to (xp, yp), the coordinates of CP1 may be (xp+W, yp), and the coordinates of CP2 may be (xp, yp+H). In this case, the sample-based motion vector within the current block may be derived according to Equation 21 as follows.

[0328]

number

[0329] Here, v_x and v_y respectively represent the x and y components of the motion vector of the sample at (x, y) coordinate in the current block, v_x0 and v_y0 respectively represent the x and y components of the motion vector v_0 of the CP0, v_x1 and v_y1 respectively represent the x and y components of the motion vector v_1 of the CP1, and v_x2 and v_y2 respectively represent the x and y components of the motion vector v_2 of the CP2. Equation 21 shows an equation for deriving an asymmetric, sample-based motion vector taking into account the width and height of the current block.

[0330] Meanwhile, according to the present invention, in order to reduce the amount of data of CP motion information indicated in block units, at least one CP motion information prediction candidate may be selected based on motion information of neighboring blocks or neighboring samples of a current block. The motion information prediction candidate may be called an affine motion information candidate or an affine motion vector candidate. The affine motion information candidate may include, for example, the contents disclosed in Figures 33 to 38.

[0331] ATMVP simplification

[0332] ATMVP (Advanced Temporal Motion Vector Prediction) is an inter prediction using a temporal motion information candidate, and may indicate an inter prediction using a motion information candidate proposed to improve the existing TMVP. In the embodiments of the present specification, the ATMVP is not limited to the name, and may be referred to as a sub-block temporal merge candidate, a sub-block-based temporal merge candidate, a sub-block temporal motion vector predictor, a sub-block-based temporal motion vector predictor, etc.

[0333] Specifically, TMVP, which uses the motion vector of the right-bottom block of the current block or the colPB at the center position of the current block as a temporal motion information candidate, may not reflect the motion within the screen. On the other hand, when the ATMVP is applied, the motion vector of the colPB at the position indicated by the motion vector of the surrounding block is used as the ATMVP candidate.

[0334] As an example of application of the ATMVP, the motion vector (temporal vector) of the spatial neighbor block that is available can be found first while checking the merge candidates in the order of configuration, and then the position pointed to by the temporal vector in the reference picture can be derived as a col-PB (ATMVP candidate). In addition, the motion vector of the corresponding block can be used in each sub-block unit using the temporal vector. In this case, if there is no MV in a specific sub-block, the MV of the block located at the center of the corresponding block is used as the MV of the unavailable sub-block and stored as a representative MV.

[0335] In addition, various ATMVP simplification schemes have been proposed that use the ATMVP to further reduce memory usage.

[0336] In one embodiment, the reference pictures (specified in the slice segment header) for finding the reference block are restricted to the collocated pictures, so that memory usage can be reduced. As an example, a maximum of four reference pictures are used. To find the reference block, a temporal vector can be derived from one of the spatial candidates according to the scan order. When the current candidate has the same reference picture as the collocated picture, the search process can be terminated.

[0337] In one embodiment, modifications of the ATMVP and / or STMVP sub-block merge modes can be applied.

[0338] 1. Signaling is added at the picture / slice-level so that different sub-block sizes are used for ATMVP / STMVP-based motion information derivation.

[0339] 2. A simplified ATMVP-based motion information derivation is used by allowing one same-position picture to be used.

[0340] In one embodiment, it is proposed to support picture / slice-level adaptation of the granularity of the motion fields derived by ATMVP and STMVP modes. Specifically, the default value of the sub-block size used to derive the ATMVP and STMVP motion parameters in a slice can be signaled in a sequence parameter set (sps) that points to the sps. Additionally, one flag is signaled in the slice's segment header. If this flag is 0, it indicates that the default sub-block size as specified in sps is used for ATMVP / STMVP based motion derivation in the current slice. Otherwise (i.e., this flag is set to 1), another syntax element can be signaled in the slice's segment header to specify the ATMVP / STMVP sub-block size to use for the slice.

[0341] In one embodiment, a simple encoder method is used to determine the ATMVP / STMVP sub-block size of the current picture. One of two ATMVP / STMVP sub-block sizes, 4 and 8, is selected based on the average size of ATMVP / STMVP blocks from the last coded picture in the same temporal layer. Assume that the previously coded picture is the i-th picture in the k-th temporal layer and contains N CUs coded by ATMVP and STMVP modes. Assume also that the sizes of the CUs are S_0, S_1, ..., S_(N-1). The average size of ATMVP / STMVP CUs is calculated as σ^k=(Σ_(i=0)^(N-1)S_i) / N. Then, when coding the (i +1)-th picture in the same k-th temporal layer, its ATMVP / STMVP sub-block size g_(i +1)^k can be determined based on the following Equation 22.

[0342]

number

[0343] In one embodiment, for the first picture in each temporal layer, the size of ATMVP / STMVP sub-block 7 may always be set to four.

[0344] In one embodiment, the ATMVP can be derived using the following two steps.

[0345] (1) If a neighboring CU is available and the MV of this CU is different from the MV from the existing candidate list, the encoder / decoder can add motion vectors from spatially neighboring CUs in the order of A1, B1, B0, and A0 as shown in the previously described Figure 12. The number of available spatial candidates can be represented as N0.

[0346] (2) The first MV candidate from the N0 spatial candidates is used to determine (by adding the displacement due to MV) the location to fetch the same position picture and the motion of each sub-block. If N0 is 0, the same location with the same position picture and zero motion signaled in the slice header is used to fetch the motion of each sub-block.

[0347] The ATMVP different CU co-located pictures may not always be the same when multiple reference pictures are used. Having ATMVP derived different co-located pictures for different CUs in the current picture means that motion fields of multiple reference pictures need to be fetched, which is undesirable due to increased memory bandwidth. Therefore, in one embodiment, a simplified design is provided that uses the same co-located pictures as ATMVP derived co-located pictures as signaled in the slice header in HEVC. At the block level, if the reference picture of the surrounding block A is different from this co-located picture, the MV of block A is scaled using the HEVC temporal MV scaling method, and the scaled MV of block A is used for ATMVP.

[0348] The motion vector used to fetch the motion field at the co-located picture Rcol is denoted as MVcol. To minimize the impact of MV scaling, the MV from the spatial candidate list used to derive MVcol is selected in this answer in the following way: If the reference picture of the candidate MV is a co-located picture, this MV is selected and used as MVcol without any scaling. Otherwise, the MV with the reference picture closest to the co-located picture is selected to derive MVcol with scaling.

[0349] HMVP (History-based Motion Vector Prediction) general

[0350] Generally, video compression techniques exploit spatial and temporal redundancy as two main techniques. For example, High Efficiency Video Coding (HEVC) and VVC both use two motion compression techniques on the basis of inter coding. One is merge motion and the other is advanced motion vector prediction (AMVP). Various modifications have been discussed to improve these two prediction modes. These include increasing the number of candidates, exploring more spatially extended candidates, and testing temporal candidates at non-traditional positions. These two techniques primarily build a list of possible candidates, minimize the rate distortion cost, and signal the selected candidate from the bitstream.

[0351] In particular, in recent video compression technology, HMVP (History-based Motion Vector Prediction) is being discussed, which stores motion information of previously encoded blocks and uses the stored motion information to predict the motion of blocks to be encoded in the future. Such HMVP can be added to the merge list (or merge candidate list) or the AMVP list (or AMVP candidate list).

[0352] The decoder maintains a look-up table (LUT) that operates in a first in first out (FIFO) system (or method) for HMVP. In this specification, the LUT is not limited to the name and may be referred to as a table, HMVP table, HMVP candidate table, buffer, HMVP buffer, HMVP candidate buffer, HMVP list, HMVP candidate list, etc. Specifically, when a non-affine prediction unit (PU) (or coding unit (CU)) is decoded, its motion information is stored in the LUT, and the decoder decodes the next PU. At this time, the stored motion information may include motion vectors in the x (horizontal) and y (vertical) directions, reference index information, mode information, etc.

[0353] The decoder can maintain a LUT in which motion information of progressively decoded non-affine candidates is stored. The size of the LUT can be limited to a predefined number of S candidates. In one embodiment, the LUT can be reset from the start of a slice, the start of a CTU row, or the start of a CTU.

[0354] HMVP can be applied in both merge and AMVP modes. A merge list can have B candidates, and an AMVP list can have two candidates. In conventional video compression technology, a merge list consists of the following candidates: i) spatial candidates, ii) temporal candidates, iii) bi-predictive (Bi-Pred) candidates, and iv) zero motion candidates. Recently, a method in which ATMVP (advanced motion vector prediction) is considered as an additional candidate has been discussed. As an example, an ATMVP candidate can be inserted into the merge list before a temporal candidate. Candidates in the merge list are added to the merge list until the maximum merge list size is reached. Duplicate candidates may not be added to the merge list.

[0355] An AMVP list can be populated with two candidates. As an example, one of the two candidates can be selected from the available spatial candidates and the second candidate can be selected from the temporal candidates, and a zero motion vector candidate can be added if the list is not filled.

[0356] HMVP is applied on a FIFO basis, with candidates leaving the table in the same order they were put into the LUT.

[0357] In one embodiment, when HMVP is applied to construct a merge list, an HMVP candidate can be inserted (or added) in the third position of the list as follows:

[0358] 1. Spatial Candidate

[0359] 2.Temporal Candidate

[0360] 3. Up to S HMVP Candidates for a LUT

[0361] 4. Combined Bi-Pred Candidate

[0362] 5.Zero Motion Vector Candidate

[0363] In one embodiment, when HMVP is applied to the construction of an AMVP list, HMVP can be inserted in the third position after the time candidate as follows:

[0364] 1. Spatial Candidate

[0365] 2. Temporal Candidate

[0366] 3. Up to K HMVP Candidates

[0367] 4.Zero Motion Vector Candidate

[0368] FIG. 39 is a flowchart illustrating a method for storing HMVP according to an embodiment of the present specification.

[0369] Referring to FIG. 39, the decoder decodes the current PU (or CU) (S3901).

[0370] The decoder checks whether the current PU is a block coded in non-affine mode (S3902). To facilitate the use of HMVP candidates, if the current PU is a block coded in affine mode, the decoder does not store the motion information of the current PU in a table.

[0371] If the current PU is a block coded in non-affine mode, the decoder stores (or updates) the motion information of the current PU in a table (S3903).

[0372] In the embodiment of the present specification, the HMVP table can be updated in two ways, i) unconstrained FIFO, ii) constraint FIFO. In electronic, duplicate information can exist, but no pruning process is applied. This helps to reduce the overall process complexity. The following description is given with reference to the figure below.

[0373] FIG. 40 is a diagram illustrating an HMVP table that operates in a non-limiting FIFO manner according to an embodiment of the present specification.

[0374] 40, candidates added to the table are added to the end (right side) of the table, while candidates removed from the table on a FIFO basis are located at the front end (left side, oldest candidate) of the table.

[0375] If the table at index L-1 (i.e., the edge) is not completely filled with a pre-defined maximum number of candidates, a new candidate is added without removing any candidates. On the other hand, if the table is already completely filled, i.e., the maximum number of candidates in the table is filled, the oldest front-most candidate in the table is removed and a new candidate is added.

[0376] FIG. 41 is a diagram for explaining an HMVP table that operates in a restrictive FIFO manner according to an embodiment of the present specification.

[0377] 41, when a restrictive FIFO is used, pruning is performed if adding a new candidate causes any duplication. As an example, if a candidate with duplicate motion information exists in the table, the duplicate candidate in the table can be removed and the motion information of the current candidate can be added.

[0378] Example 1

[0379] For HMVP candidates, in many cases, the recent history MVs may overlap with the motion information of spatial candidates (or spatial neighbor candidates). Therefore, in this embodiment, when adding HMVP candidates to the AMVP or merge list, we propose a method to set the addition order of the candidates to be different from the order of the HMVP LUT indexes.

[0380] According to the embodiments of the present specification, a candidate list can be efficiently constructed by adaptively adjusting HMVP candidates, thereby reducing the number of signaling bins used for binarization and improving coding efficiency.

[0381] That is, the HMVP candidates added to the merge list or AMVP list may not be restricted by index in the HMVP list. As an example, the following Table 2 illustrates a method for changing the order in which HMVP candidates are added to the AMVP or merge list.

[0382] [Table 2]

[0383] Referring to Table 2, as mentioned above, the most recently inserted HMVP candidate is likely to have the same motion information as the spatial candidate, so taking this into consideration, the addition order of the HMVP candidates can be predefined regardless of the HMVP index.

[0384] Also, in one embodiment, HMVP candidates can be added to the merge or AMVP list starting from the nth candidate in the list. Table 3 below illustrates a modified order for adding candidates to the AMVP or merge list.

[0385] [Table 3]

[0386] Referring to Table 3, HMVP candidates from the second index can be added to the merge list or AMVP list.

[0387] In one embodiment, information about the order of addition of HMVP candidates in a table (LUT) can be signaled from the encoder to the decoder. For example, such ordering information can be conveyed via a high level syntax (HLS). The high level syntax can be, for example, a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit, and / or a data header of other suitable syntax.

[0388] Table 4 below illustrates a high-level syntactic "syntax" structure to which the methods proposed herein can be applied.

[0389] [Table 4]

[0390] Referring to Table 4, set_HMVP_order_flag being 1 indicates that set_HMVP_order_flag is present in the slice header of a non-IDR picture from the CVS, and set_HMVP_order_flag being 0 indicates that set_HMVP_order_flag is not present in the slice header and adaptive HMVP is not used in the VCS.

[0391] Table 5 below illustrates a slice segment header syntax structure to which the methods proposed herein can be applied.

[0392] [Table 5]

[0393] Referring to Table 5, slice_HMVP_idx means the index of the candidate order to be used. For example, slice_HMVP_idx of 0 can represent a basic HMVP order such as 0, 1, 2, 3, etc. Similarly, an index value of 1 is used to represent an HMVP order of 3, 2, 1, 0.

[0394] Example 2

[0395] In an embodiment of the present specification, a method is proposed in which a long term list is used for motion example prediction in addition to the HMVP LUT. This can increase the number of HMVP candidates that are maintained. As an embodiment, a 2-HMVP table can be considered, where one stores general HMVP candidates and the other is used as a long term list that stores candidates that need to be further maintained.

[0396] The following is an example of how to initialize and configure a long-term list (or a long-term HMVP list).

[0397] After decoding the first CTU in a CTU row, one or more history MVs of subsequent CTUs may be added to the long-term HMVP LUT, which may not be used or updated until the next CTU row.

[0398] - At the start of the next CTU row, the long-term HMVP LUT is used to initialize the regular HMVP LUT, because the HMVP candidates of the CTU from the start of the CTU row can be more co-related than the history MVs at the end of the previous CTU row.

[0399] The process can be repeated.

[0400] FIG. 42 is a diagram illustrating an HMVP LUT and a long-term HMVP LUT according to an embodiment of the present specification.

[0401] Referring to Figure 42, there are two LUTs. One is HMVP LUT (or general HMVP LUT, short-term HMVP LUT) and the other is long-term HMVP LUT. When HMVP candidates are added to both merged AMVP lists, they can be added from HMVP LUT or long-term LUT as shown in Figure 42.

[0402] The use of the aforementioned long-term LUTs can be indicated through a higher level syntax by introducing new syntax elements, for example, the syntax elements can be present in the headers of a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit, and / or other suitable syntax data.

[0403] Example 3

[0404] In the embodiment of the present specification, a method is proposed to consider the flexibility for decoding when adding HMVP candidates to the HMVP LUT. The encoder / decoder can consider the decision of one or more characteristics of a PU (or CU).

[0405] In an embodiment, the encoder / decoder may consider the following when adding an HMVP candidate to the table. The encoder / decoder may consider characteristics such as the mode of the PU (e.g., merge mode, affine mode, AMVP mode, etc.) and / or the size of the block individually or in combination to add the candidate. In an embodiment, other characteristics may be considered in addition. For example, the merge type (e.g., spatial candidate or temporal candidate) for which the HMVP LUT is updated, whether it is a sub-PU, etc. may be considered as a candidate selection criterion. The above selection criteria may be determined to reduce overlap with previous history (or previous HMVP). For example, if a PU is coded in merge mode and the merge type is spatial merge, the decoder may not update the motion information of the PU to the HMVP LUT.

[0406] FIG. 43 illustrates an example method for updating an HMVP LUT according to an embodiment of the present disclosure.

[0407] Referring to FIG. 43, the encoder / decoder obtains motion information of a coded candidate (S4301).

[0408] The encoder / decoder evaluates whether to update the motion information of the candidate to the LUT based on a predefined decision criterion (S4302). The decision criterion may include at least one characteristic of the mode of the candidate (e.g., merge mode, affine mode, AMVP mode, etc.), the block size of the candidate, and / or the merge type of the candidate.

[0409] The encoder / decoder updates the LUT based on the decision criterion (S4303). That is, if the candidate satisfies a predefined decision criterion, the encoder / decoder can add motion information of the candidate to the LUT.

[0410] Example 4

[0411] In the embodiments of the present specification, we propose a restriction of the redundancy check for adding HMVP candidates to the merge list (or AMVP list). The restriction of the redundancy check can be realized in a number of different ways.

[0412] In one embodiment, the encoder / decoder may limit the number of pruning checks of a specific number of candidates in the merge list. The encoder / decoder may limit the number of pruning checks of candidates from the first candidate in the merge list to a specific number of candidates. For example, the encoder / decoder may perform a pruning process of candidates from the first candidate in the merge list to a specific number of candidates, and at this time, the number of HMVP candidates to be subject to the pruning check may be limited to a predefined number.

[0413] In addition, in one embodiment, the encoder / decoder may limit the pruning check by performing the pruning check on a specific type of merge candidates in the merge list. For example, the encoder / decoder may perform the pruning check only on spatial candidates in the merge list when adding an HMVP candidate. Or, for example, the encoder / decoder may perform the pruning check only on a portion of spatial candidates in the merge list when adding an HMVP candidate. The portion of spatial candidates may be predefined. For example, the portion of spatial candidates predefined may be at least one of a left adjacent spatial candidate and / or an upper adjacent spatial candidate. The embodiments of the present specification are not limited thereto, and other types may be combined to limit the targets of the pruning check.

[0414] FIG. 44 is a diagram illustrating a method for limiting the number of HMVP candidates that are subject to pruning checks according to an embodiment of the present specification.

[0415] Referring to FIG. 44, in an embodiment of the present specification, the number of HMVP candidates that are subject to pruning checks can be limited to M times.

[0416] The encoder / decoder can use the HMVP candidates to construct a merge list and check whether there is redundancy of motion information between the first M candidates in the HMVP LUT and the merge candidates in the merge list.

[0417] Alternatively, when the encoder / decoder adds the motion information of the currently decoded PU to the HMVP LUT, it may check whether there is redundancy between the first M candidates in the HMVP LUT and the motion information of the currently decoded PU.

[0418] FIG. 45 illustrates an example method for performing a pruning check in accordance with an embodiment of the present disclosure.

[0419] Referring to FIG. 45, the encoder / decoder obtains motion information of a decoded candidate and determines (or decodes) the number of pruning checks (S4501, S4502).

[0420] The encoder / decoder performs pruning checks based on the determined number of pruning checks (S4503).

[0421] In one embodiment, information related to the pruning check may be signaled through a higher level syntax in a manner similar to that of Tables 4 and 5 above. In this case, a syntax element transmitted from the encoder to the decoder may be included in any header to indicate the number of pruning checks that need to be performed. The higher level syntax may be, for example, a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit, and / or any other suitable syntax data header.

[0422] Example 5

[0423] In one embodiment of the present specification, we propose an efficient method for selecting HMVP candidates. When inserting a history motion vector candidate (i.e., an HMVP candidate) into a merge list (or an AMVP list), a pruning check can be performed to ensure that the HMVP candidate does not overlap with existing merge lists.

[0424] To perform a full redundancy check between an M-sized merge list and an N-sized history LUT, (M-1)xN checks are required. Thus, in the embodiments of the present specification, the number of HMVP candidates may depend on the existing parallel merging candidates. For example, the number of HMVP candidates may depend on the number of spatial candidates present in the merge list. Or, for example, the number of HMVP candidates may depend on the number of spatial and temporal candidates present in the merge list.

[0425] If there are more merge candidates in the merge list, the number of HVMP candidates for which pruning checks are performed can be reduced based on certain criteria (or rules) based on the number of merge candidates in the merge list and / or the number of HVMPs, thereby reducing the number of duplicate checks in the worst case.

[0426] For example, for a merge list with size (or length) 6, the merge list can contain up to 5 spaces or other merge candidates if the merge list is not full. Inserting an HMVP candidate from 6 HMVP lists can require 30 duplicate checks in the worst case.

[0427] In one embodiment, an example related to limiting the number of HMVPs to be checked is shown in the following Equation 23 and Table 6.

[0428]

number

[0429] [Table 6]

[0430] Referring to Table 6, by limiting the number of HMVPs subject to pruning checks to two, in the worst case, the number of duplicate checks for adding HMVPs can be reduced to 12 instead of 30.

[0431] Example 6

[0432] In one embodiment of the present specification, a method for constructing a merge list using history-based spatial temporal motion vector prediction (H-STMVP) is proposed. H-STMVP indicates a candidate derived by averaging two history-based spatial MVPs and TMVPs. The two spatial HMVPs can be obtained from an HMVP buffer, and the TMVP can be obtained from the current merge list. Here, the spatial candidate can be a candidate obtained from the last two coded MVs in decoding order before the current block.

[0433] For example, the last coded MV (referred to herein as MV_L), the penultimate coded MV (referred to herein as MV_(L-1)), and MV_TMVP are used to generate H-STMVP candidates to be inserted into the merge list.

[0434] When all three candidates mentioned above are used, the MV to be added to the merge list can be calculated by the following Equation 24.

[0435]

number

[0436] In one embodiment, if only two of the three candidates are available, then only the two candidates are averaged to generate the H-STMVP. Similarly, if only one candidate is available, then only the one candidate is used. If no candidates are available, then the H-STMVP is not used to construct the merge list.

[0437] In one embodiment of the present specification, a method for acquiring motion vectors of H-STMVP candidates using another method other than Equation 24 is proposed.

[0438] For example, instead of averaging three or more candidates at once, it may be computationally simpler to average the spatial candidates first and then use this result to re-average two candidates. An example of this is equivalent to the following formula:

[0439]

number

[0440] Alternatively, the average value can be obtained as follows:

[0441]

number

[0442]

number

[0443]

number

[0444] The encoder / decoder can first average two candidates and then finally average the resulting values ​​with the third candidate as in Equations 25-27. Or the encoder / decoder can give a higher importance / weight to the candidate, i.e., MV_L, by applying a shift operation by 2 as in Equation 28. Using the above-mentioned Equations 25-28, the average value can be derived using only shift operations and no division.

[0445] Example 7

[0446] In one embodiment of the present invention, we propose a method to use an arbitrary number (n) of spatial candidates in deriving H-STMVP, instead of the two history-based spatial candidates. These n candidates do not necessarily have to be in consecutive decoding order. They can be selected based on any or some rules.

[0447] Therefore, the above-mentioned Equation 24 can be expressed in a more general manner as Equation 29 below.

[0448]

number

[0449] In another embodiment, assuming five spatial candidates are used, the weights applied to the temporal candidates can be increased to minimize the impact of the increased spatial candidates for generating H-STMVP candidates and appropriately reflect the spatial and temporal candidates.

[0450] Therefore, to achieve this, the spatial candidates can be averaged together using the following Equation 30, and then the result can be used to average the MV_TMVP to achieve the aforementioned objective.

[0451]

number

[0452] Example 8

[0453] In one embodiment of the present specification, a method is proposed to add weights (or weighting factors) to motion vector candidates used to derive H-STMVP. In this case, the weights can be determined empirically, can be determined by considering the time distance to a fixed reference frame, or can be determined by considering the position in a history table. As an example, a new candidate can have more weight than a previous candidate.

[0454] That is, in this embodiment, the above-mentioned Equation 24 can be expressed as the following Equation 31.

[0455]

number

[0456] In this case, the weights may have the same value or may have unevenly distributed values.

[0457] Example 9

[0458] In one embodiment of the present invention, we propose a method for scaling the motion vectors used to derive H-STMVP candidates as a single reference picture.

[0459] FIG. 46 is a diagram illustrating a method for deriving H-STMVP candidates using motion vectors that refer to different reference pictures according to one embodiment of the present specification.

[0460] Referring to Figure 46, it is assumed that the MV_L, MV_L-1 and MV_TMVP candidates each refer to (or point to) different reference pictures. That is, Figure 46 shows that each candidate used to generate the H-STMVP candidates can have a different reference index and, as a result, a different reference frame.

[0461] Since frames with nearby reference frames may have a larger effect on the motion vectors of H-STMVP, the average of Equations 24 to 31 may result in unequal results. Therefore, for equal comparison and reflection, we propose a method of scaling all motion vectors to a single reference frame.

[0462] At this time, it can be performed as part of the RD optimization from the encoder to determine what single frame is best to use as a reference frame. As an example, the selected reference frame can be signaled in the slice header similar to the index of the TMVP array present in the slice header. For example, it may be possible to generate the reference frame to use using a fixed rule. Or, for example, it can be scaled to the first available reference frame from L0 or scaled with the current picture order count.

[0463] In one embodiment, to achieve the above-mentioned objectives, an encoder may transmit information about a single fixed picture to a decoder using a higher level syntax (HLS), which may be part of a sequence parameter set, a picture parameter set, a slice header, a coded tree unit, and / or a header of other data. For example, a higher level syntax structure such as Table 7 and / or Table 8 below may be defined.

[0464] [Table 7]

[0465] Referring to Table 7, set_HSTMVP_ref_pic_flag equal to 1 indicates that set_HSTMVP_idx is present in the slice header of a non-IDR picture from the CVS. set_HSTMVP_ref_pic_flag equal to 0 indicates that set_HSTMVP_idx is not present in the slice header.

[0466] [Table 8]

[0467] With reference to Table 8, slice_HMVP_idx specifies the reference index. As an example, the reference index can be selected for list L0.

[0468] Although the embodiments of the present specification described above have been described separately for the sake of convenience, the present invention is not limited to this. In other words, the embodiments described in the above-described embodiments 1 to 9 may be implemented independently, or one or more of the embodiments may be implemented in combination.

[0469] FIG. 47 is a flowchart illustrating a method for processing a video signal based on inter prediction according to an embodiment to which the present invention is applied.

[0470] Referring to Figure 47, for convenience of explanation, the description will focus on a decoder, but the present invention is not limited thereto, and the inter-prediction based video signal processing method according to the embodiments of this specification can be performed in the same way in both the encoder and the decoder.

[0471] The decoder constructs a first merge candidate list for a current block using spatial and temporal merge candidates for the current block (S4701).

[0472] The decoder constructs a second merging candidate list by adding history-based merging candidates indicative of motion information of previously coded blocks of the current block to the first merging candidate list (S4702).

[0473] The decoder obtains a merge index indicating a merge candidate in the second merging candidate list to be applied to the current block (S4703).

[0474] The decoder generates a prediction block for the current block using motion information of the merge candidate indicated by the merge index (S4704).

[0475] As described above, the history-based merge candidate can be added to the first merge candidate list if it has motion information that does not overlap with predefined merge candidates among the merge candidates included in the first merge candidate list.

[0476] As described above, the history-based merge candidate can be added to the first merge candidate list if it has motion information that does not overlap with a predefined number of merge candidates among the merge candidates included in the first merge candidate list.

[0477] As mentioned above, the history-based merge candidate may be added to the first merge candidate list if it has non-overlapping motion information with a particular spatial merge candidate included in the first merge candidate list.

[0478] As described above, the history-based merge candidates may be derived from a predefined number of candidates in a history-based buffer that stores the history-based merge candidates. For example, the predefined number may be two.

[0479] As described above, the step of constructing the second merge candidate list may further include the step of adding a history-based spatio-temporal merge candidate, and the history-based spatio-temporal candidate may be derived using the two history-based merge candidates and the temporal merge candidate.

[0480] As described above, the motion vector of the history-based spatio-temporal candidate can be derived as the average of the motion vectors of the two history-based merge candidates and the average of the motion vector of the temporal merge candidate.

[0481] 48 shows an example of a block diagram of an apparatus for processing a video signal according to an embodiment of the present specification. The video signal processing apparatus of FIG. 48 may correspond to the encoding apparatus 100 of FIG. 1 or the decoding apparatus 200 of FIG. 2.

[0482] An image processing device 4800 for processing an image signal includes a memory 4820 for storing an image signal, and a processor 4810 coupled to the memory for processing the image signal.

[0483] The processor 4810 according to the embodiment of the present specification may be configured with at least one processing circuit for processing a video signal, and may process the video signal by executing a command for encoding or decoding the video signal. That is, the processor 4810 may encode original video data or decode an encoded video signal by executing the encoding or decoding method described above.

[0484] FIG. 49 shows a video coding system to which the present invention is applied.

[0485] A video coding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device via a digital storage medium or a network in the form of a file or streaming.

[0486] The source device may include a video source, an encoding apparatus, and a transmitter. The receiving device may include a receiver, a decoding apparatus, and a renderer. The encoding apparatus may be called a video / video encoding apparatus, and the decoding apparatus may be called a video / video decoding apparatus. The transmitter may be included in the encoding apparatus. The receiver may be included in the decoding apparatus. The renderer may include a display unit, which may be a separate device or an external component.

[0487] A video source may acquire video / video through a video / video capture, synthesis, or generation process. A video source may include a video / video capture device and / or a video / video generation device. A video / video capture device may include, for example, one or more cameras, a video / video archive containing previously captured video / video, etc. A video / video generation device may include, for example, a computer, a tablet, a smartphone, etc., and may (electronically) generate video / video. For example, a virtual video / video may be generated through a computer, etc., in which case the video / video capture process may be substituted for the process in which the associated data is generated.

[0488] An encoding device can encode the input video / image. The encoding device can perform a series of steps such as prediction, transformation, quantization, etc. for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0489] The transmitting unit can transmit the encoded video / image information or data output in the form of a bitstream to a receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit can include elements for generating a media file through a predetermined file format and can include elements for transmission via a broadcasting / communication network. The receiving unit can extract the bitstream and transmit it to a decoding device.

[0490] The decoding device can decode the video / image by performing a series of steps such as inverse quantization, inverse transformation, prediction, etc. that correspond to the operations of the encoding device.

[0491] The renderer can render the decoded video / image. The rendered video / image can be displayed via a display unit.

[0492] FIG. 50 shows a structural diagram of a content streaming system as an embodiment to which the present invention is applied.

[0493] Referring to FIG. 50, a content streaming system to which the present invention is applied can broadly include an encoding server, a streaming server, a Web server, a media storage, a user device, and a multimedia input device.

[0494] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, video camera, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, video camera, etc. directly generates a bitstream, the encoding server can be omitted.

[0495] The bitstream may be generated by an encoding method or a bitstream generating method to which the present invention is applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0496] The streaming server transmits multimedia data to a user device via a web server based on a user request, and the web server acts as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server transfers the request to the streaming server, and the streaming server transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, and in this case, the control server controls commands / responses between devices in the content streaming system.

[0497] The streaming server may receive content from a media storage and / or an encoding server. For example, when content is received from the encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a certain period of time in order to provide a smooth streaming service.

[0498] Examples of the user devices include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glass, head mounted displays (HMDs)), digital TVs, desktop computers, digital signage, etc.

[0499] Each server in the content streaming system can be operated as a distributed server, in which case data received by each server can be processed in a distributed manner.

[0500] As described above, the embodiments described in the present invention may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in each drawing may be implemented and executed on a computer, processor, microprocessor, controller, or chip.

[0501] In addition, the decoder and encoder to which the present invention is applied may be included in a multimedia broadcasting transmitting / receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a video camera, a video on demand (VoD) service providing device, an OTT video (Over the top video) device, an Internet streaming service providing device, a three-dimensional (3D) video device, an image telephone video device, and a medical video device, and are used to process a video signal or a data signal. For example, an OTT video (Over the top video) device may include a game console, a Blu-ray player, an Internet-connected TV, a home theater system, a smartphone, a tablet PC, a DVR (Digital Video Recorder), and the like.

[0502] In addition, the processing method to which the present invention is applied can be produced in the form of a program executed by a computer and can be stored in a computer-readable recording medium. Multimedia data having a data structure according to the present invention can also be stored in a computer-readable recording medium. The computer-readable recording medium includes any kind of storage device and distributed storage device in which computer-readable data is stored. The computer-readable recording medium can include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable recording medium includes a medium realized in the form of a carrier wave (e.g., transmission through the Internet). In addition, a bit stream generated by the encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0503] Furthermore, the embodiments of the present invention may be realized as a computer program product by program code, which may be executed by a computer according to the embodiments of the present invention. The program code may be stored on a carrier readable by a computer.

[0504] The above-described embodiments are combinations of the components and features of the present invention in a predetermined form. Each component or feature should be considered as optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. It is also possible to combine some components and / or features to configure an embodiment of the present invention. The order of operations described in the embodiments of the present invention may be changed. Some configurations or features of any embodiment may be included in other embodiments, or may be replaced with corresponding configurations or features of other embodiments. It is obvious that claims that do not have an explicit reference relationship in the claims may be combined to configure an embodiment, or may be included as a new claim by amendment after filing.

[0505] An embodiment of the present invention may be implemented in various ways, such as hardware, firmware, software, or a combination thereof. In a hardware implementation, an embodiment of the present invention may be implemented using one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0506] In the case of implementation by firmware or software, an embodiment of the present invention may be implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above. The software code may be stored in a memory and driven by a processor. The memory may be located inside or outside the processor and may transmit and receive data to and from the processor by various means known in the art.

[0507] It is obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential features of the present invention. Therefore, the above detailed description should not be interpreted as limiting in all respects, but should be considered as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention. [Industrial Applicability]

[0508] The above-described preferred embodiments of the present invention have been disclosed for illustrative purposes, and those skilled in the art may improve, modify, substitute or add various other embodiments within the technical spirit and scope of the present invention disclosed in the appended claims below.

Claims

1. 1. A method for decoding a video signal by an apparatus, comprising: generating a first merge candidate list for the current block based on a plurality of spatial candidates, including a lower-left corner neighboring spatial candidate, a left neighboring spatial candidate, a top-right corner neighboring spatial candidate, an upper neighboring spatial candidate, and a top-left corner neighboring spatial candidate, and a temporal candidate of the current block, the plurality of spatial candidates preceding the temporal candidate in the first merge candidate list; generating a second merging candidate list by adding history-based candidates including motion information for blocks decoded before the current block to the first merging candidate list; parsing a merge index in the second merge candidate list that indicates a merge candidate to be applied to the current block; generating a prediction block of the current block based on motion information of the merge candidate indicated by the merge index; generating a reconstructed block of the current block based on the predicted block; The step of generating a second merge candidate list comprises: performing a pruning check whether the history-based candidate has the same motion information as the left neighboring spatial candidate and the top neighboring spatial candidate included in the first merge candidate list; adding the history-based candidate only if the pruning check indicates that the history-based candidate does not have the same motion information as the left neighboring spatial candidate and the top neighboring spatial candidate included in the first merge candidate list; A decoding method, wherein the pruning check is not performed on the lower-left corner neighboring spatial candidate, the upper-right corner neighboring spatial candidate, and the upper-left corner neighboring spatial candidate included in the first merge candidate list.

2. 2. The method of claim 1, wherein the history-based candidates are derived from a predefined number of candidates in a history-based buffer that stores the history-based candidates.

3. The step of generating a second merge candidate list comprises: adding history-based space-time candidates; The decoding method of claim 1 , wherein the history-based spatio-temporal candidate is derived using two history-based candidates and the temporal candidate.

4. The method of claim 3 , wherein the motion vector of the history-based spatio-temporal candidate is derived as an average of the motion vector of the temporal candidate and an average of the motion vectors of the two history-based candidates.

5. The step of generating a second merge candidate list comprises:

2. The method of claim 1, further comprising: determining whether to add a history-based candidate to the first merge candidate list, starting from a last history-based candidate that was last added to the history-based merge candidate list, among a plurality of history-based candidates included in the history-based merge candidate list.

6. 1. A method of encoding a video signal by an apparatus, comprising: generating a first merge candidate list for the current block based on a plurality of spatial candidates, including a lower-left corner neighboring spatial candidate, a left neighboring spatial candidate, a top-right corner neighboring spatial candidate, an upper neighboring spatial candidate, and a top-left corner neighboring spatial candidate, and a temporal candidate of the current block, the plurality of spatial candidates preceding the temporal candidate in the first merge candidate list; generating a second merging candidate list by adding history-based candidates including motion information for blocks decoded before the current block to the first merging candidate list; generating a merge index indicating a merge candidate in the second merge candidate list to be applied to the current block; generating a prediction block of the current block based on motion information of the merge candidate indicated by the merge index; generating a residual block of the current block based on the predicted block; The step of generating a second merge candidate list comprises: performing a pruning check whether the history-based candidate has the same motion information as the left neighboring spatial candidate and the top neighboring spatial candidate included in the first merge candidate list; adding the history-based candidate only if the pruning check indicates that the history-based candidate does not have the same motion information as the left neighboring spatial candidate and the top neighboring spatial candidate included in the first merge candidate list; The method of claim 1, wherein the pruning check is not performed on the lower-left corner neighboring spatial candidate, the upper-right corner neighboring spatial candidate, and the upper-left corner neighboring spatial candidate included in the first merging candidate list.

7. A method for transmitting data including a bitstream for video, comprising: obtaining a bitstream for the video; transmitting the data including the bitstream; The bitstream comprises: generating a first merge candidate list for the current block based on a plurality of spatial candidates, including a lower-left corner neighboring spatial candidate, a left neighboring spatial candidate, a top-right corner neighboring spatial candidate, an upper neighboring spatial candidate, and a top-left corner neighboring spatial candidate, and a temporal candidate of the current block, the plurality of spatial candidates preceding the temporal candidate in the first merge candidate list; generating a second merging candidate list by adding to the first merging candidate list a history-based candidate including motion information for a block decoded prior to the current block; generating a merge index that indicates a merge candidate in the second merge candidate list that has been applied to the current block; generating a prediction block of the current block based on motion information of the merging candidate indicated by the merging candidate; generating a residual block of the current block based on the predicted block; The step of generating a second merging candidate list comprises: performing a pruning check whether the history-based candidate has the same motion information as the left neighboring spatial candidate and the top neighboring spatial candidate included in the first merge candidate list; adding the history-based candidate only if the pruning check indicates that the history-based candidate does not have the same motion information as the left neighboring spatial candidate and the top neighboring spatial candidate included in the first merge candidate list; A method of transmitting, wherein the pruning check is not performed on the lower-left corner neighboring space candidate, the upper-right corner neighboring space candidate, and the upper-left corner neighboring space candidate included in the first merging candidate list.

Images