METHOD AND APPARATUS FOR PROCESSING A VIDEO SIGNAL BASED ON INTER PREDICTION - Patent application

Abstract

To provide a method and apparatus for processing video signals to improve the complexity of redundancy checks and increase efficiency.SOLUTION: A method includes constructing a first merge candidate list of a current block using a spatial merge candidate and temporal merge candidate of the current block, constructing a second merge candidate list by adding a history-based merge candidate indicating motion information of a block coded prior to the current block to the first merge candidate list, obtaining a merge index indicating a merge candidate applied to the current block within the second merge candidate list, and generating a prediction block of the current block using motion information of a merge candidate indicated by the merge index. The history-based merge candidate may be added to the first merge candidate list if the history-based merge candidate has motion information not overlapping motion information of a pre-defined merge candidate among merge candidates included in the first merge candidate list.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The embodiments of the present specification relate to a method and apparatus for processing a video signal using inter prediction, and in particular to a method and apparatus for performing inter prediction using history-based motion vector prediction. [Background technology]

[0002] Compression coding refers to a series of signal processing techniques for transmitting digitized information over communication lines or storing it in a form suitable for storage media. Media such as video, images, and audio can be compressed and coded, and the technology that performs compression coding on video in particular is called video compression.

[0003] Next-generation video content will have characteristics known as high spatial resolution, high frame rate, and high dimensionality of scene representation. Processing such content will bring about significant increases in memory storage, memory access rate, and processing power.

[0004] Therefore, there is a need to design coding tools to process the next generation of video content more efficiently. Summary of the Invention [Problem to be solved by the invention]

[0005] The purpose of the embodiments herein is to propose a restriction on the redundancy check for adding HMVP candidates to the merge list (or AMVP list).

[0006] The technical problems to be solved in the examples of this specification are not limited to the technical problems mentioned above, and other technical problems not mentioned here will be clearly understood by those skilled in the art to which the present invention pertains from the description below. [Means for solving the problem]

[0007] In one aspect of the present specification, a method for processing a video signal based on inter prediction includes constructing a first merge candidate list for a current block using spatial merge candidates and temporal merge candidates for the current block; constructing a second merge candidate list by adding history-based merge candidates indicating motion information of blocks previously coded for the current block to the first merge candidate list; obtaining a merge index in the second merge candidate list that indicates a merge candidate to be applied to the current block; and generating a prediction block for the current block using motion information of the merge candidate indicated by the merge index. The history-based merge candidate may be added to the first merge candidate list if it has motion information that does not overlap with predefined merge candidates from among the merge candidates included in the first merge candidate list.

[0008] Preferably, the history-based merge candidate can be added to the first merge candidate list if it has motion information that does not overlap with a predefined specific number of merge candidates from among the merge candidates included in the first merge candidate list.

[0009] Preferably, the history-based merge candidate may be added to the first merge candidate list if it has motion information that does not overlap with a particular spatial merge candidate included in the first merge candidate list.

[0010] Preferably, the history-based merge candidates may be derived from a predefined number of candidates in a history-based buffer that stores the history-based merge candidates.

[0011] Preferably, the step of constructing the second merge candidate list further includes the step of adding a history-based spatio-temporal merge candidate, and the history-based spatio-temporal candidate can be derived using two of the history-based merge candidates and the temporal merge candidate.

[0012] Preferably, the motion vector of the history-based spatio-temporal candidate can be derived from an average value of the motion vectors of the two history-based merge candidates and an average value of the motion vector of the temporal merge candidate.

[0013] Another aspect of an embodiment of the present specification is an apparatus for processing a video signal based on inter prediction, comprising: a memory for storing the video signal; and a processor coupled to the memory, wherein the processor constructs a first merge candidate list for a current block using spatial merge candidates and temporal merge candidates for the current block; constructs a second merge candidate list by adding a history-based merge candidate indicating motion information of a block previously coded for the current block to the first merge candidate list; obtains a merge index in the second merge candidate list that indicates a merge candidate to be applied to the current block; and generates a prediction block for the current block using motion information of the merge candidate indicated by the merge index, wherein the history-based merge candidate may be added to the first merge candidate list if it has motion information that does not overlap with a predefined merge candidate from among the merge candidates included in the first merge candidate list. [Effects of the Invention]

[0014] According to embodiments of the present disclosure, by limiting the redundancy checks for addition to the merge list (or AMVP list), the complexity associated with redundancy checks can be improved and efficiency can be increased.

[0015] The effects obtained by the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the following description. [Brief explanation of the drawings]

[0016] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide examples of the present invention and, together with the detailed description, explain the technical features of the present invention.

[0017] [Figure 1] As an example of a video signal processing device according to an embodiment of the present invention, an example of the functional configuration of an encoder will be shown. [Figure 2] 1 shows a schematic block diagram of a decoding device for decoding a video / image signal as an embodiment to which the present invention is applied. [Figure 3] 1 is a diagram showing an example of a multi-type tree structure as an embodiment to which the present invention can be applied; [Figure 4] 1 is a diagram illustrating a signaling mechanism for partition information of a quadtree with nested multi-type tree structure as an embodiment to which the present invention can be applied. [Figure 5] 1 is a diagram illustrating a method for dividing a CTU into multiple CUs based on a quadtree and nested multi-type tree structure, as an embodiment to which the present invention can be applied. [Figure 6] 1 is a diagram illustrating a method for restricting ternary-tree splitting as an example to which the present invention can be applied; [Figure 7]1 is a diagram illustrating redundant division patterns that can occur in the division of a binary tree and the division of a ternary tree, as an example to which the present invention can be applied. [Figure 8] 1 is a diagram illustrating an inter-prediction-based video / image encoding method according to an embodiment of the present invention and an inter-prediction unit within an encoding device according to an embodiment of the present invention. [Figure 9] 1 is a diagram illustrating an inter-prediction-based video / image encoding method according to an embodiment of the present invention and an inter-prediction unit within an encoding device according to an embodiment of the present invention. [Figure 10] 1 is a diagram illustrating a method for decoding inter-prediction-based video / image according to an embodiment of the present invention and an inter-prediction unit in a decoding device according to an embodiment of the present invention; [Figure 11] 1 is a diagram illustrating a method for decoding inter-prediction-based video / image according to an embodiment of the present invention and an inter-prediction unit in a decoding device according to an embodiment of the present invention; [Figure 12] FIG. 10 is a diagram for explaining peripheral blocks used in a merge mode or a skip mode as an embodiment to which the present invention is applied. [Figure 13] 1 is a flowchart illustrating a method for constructing a merge candidate list according to an embodiment of the present invention. [Figure 14] 1 is a flowchart illustrating a method for constructing a merge candidate list according to an embodiment of the present invention. [Figure 15] 1 shows examples of motion models according to an embodiment of the present invention. [Figure 16] 10 illustrates an example of motion vectors of control points for affine motion prediction according to an embodiment of the present invention. [Figure 17] 10 illustrates an example of motion vectors for each sub-block of a block to which affine motion prediction is applied, according to an embodiment of the present invention. [Figure 18] 10 illustrates an example of neighboring blocks used for affine motion prediction in affine merge mode according to an embodiment of the present invention. [Figure 19] 10 illustrates an example of a block for which affine motion prediction is performed using neighboring blocks to which affine motion prediction is applied according to an embodiment of the present invention. [Figure 20] 10A and 10B are diagrams illustrating a method for generating a merge candidate list using neighboring affine coding blocks according to an embodiment of the present invention. [Figure 21] 10A and 10B are diagrams illustrating a method for constructing an affine merge candidate list using neighboring blocks coded using affine prediction according to an embodiment of the present invention; [Figure 22] 10A and 10B are diagrams illustrating a method for constructing an affine merge candidate list using neighboring blocks coded using affine prediction according to an embodiment of the present invention; [Figure 23] 10 shows examples of neighboring blocks used for affine motion prediction in an affine inter mode according to an embodiment of the present invention. [Figure 24] 10 shows examples of neighboring blocks used for affine motion prediction in an affine inter mode according to an embodiment of the present invention. [Figure 25] 10 is a diagram illustrating a method for deriving motion vector candidates using motion information of neighboring blocks in an affine inter mode according to an embodiment of the present invention; [Figure 26] 10 is a diagram illustrating a method for deriving motion vector candidates using motion information of neighboring blocks in an affine inter mode according to an embodiment of the present invention; [Figure 27] 10 illustrates an example of a method for deriving a sub-block-based affine motion vector field according to an embodiment of the present invention. [Figure 28] 10 illustrates an exemplary method for generating a prediction block and a motion vector in inter prediction using an affine motion model according to an embodiment of the present invention. [Figure 29]4A and 4B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points according to an embodiment of the present invention; [Figure 30] 10A and 10B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention; [Figure 31] 10A and 10B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention; [Figure 32] 10A and 10B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention; [Figure 33] 10A and 10B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention; [Figure 34] 10A and 10B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention; [Figure 35] 10A and 10B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention; [Figure 36] 10A and 10B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention; [Figure 37] 10A and 10B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention; [Figure 38] 10A and 10B are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention; [Figure 39] 1 is a flowchart illustrating a method for storing HMVP according to an embodiment of the present specification. [Figure 40] FIG. 10 is a diagram illustrating an HMVP table that operates in a non-restrictive FIFO manner according to an embodiment of the present specification. [Figure 41]FIG. 10 is a diagram illustrating an HMVP table that operates in a restrictive FIFO manner according to an embodiment of the present specification. [Figure 42] 1 is a diagram illustrating an HMVP LUT and a long-term HMVP LUT according to an embodiment of the present specification. [Figure 43] FIG. 10 illustrates an example method for updating an HMVP LUT according to an embodiment of the present specification. [Figure 44] FIG. 1 illustrates a method for limiting the number of HMVP candidates that are subject to pruning checks according to an embodiment of the present specification. [Figure 45] FIG. 1 illustrates an example method for performing a pruning check according to an embodiment of the present disclosure. [Figure 46] 10 is a diagram illustrating a method for deriving H-STMVP candidates using motion vectors that refer to different reference pictures according to an embodiment of the present specification. FIG. [Figure 47] 1 is a flowchart illustrating a method for processing a video signal based on inter prediction according to an embodiment of the present invention; [Figure 48] 1 shows an example of a block diagram of an apparatus for processing a video signal according to an embodiment of the present disclosure; [Figure 49] 1 shows a video coding system to which the present invention is applied; [Figure 50] As an embodiment to which the present invention is applied, a structural diagram of a content streaming system is shown. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below together with the accompanying drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention can be practiced. The following detailed description includes specific details to provide a thorough understanding of the present invention. However, those skilled in the art will recognize that the present invention can be practiced without such specific details.

[0019] In some cases, in order to avoid obscuring the concept of the present invention, well-known structures and devices may be omitted or shown in block diagram form, focusing on the core functions of each structure and device.

[0020] Furthermore, the terms used in this invention are currently commonly used and general terms, but in certain cases, the applicant will use terms arbitrarily selected by the applicant. In such cases, the meanings will be clearly described in the detailed description of the relevant parts, so it is important to clarify that the terms used in this invention should not be interpreted simply based on their names, but should be interpreted by understanding the meanings of the relevant terms.

[0021] Specific terms used in the following description are provided to aid in understanding the present invention, and the use of such specific terms may be changed to other forms without departing from the technical spirit of the present invention. For example, in the case of a signal, data, sample, picture, frame, block, etc., they may be appropriately substituted and interpreted in each encoding process.

[0022] Hereinafter, in this specification, the term "processing unit" refers to a unit in which encoding / decoding processes such as prediction, transformation, and / or quantization are performed. For convenience of explanation, the processing unit may also be referred to as a "processing block" or a "block."

[0023] The processing unit may be interpreted as including a unit of a luma component and a unit of a chroma component. For example, the processing unit may correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0024] Furthermore, the processing unit may be interpreted to include a unit of a luma component or a unit of a chroma component. For example, the processing unit may correspond to a coding tree block (CTB), coding block (CB), prediction block (PU), or transform block (TB) of the luma component. Alternatively, the processing unit may correspond to a coding tree block (CTB), coding block (CB), prediction block (PU), or transform block (TB) of the chroma component. Furthermore, without being limited thereto, the processing unit may be interpreted to include a unit of a luma component and a unit of a chroma component.

[0025] Furthermore, the processing units are not necessarily limited to square blocks, but may also be configured in the form of polygons having three or more vertices.

[0026] In the following description, pixels or picture elements are commonly referred to as samples. Using a sample can mean using a pixel value or picture element value.

[0027] FIG. 1 shows a schematic block diagram of an encoding device in which encoding of a video / image signal is performed as an embodiment to which the present invention is applied.

[0028] Referring to FIG. 1, the encoding device 100 may include an image dividing unit 110, a subtraction unit 115, a transform unit 120, a quantization unit 130, an inverse quantization unit 140, an inverse transform unit 150, an addition unit 155, a filtering unit 160, a memory 170, an inter prediction unit 180, an intra prediction unit 185, and an entropy encoding unit 190. The inter prediction unit 180 and the intra prediction unit 185 may be commonly referred to as a prediction unit. That is, the prediction unit may include the inter prediction unit 180 and the intra prediction unit 185. The transform unit 120, the quantization unit 130, the inverse quantization unit 140, and the inverse transform unit 150 may be included in a residual processing unit. The residual processing unit may further include a subtraction unit 115. In one embodiment, the image division unit 110, subtraction unit 115, transformation unit 120, quantization unit 130, inverse quantization unit 140, inverse transformation unit 150, addition unit 155, filtering unit 160, inter prediction unit 180, intra prediction unit 185, and entropy encoding unit 190 may be configured as a single hardware component (e.g., an encoder or processor). Also, the memory 170 may include a decoded picture buffer (DPB) and may be configured as a digital storage medium.

[0029] The image division unit 110 may divide an input image (or picture or frame) input to the encoding device 100 into one or more processing units. For example, the processing units may be referred to as coding units (CUs). In this case, the coding units may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) based on a quad-tree binary-tree (QTBT) structure. For example, one coding unit may be divided into multiple coding units of deeper depths based on a quad-tree structure and / or a binary tree structure. In this case, for example, the quad-tree structure may be applied first, and then the binary tree structure may be applied later. Alternatively, the binary tree structure may be applied first. The coding procedure according to the present invention may be performed based on the final coding unit that is not further divided. In this case, based on coding efficiency according to image characteristics, the largest coding unit is immediately used as the final coding unit, or the coding unit is recursively divided into coding units of lower depths as needed, and the coding unit with the optimal size is used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be divided or partitioned from the final coding unit. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0030] The term "unit" is sometimes used interchangeably with terms such as "block" or "area." In general, an MxN block can refer to a set of samples or transform coefficients consisting of M columns and N rows. A sample generally refers to a pixel or pixel value, and can refer to only the pixel / pixel value of the luma component or only the pixel / pixel value of the chroma component. A sample is used as a term that corresponds to one pixel or pel of a picture (or image).

[0031] The encoding device 100 may subtract a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 180 or the intra prediction unit 185 from an input video signal (original block, original sample array) to generate a residual signal (residual signal, remaining block, remaining sample array), and the generated residual signal is transmitted to the conversion unit 120. In this case, as shown in the figure, a unit in the encoder 100 that subtracts a prediction signal (predicted block, prediction sample array) from an input video signal (original block, original sample array) may be referred to as a subtraction unit 115. The prediction unit may predict a block to be processed (hereinafter referred to as a current block) and generate a predicted block including prediction samples of the current block.

[0032] The prediction unit may determine whether intra prediction or inter prediction is applied in units of a current block or CU. As will be described later in the description of each prediction mode, the prediction unit may generate various information related to prediction, such as prediction mode information, and transmit the information to the entropy encoding unit 190. The prediction information may be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

[0033] The intra prediction unit 185 may predict the current block by referencing samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located far away, depending on the prediction mode. In intra prediction, prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, DC mode and planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the granularity of the prediction direction. However, this is merely an example, and more or fewer directional prediction modes may be used depending on the settings. The intra prediction unit 185 may also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0034] The inter prediction unit 180 may derive a predicted block of a current block based on a reference block (an array of reference samples) identified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter prediction mode, motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring blocks may be referred to as collocated reference blocks, collocated control units (colCUs), etc., and the reference picture including the temporal neighboring blocks may be referred to as collocated pictures (colPic). For example, the inter predictor 180 may construct a motion information candidate list based on neighboring blocks and generate information indicating which candidates are used to derive a motion vector and / or a reference picture index for the current block. Inter prediction may be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter predictor 180 may use motion information of neighboring blocks as motion information for the current block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted.In the case of motion vector prediction (MVP) mode, the motion vector of the current block can be indicated by signaling the motion vector difference using the motion vector of the surrounding blocks in the motion vector predictor.

[0035] The prediction signal generated by the inter prediction unit 180 or the intra prediction unit 185 is used to generate a reconstructed signal or a residual signal.

[0036] The transform unit 120 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-Linear Transform (CNT). Here, GBT refers to a transform obtained from a graph representing inter-pixel relationship information. CNT refers to a transform obtained based on a predicted signal generated using all previously reconstructed pixels. The transform process may be applied to pixel blocks having the same square size, or to non-square blocks of variable size.

[0037] The quantization unit 130 quantizes the transform coefficients and transmits the quantized signal to the entropy encoding unit 190. The entropy encoding unit 190 encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. The information about the quantized transform coefficients may be referred to as residual information. The quantization unit 130 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit 190 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoding unit 190 may encode information required for video / image restoration (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) can be transmitted or stored in the form of a bitstream in NAL (network abstraction layer) units. The bitstream can be transmitted via a network or stored in a digital storage medium. Here, the network can include a broadcasting network and / or a network, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitting unit (not shown) that transmits the signal output from the entropy encoding unit 190 and / or a storing unit (not shown) that stores the signal can be configured as an internal / external element of the encoding device 100, or the transmitting unit can be a component of the entropy encoding unit 190.

[0038] The quantized transform coefficients output from the quantization unit 130 are used to generate a prediction signal. For example, the quantized transform coefficients may be inversely quantized and inversely transformed via the inverse quantization unit 140 and the inverse transform unit 150 in a loop to reconstruct a residual signal. The adder 155 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit 180 or the intra prediction unit 185. When there is no residual for the current block, as in the case where skip mode is applied, the predicted block is used as the reconstructed block. The adder 155 may be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next block to be processed in the current picture, and may also be used for inter prediction of the next picture after filtering, as described below.

[0039] The filtering unit 160 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 160 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture and store the modified reconstructed picture in the memory 170, specifically, in the DPB of the memory 170. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc. The filtering unit 160 may generate various information related to filtering, as will be described later in the description of each filtering method, and transmit the information to the entropy encoding unit 190. The filtering information may be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

[0040] The modified reconstructed picture transmitted to the memory 170 is used as a reference picture by the inter prediction unit 180. This allows the encoding device to avoid prediction mismatch between the encoding device 100 and the decoding device when inter prediction is applied, and also improves coding efficiency.

[0041] The memory 170DPB may store the modified reconstructed picture for use as a reference picture from the inter predictor 180. The memory 170 may store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter predictor 180 to be used as motion information of a spatially neighboring block or motion information of a temporally neighboring block. The memory 170 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra predictor 185.

[0042] FIG. 2 shows a schematic block diagram of a decoding device for decoding a video / image signal as an embodiment to which the present invention is applied.

[0043] Referring to FIG. 2, the decoding device 200 may include an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an adder 235, a filtering unit 240, a memory 250, an inter prediction unit 260, and an intra prediction unit 265. The inter prediction unit 260 and the intra prediction unit 265 may be collectively referred to as a prediction unit. That is, the prediction unit may include an inter prediction unit 180 and an intra prediction unit 185. The inverse quantization unit 220 and the inverse transform unit 230 may be collectively referred to as a residual processing unit. That is, the residual processing unit may include an inverse quantization unit 220 and an inverse transform unit 230. The entropy decoding unit 210, the inverse quantization unit 220, the inverse transform unit 230, the adder 235, the filtering unit 240, the inter prediction unit 260, and the intra prediction unit 265 may be configured as a single hardware component (e.g., a decoder or a processor) depending on the embodiment. The memory 170 may also include a decoded picture buffer (DPB) and may be configured by a digital storage medium.

[0044] When a bitstream including video / image information is input, the decoding device 200 can reconstruct an image corresponding to the process by which the video / image information was processed from the encoding device of Fig. 1. For example, the decoding device 200 can perform decoding using a processing unit applied in the encoding device. Therefore, the decoding processing unit can be, for example, a coding unit, which can divide a quad tree structure and / or a binary tree structure from a coding tree unit or a maximal coding unit. The reconstructed video signal decoded and output by the decoding device 200 can be played back in a playback device.

[0045] The decoding device 200 may receive a signal output from the encoding device of FIG. 1 in the form of a bitstream, and may decode the received signal via the entropy decoding unit 210. For example, the entropy decoding unit 210 may analyze the bitstream to derive information (e.g., video / image information) required for image restoration (or picture restoration). For example, the entropy decoding unit 210 may decode information in the bitstream based on a coding method such as Exponential Golomb Coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients related to residuals. More specifically, the CABAC entropy decoding method may receive nulls corresponding to each syntax element from the bitstream, determine a context model using information on the syntax element to be decoded and decoded information on neighboring and current blocks, or symbol / null information decoded in a previous step, predict the probability of occurrence of a null based on the determined context model, and perform arithmetic decoding of the null to generate a symbol corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method may determine a context model and then update the context model using decoded symbol / void information for the context model of the next symbol / void. Prediction-related information from the information decoded by the entropy decoding unit 2110 may be provided to a prediction unit (inter prediction unit 260 and intra prediction unit 265), and entropy-decoded residual values ​​from the entropy decoding unit 210, i.e., quantized transform coefficients and related parameter information, may be input to the inverse quantization unit 220. Furthermore, filtering-related information from the information decoded by the entropy decoding unit 210 may be provided to the filtering unit 240. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding device may be configured as an internal / external element of the decoding device 200, or the receiving unit may be a component of the entropy decoding unit 210.

[0046] The inverse quantization unit 220 may inverse quantize the quantized transform coefficients to output transform coefficients. The inverse quantization unit 220 may rearrange the quantized transform coefficients in the form of two-dimensional blocks. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit 220 may inverse quantize the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0047] The inverse transform unit 230 performs an inverse transform on the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0048] The prediction unit may perform prediction of the current block and generate a predicted block including prediction samples of the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoding unit 210, and may determine a specific intra / inter prediction mode.

[0049] The intra prediction unit 265 may predict a current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located far away from the current block depending on the prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 265 may also determine a prediction mode to be applied to the current block using prediction modes applied to neighboring blocks.

[0050] The inter prediction unit 260 may derive a predicted block of the current block based on a reference block (an array of reference samples) identified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in the inter prediction mode, motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. For example, the inter prediction unit 260 may construct a motion information candidate list based on neighboring blocks and derive a motion vector and / or a reference picture index of the current block based on received candidate selection information. Inter prediction may be performed based on various prediction modes, and the prediction information may include information indicating the inter prediction mode of the current block.

[0051] The adder 235 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to the predicted signal (predicted block, predicted sample array) output from the inter prediction unit 260 or the intra prediction unit 265. When there is no residual for the current block, such as when the skip mode is applied, the predicted block is used as the reconstructed block.

[0052] The adder 235 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of the next block to be processed in the current picture, and may also be used for inter prediction of the next picture after filtering, as described below.

[0053] The filtering unit 240 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 240 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture and transmit the modified reconstructed picture to the memory 250, specifically, to the DPB of the memory 250. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc.

[0054] The (modified) reconstructed picture stored in the DPB of the memory 250 is used as a reference picture by the inter predictor 260. The memory 250 may store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter predictor 260 to be used as motion information of a spatially neighboring block or a temporally neighboring block. The memory 170 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra predictor 265.

[0055] In this specification, the embodiments described for the filtering unit 160, inter prediction unit 180 and intra prediction unit 185 of the encoding device 100 can also be applied identically or correspondingly to the filtering unit 240, inter prediction unit 260 and intra prediction unit 265 of each decoding device 200.

[0056] Block Partitioning

[0057] The video / image coding method according to the present document may be performed based on various detailed techniques, each of which will be briefly described below. It will be apparent to those skilled in the art that the techniques described below can be related to related procedures such as prediction, residual processing ((inverse) transform, (inverse) quantization, etc.), coding of syntax elements, filtering, partitioning, etc. in the video / image encoding / decoding procedures described above and / or below.

[0058] The block partitioning procedure according to this document is performed by the image partitioning unit 110 of the encoding device described above, and partitioning-related information is processed (encoded) by the entropy encoding unit 190 and can be transmitted to the decoding device in the form of a bitstream. The entropy decoding unit 210 of the decoding device can derive a block partitioning structure of the current picture based on the partitioning-related information acquired from the bitstream, and can perform a series of procedures for image decoding (e.g., prediction, residual processing, block reconstruction, in-loop filtering, etc.) based on the deriving block partitioning structure of the current picture based on the partitioning-related information acquired from the bitstream.

[0059] Partitioning of picture into CTUs

[0060] A picture can be divided into a sequence of coding tree units (CTUs). A CTU can correspond to a coding tree block (CTB). Alternatively, a CTU can contain a coding tree block for luma samples and two coding tree blocks for corresponding chroma samples. That is, for a picture containing three sample arrays, a CTU contains an NxN block of luma samples and two corresponding blocks of chroma samples.

[0061] The maximum allowable size of a CTU for coding, prediction, etc. may be different from the maximum allowable size of a CTU for transforms. For example, the maximum allowable size of a luma block in a CTU may be 128x128.

[0062] Partitioning of the CTUs using a tree structure

[0063] A CTU may be divided into CUs based on a quad-tree (QT) structure. A quad-tree structure may also be referred to as a quaternary tree structure because it reflects various local characteristics. In this document, a CTU may be divided based on division of a multi-type tree structure, including not only a quad-tree but also a binary-tree (BT) and a ternary-tree (TT). Hereinafter, the QTBT structure may include quad-tree and binary-tree based division structures, and the QTBTTT may include quad-tree, binary-tree, and ternary-tree based division structures. Alternatively, the QTBT structure may include quad-tree, binary-tree, and ternary-tree based division structures. In a coding tree structure, a CU may have a square or rectangular shape. A CTU may first be divided into a quad-tree structure. The leaf nodes of the quad-tree structure may then be further divided into a multi-type tree structure.

[0064] FIG. 3 is a diagram showing an example of a multi-type tree structure as an embodiment to which the present invention can be applied.

[0065] In one embodiment of the present specification, the multi-type tree structure may include four splitting types as shown in Figure 3. The four splitting types may include vertical binary splitting (SPLIT_BT_VER), horizontal binary splitting (SPLIT_BT_HOR), vertical ternary splitting (SPLIT_TT_VER), and horizontal ternary splitting (SPLIT_TT_HOR).

[0066] The leaf nodes of the multi-type tree structure may be called CUs. Such CUs are used for prediction and transformation procedures. In this document, CUs, PUs, and TUs may generally have the same block size. However, if the maximum supported transform length is smaller than the width or height of the color component of the CU, the CU and TU may have different block sizes.

[0067] FIG. 4 is a diagram illustrating a signaling mechanism for partition division information of a quadtree with nested multi-type trees, as an embodiment to which the present invention can be applied.

[0068] Here, the CTU is treated as the root of the quadtree and is first partitioned in the quadtree structure. The leaf nodes of each quadtree can then be further partitioned in the multitype tree structure. In the multitype tree structure, a first flag (e.g., mtt_split_cu_flag) is signaled to indicate whether the node is to be additionally partitioned. If the node is to be additionally partitioned, a second flag (e.g., mtt_split_cu_vertical_flag) can be signaled to indicate the splitting direction. Then, a third flag (e.g., mtt_split_cu_binary_flag) can be signaled to indicate whether the split type is binary or turnary. For example, based on the mtt_split_cu_vertical_flag and the mtt_split_cu_binary_flag, the multi-type tree splitting mode (MttSplitMode) of the CU can be derived as shown in Table 1 below.

[0069] [Table 1]

[0070] FIG. 5 is a diagram illustrating a method for dividing a CTU into multiple CUs based on a quadtree and nested multi-type tree structure, as an embodiment to which the present invention can be applied.

[0071] Here, bold block edges indicate quad-tree partitioning, and the remaining edges indicate multi-type tree partitioning. Quad-tree partitioning with multi-type trees can provide a content-adapted coding tree structure. A CU can correspond to a coding block (CB). Alternatively, a CU can include a coding block of luma samples and two coding blocks of corresponding chroma samples. The size of a CU can be as large as a CTU or as small as 4x4 luma sample units. For example, in a 4:2:0 color format (or chroma format), the maximum chroma CB size can be 64x64, and the minimum chroma CB size can be 2x2.

[0072] In this document, for example, the maximum allowed luma TB size may be 64x64 and the maximum allowed chroma TB size may be 32x32. If the width or height of a CB split along the tree structure is greater than the maximum transform width or height, the CB may be automatically (or implicitly) split until the horizontal and vertical TB size restrictions are met.

[0073] Meanwhile, for a quad-tree coding tree scheme with multi-type trees, the following parameters can be defined and identified in the sps syntax element:

[0074] - CTU size: The root node size of a quaternary tree

[0075] - MinQTSize: The minimum allowed quaternary tree leaf node size

[0076] - MaxBtSize: (the maximum allowed binary tree root node size)

[0077] - MaxTtSize: (the maximum allowed ternary tree root node size)

[0078] - MaxMttDepth: The maximum allowed hierarchy depth of multi-type tree splitting from a quadtree leaf

[0079] - MinBtSize: The minimum allowed binary tree leaf node size

[0080] - MinTtSize: The minimum allowed ternary tree leaf node size

[0081] As an example of a quad-tree coding tree structure with a multi-type tree, the CTU size can be set to 64x64 blocks of 128x128 luma samples and two corresponding chroma samples (in 4:2:0 chroma format). In this case, MinOTSize can be set to 16x16, MaxBtSize can be set to 128x128, MaxTtSzie can be set to 64x64, MinBtSize and MinTtSize (for both width and height) can be 4x4, and MaxMttDepth can be set to 4. Quartet-tree partitioning can be applied to the CTU to generate quad-tree leaf nodes. The quad-tree leaf nodes can be referred to as leaf QT nodes. The quad-tree leaf nodes can range in size from 16x16 (i.e., the MinOTSize) to 128x128 (i.e., the CTU size). If a leaf QT node is 128x128, it may not be further split into a binary tree / ternary tree because the split would exceed MaxBtSize and MaxTtszie (i.e., 64x64). Otherwise, the leaf QT node may be further split into a multitype tree. Therefore, the leaf QT node is the root node of the multitype tree, and the leaf QT node may have a multitype tree depth (mttDepth) value of 0. If the multitype tree depth reaches MaxMttDepth (e.g., 4), no further splits may be considered. If the width of the multitype tree node is equal to MinBtSize and less than or equal to 2xMinTtSize, no further horizontal splits may be considered. If the height of the multitype tree node is equal to MinBtSize and less than or equal to 2xMinTtSize, no further vertical splits may be considered.

[0082] FIG. 6 is a diagram illustrating a method for restricting ternary-tree splitting as an embodiment to which the present invention can be applied.

[0083] 6, to allow for the design of a 64x64 luma block and a 32x32 chroma pipeline in a hardware decoder, the TT partitioning may be restricted in certain cases. For example, if the width or height of the luma coding block is greater than a predetermined value (e.g., 32, 64), the TT partitioning may be restricted as shown in FIG.

[0084] In this document, the coding tree scheme can support luma and chroma blocks having a tree structure of separate blocks.

[0085] For P and B slices, the luma and chroma CTBs within a CTU may be restricted to have the same coding tree structure. However, for an I slice, the luma and chroma blocks may have separate block tree structures. If the separate block tree mode is applied, the luma CTB may be divided into CUs based on a specific coding tree structure, and the chroma CTB may be divided into chroma CUs based on another coding tree structure. This may mean that a CU within an I slice may consist of a coding block for a luma component or two coding blocks for chroma components, and a CU in a P or B slice may consist of three color component blocks.

[0086] Although the quad-tree coding tree structure with a multi-type tree was described above in "Partitioning of the CTUs using a tree structure," the structure in which a CU is partitioned is not limited to this. For example, the BT structure and the TT structure can be interpreted as concepts included in the structure of a multiple partitioning tree (MPT), and a CU can be interpreted as being partitioned through a QT structure and an MPT structure. In one example of partitioning a CU through a QT structure and an MPT structure, the partitioning structure can be determined by signaling a syntax element (e.g., MPT_split_type) containing information on whether a leaf node of the QT structure is partitioned into several blocks and a syntax element (e.g., MPT_split_mode) containing information on whether a leaf node of the QT structure is partitioned vertically or horizontally.

[0087] In another example, the CUs may be divided in a different manner, such as a QT structure, a BT structure, or a TT structure. That is, unlike the case where lower-depth CUs are divided into 1 / 4 the size of higher-depth CUs based on the QT structure, or into 1 / 2 the size of higher-depth CUs based on the BT structure, or the case where lower-depth CUs are divided into 1 / 4 or 1 / 2 the size of higher-depth CUs based on the TT structure, lower-depth CUs may be divided into 1 / 5, 1 / 3, 3 / 8, 3 / 5, 2 / 3, or 5 / 8 the size of higher-depth CUs, as the case may be, and the method of dividing CUs is not limited thereto.

[0088] If a portion of a tree node block exceeds the bottom or right picture boundary, the tree node block can be constrained so that all samples of all coded CUs are located within the picture boundary. In this case, for example, the following partitioning rules can be applied:

[0089] - If a portion of a tree node block exceeds both the bottom and the right picture boundaries,

[0090] - If the block is a QT node and the size of the block is larger than the minimum QT size, the block is forced to be split with QT split mode.

[0091] - Otherwise, the block is forced to be split with SPLIT_BT_HOR mode

[0092] - Otherwise if a portion of a tree node block exceeds the bottom picture boundaries,

[0093] - If the block is a QT node, and the size of the block is larger than the minimum QT size, and the size of the block is larger than the maximum BT size, the block is forced to be split with QT split mode.

[0094] - Otherwise, if the block is a QT node, and the size of the block is larger than the minimum QT size and the size of the block is smaller than or equal to the maximum BT size, the block is forced to be split with QT split mode or SPLIT_BT_HOR mode.

[0095] - Otherwise(the block is a BTT node or the size of the block is smaller than or equal to the minimum QT size), the block is forced to be split with SPLIT_BT_HOR mode.

[0096] - Otherwise if a portion of a tree node block exceeds the right picture boundaries,

[0097] - If the block is a QT node, and the size of the block is larger than the minimum QT size, and the size of the block is larger than the maximum BT size, the block is forced to be split with QT split mode.

[0098] - Otherwise, if the block is a QT node, and the size of the block is larger than the minimum QT size and the size of the block is smaller than or equal to the maximum BT size, the block is forced to be split with QT split mode or SPLIT_BT_VER mode.

[0099] - Otherwise(the block is a BTT node or the size of the block is smaller than or equal to the minimum QT size), the block is forced to be split with SPLIT_BT_VER mode.

[0100] Meanwhile, the quad-tree coding block structure with the multi-type tree described above can provide a very flexible block partitioning structure. Because of the partitioning types supported by the multi-type tree, other partitioning patterns can potentially result in the same coding block structure in some cases. By limiting the occurrence of such redundant partitioning patterns, the amount of data required for partitioning information can be reduced. This will be explained with reference to the following diagram.

[0101] FIG. 7 is a diagram illustrating redundant division patterns that can occur in the division of a binary tree and the division of a ternary tree, as an embodiment to which the present invention can be applied.

[0102] As shown in FIG. 7, two levels of consecutive binary splits in one direction have the same coding block structure as the binary split of the center partition after Turner-Lee splitting. In this case, the split of the binary tree of the center partition of the Turner-Lee tree in the given direction can be restricted. This restriction can be applied to all CUs of a picture. When such a specific split is restricted, the signaling of the corresponding syntax element can be modified to reflect this restricted case, thereby reducing the number of bits signaled for partitioning. For example, as in the example shown in FIG. 7, when the split of the binary tree of the center partition of a CU is restricted, the mtt_split_cu_binary_flag syntax element, which indicates whether the split is a binary split or a Turner-Lee split, is not signaled and can be inferred by the decoder as a value of 0.

[0103] prediction

[0104] To reconstruct the current processing unit for which decoding is performed, the decoded portions of the current picture or other pictures that contain the current processing unit can be used.

[0105] A picture (slice) that uses only the current picture for reconstruction, i.e., performs only intra-frame prediction, can be called an intra-picture or I-picture (slice), a picture (slice) that uses up to one motion vector and reference index to predict each unit can be called a predictive picture or P-picture (slice), and a picture (slice) that uses up to two motion vectors and reference indexes can be called a bi-predictive picture or B-picture (slice).

[0106] Intra prediction refers to a prediction method that derives a current processing block from data elements (e.g., sample values, etc.) of the same decoded picture (or slice), i.e., a method of predicting pixel values ​​of a current processing block by referring to a reconstructed region in the current picture.

[0107] Inter prediction will be explained in more detail below.

[0108] Inter prediction (or inter-frame prediction)

[0109] Inter prediction refers to a prediction method that derives a current processing block based on data elements (e.g., sample values ​​or motion vectors) of a picture other than the current picture, i.e., a method that predicts pixel values ​​of the current processing block by referring to reconstructed regions in other reconstructed pictures other than the current picture.

[0110] Inter-prediction (or inter-picture prediction) is a technique that removes redundancies that exist between pictures, mostly through motion estimation and motion compensation.

[0111] The embodiments of the present specification describe the detailed techniques of the inter prediction method previously described in Figures 1 and 2. In the case of a decoder, this can be represented by an inter prediction-based video / image decoding method of Figure 10 and an inter prediction unit in a decoding device of Figure 11, which will be described later. In addition, in the case of an encoder, this can be represented by an inter prediction-based video / image encoding method of Figure 8 and an inter prediction unit in an encoding device of Figure 9, which will be described later. Furthermore, data encoded according to Figures 8 and 9 can be stored in the form of a bitstream.

[0112] The prediction unit of the encoding device / decoding device may perform inter prediction on a block-by-block basis to derive predicted samples. Inter prediction may refer to a prediction derived in a manner dependent on data elements (e.g., sample values, motion information, etc.) of pictures other than the current picture. When inter prediction is applied to the current block, a predicted block (predicted sample array) of the current block may be derived based on a reference block (reference sample array) identified by a motion vector on a reference picture pointed to by a reference picture index.

[0113] In this case, to reduce the amount of motion information transmitted in inter prediction mode, the motion information of the current block can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include information on the inter prediction type (e.g., L0 prediction, L1 prediction, Bi prediction, etc.).

[0114] When inter-prediction is applied, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring blocks may be the same or different. The temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and the reference picture including the temporal neighboring blocks may be called collocated pictures (colPics).

[0115] For example, a motion information candidate list may be constructed based on neighboring blocks of the current block, and flag or index information may be signaled indicating which candidate is selected (used) to derive the motion vector and / or reference picture index of the current block.

[0116] Inter prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the motion information of the current block can be the same as the motion information of a selected neighboring block. In skip mode, unlike merge mode, a residual signal may not be transmitted. In motion vector prediction (MVP) mode, the motion vector of a selected neighboring block can be used as a motion vector predictor, and a motion vector difference can be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0117] 8 and 9 are diagrams illustrating an inter-prediction based video / image encoding method according to an embodiment of the present invention and an inter-prediction unit in an encoding device according to an embodiment of the present invention.

[0118] 8 and 9, S801 may be performed by the inter prediction unit 180 of the encoding apparatus, and S802 may be performed by the residual processing unit of the encoding apparatus. Specifically, S802 may be performed by the subtraction unit 115 of the encoding apparatus. In S803, prediction information may be derived by the inter prediction unit 180 and encoded by the entropy encoding unit 190. In S803, residual information may be derived by the residual processing unit and encoded by the entropy encoding unit 190. The residual information is information about the residual samples. The residual information may include information about quantized transform coefficients for the residual samples.

[0119] As described above, the residual samples are derived into transform coefficients through the transform unit 120 of the encoding device, and the transform coefficients can be derived as quantized transform coefficients through the quantization unit 130. Information about the quantized transform coefficients can be encoded in the entropy encoding unit 190 through a residual coding procedure.

[0120] The encoding apparatus performs inter prediction of a current block (S801). The encoding apparatus may derive an inter prediction mode and motion information of the current block and generate a prediction sample of the current block. Here, the steps of determining the inter prediction mode, deriving the motion information, and generating the prediction sample may be performed simultaneously, or one step may be performed before the other steps. For example, the inter prediction unit 180 of the encoding apparatus may include a prediction mode determination unit 181, a motion information derivation unit 182, and a prediction sample derivation unit 183. The prediction mode determination unit 181 may determine a prediction mode of the current block, the motion information derivation unit 182 may derive motion information of the current block, and the prediction sample derivation unit 183 may derive a motion sample of the current block.

[0121] For example, the inter prediction unit 180 of the encoding device may search for blocks similar to the current block within a certain region (search area) of a reference picture through motion estimation and derive a reference block whose difference from the current block is minimum or equal to or less than a certain criterion. Based on this, the inter prediction unit 180 may derive a reference picture index indicating the reference picture in which the reference block is located, and derive a motion vector based on the difference between the positions of the reference block and the current block. The encoding device may determine a mode to be applied to the current block from various prediction modes. The encoding device may compare RD costs for the various prediction modes to determine an optimal prediction mode for the current block.

[0122] For example, when a skip mode or a merge mode is applied to the current block, the encoding device may construct a merge candidate list (described below) and derive a reference block, among reference blocks indicated by merge candidates included in the merge candidate list, whose difference between the current block and the current block is minimum or equal to or less than a certain criterion. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to the decoding device.

[0123] Motion information of the current block can be derived using motion information of the selected merge candidate.

[0124] As another example, when the (A)MVP mode is applied to the current block, the encoding device may construct an (A)MVP candidate list (described below) and use the motion vector of a selected MVP (motion vector predictor) candidate from among the MVP candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, a motion vector pointing to a reference block derived by the motion estimation described above may be used as the motion vector of the current block, and the MVP candidate having the smallest difference from the motion vector of the current block may be the selected MVP candidate. A motion vector difference (MVD), which is the difference obtained by subtracting the MVP from the motion vector of the current block, may be derived. In this case, information regarding the MVD may be signaled to the decoding device. Furthermore, when the (A)MVP mode is applied, the value of the reference picture index may be configured as reference picture index information and separately signaled to the decoding device.

[0125] The encoding apparatus may derive residual samples based on the predicted samples (S802) by comparing the original samples of the current block with the predicted samples.

[0126] The encoding device encodes video information including prediction information and residual information (S803). The encoding device may output the encoded video information in the form of a bitstream. The prediction information may include prediction mode information (e.g., skip flag, merge flag, or mode index) and information about motion information, which are information related to the prediction procedure. The information about the motion information may include candidate selection information (e.g., merge index, MVP flag, or MVP index), which is information for deriving a motion vector. In addition, the information about the motion information may include the above-mentioned information about MVD and / or reference picture index information.

[0127] Furthermore, the information regarding the motion information may include information indicating whether L0 prediction, L1 prediction, or pair (bi) prediction is applied. The residual information is information regarding the residual samples. The residual information may include information regarding quantized transform coefficients for the residual samples.

[0128] The output bitstream can be stored in a (digital) storage medium and then transmitted to the decoding device, or can be transmitted to the decoding device via a network.

[0129] Meanwhile, as described above, the encoding apparatus can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is because the encoding apparatus derives the same prediction result as that performed by the decoding apparatus, thereby improving coding efficiency. Therefore, the encoding apparatus can store the reconstructed picture (or reconstructed samples, reconstructed blocks) in memory and use it as a reference picture for inter prediction. As described above, an in-loop filter procedure can be further applied to the reconstructed picture.

[0130] 10 and 11 are diagrams illustrating an inter-prediction based video / image decoding method according to an embodiment of the present invention and an inter-prediction unit in a decoding device according to an embodiment of the present invention.

[0131] 10 and 11, the decoding device may perform actions corresponding to the operations performed by the encoding device. The decoding device may perform prediction on the current block based on the received prediction information and derive a prediction sample.

[0132] Steps S1001 to S1003 may be performed by the inter prediction unit 260 of the decoding device, and the residual information of step S1004 may be obtained from a bitstream by the entropy decoding unit 210 of the decoding device. The residual processing unit of the decoding device may derive residual samples of the current block based on the residual information. Specifically, the inverse quantization unit 220 of the residual processing unit may perform inverse quantization based on quantized transform coefficients derived based on the residual information to derive transform coefficients, and the inverse transform unit 230 of the residual processing unit may perform inverse transform of the transform coefficients to derive residual samples of the current block. Step S1005 may be performed by the adder 235 or a reconstruction unit of the decoding device.

[0133] Specifically, the decoding device may determine a prediction mode of the current block based on received prediction information (S1001). The decoding device may determine which inter-prediction mode is applied to the current block based on prediction mode information in the prediction information.

[0134] For example, it may determine whether the merge mode or (A)MVP mode is applied to the current block based on the merge flag, or it may select one of various inter prediction mode candidates based on the mode index. The inter prediction mode candidates may include skip mode, merge mode, and / or (A)MVP mode, or may include various inter prediction modes described below.

[0135] The decoding device derives motion information of the current block based on the determined inter prediction mode (S1002). For example, when a skip mode or a merge mode is applied to the current block, the decoding device may construct a merge candidate list (described below) and select one merge candidate from among the merge candidates included in the merge candidate list. The selection may be performed based on the selection information (merge index) described above. Motion information of the current block may be derived using motion information of the selected merge candidate. The motion information of the selected merge candidate is used as the motion information of the current block.

[0136] As another example, when the (A)MVP mode is applied to the current block, the decoding device may construct an (A)MVP candidate list (described below) and use the motion vector of a selected MVP (motion vector predictor) candidate from among the MVP candidates included in the (A)MVP candidate list as the MVP of the current block. The selection may be performed based on the selection information (MVP flag or MVP index) described above. In this case, the MVD of the current block may be derived based on information related to the MVD, and the motion vector of the current block may be derived based on the MVP of the current block and the MVD. Furthermore, the decoding device may derive a reference picture index of the current block based on index information of the reference picture. In the reference picture list for the current block, a picture pointed to by the reference picture index may be derived as a reference picture referenced for inter-prediction of the current block.

[0137] On the other hand, as will be described later, the motion information of the current block may be derived without constructing a candidate list, and in this case, the motion information of the current block may be derived according to a procedure disclosed in a prediction mode, which will be described later. In this case, the construction of the candidate list as described above may be omitted.

[0138] The decoding device may generate a predicted sample of the current block based on the motion information of the current block (S1003). In this case, the reference picture may be derived based on the reference picture index of the current block, and the predicted sample of the current block may be derived using a sample of a reference block pointed to in the reference picture by the motion vector of the current block. In this case, as described below, a filtering procedure may be performed on all or some of the predicted samples of the current block, depending on the circumstances.

[0139] For example, the inter prediction unit 260 of the decoding device may include a prediction mode determination unit 261, a motion information derivation unit 262, and a prediction sample derivation unit 263, and may determine the prediction mode of the current block based on prediction mode information received by the prediction mode determination unit 261, derive motion information of the current block (such as a motion vector and / or a reference picture index) based on information regarding the motion information received from the motion information derivation unit 262, and derive a prediction sample of the current block in the prediction sample derivation unit 263.

[0140] The decoding device generates residual samples of the current block based on the received residual information (S1004). The decoding device generates reconstructed samples of the current block based on the predicted samples and the residual samples, and may generate a reconstructed picture based on the reconstructed samples (S1005). Thereafter, as described above, an in-loop filter procedure may be further applied to the reconstructed picture.

[0141] As described above, the inter-prediction procedure may include an inter-prediction mode determination step, a motion information derivation step according to the determined prediction mode, and a prediction execution step (creation of a prediction sample) based on the derived motion information.

[0142] Determination of inter prediction mode

[0143] Various inter-prediction modes can be used for predicting the current block within a picture. For example, in merge mode and skip mode, various modes such as MVP mode and affine mode can be used. Additional modes such as DMVR (Decoder side motion vector refinement) mode and AMVR (Adaptive Motion Vector Resolution) mode are also used. Affine mode can also be called affine motion prediction mode. MVP mode can also be called AMVP (Advanced Motion Vector Prediction) mode.

[0144] Prediction mode information indicating the inter-prediction mode of the current block may be signaled from the encoding device to the decoding device. The prediction mode information may be included in a bitstream and received by the decoding device. The prediction mode information may include index information indicating one of multiple candidate modes. Alternatively, the inter-prediction mode may be indicated through hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags.

[0145] For example, a skip flag may be signaled to indicate whether the skip mode is applied, a merge flag may be signaled to indicate whether the merge mode is applied if the skip mode is not applicable, and an MVP mode may be applied if the merge mode is not applicable, or a flag for additional division may be further signaled. The affine mode may be signaled as an independent mode or as a mode dependent on the merge mode or MVP mode. For example, the affine mode may be configured as one candidate in a merge candidate list or an MVP candidate list, as described below.

[0146] Derivation of motion information according to inter-prediction mode

[0147] Inter prediction can be performed using motion information of the current block. The encoding device can derive optimal motion information of the current block through a motion estimation procedure. For example, the encoding device can use an original block in the original picture of the current block to search for a similar reference block with high correlation in fractional pixel units within a predetermined search range in the reference picture, thereby deriving motion information. Block similarity can be derived based on a phase-based sample value difference. For example, block similarity can be calculated based on the SAD between the current block (or the template of the current block) and the reference block (or the template of the reference block). In this case, motion information can be derived based on the reference block with the smallest SAD within the search range. The derived motion information can be signaled to the decoding device in several ways depending on the inter prediction mode.

[0148] Merge and Skip Modes

[0149] FIG. 12 is a diagram for explaining peripheral blocks used in merge mode or skip mode as an embodiment to which the present invention is applied.

[0150] When a merge mode is applied, the motion information of the current predicted block is not directly transmitted, but is derived using the motion information of neighboring predicted blocks. Therefore, the motion information of the current predicted block can be indicated by transmitting flag information indicating that the merge mode is used and a merge index indicating which neighboring predicted block is used.

[0151] In one embodiment of the present invention,

[0152] In one embodiment of the present specification, an encoder may search for merge candidate blocks to be used to derive motion information of a current prediction block to perform a merge mode. For example, up to five merge candidate blocks may be used, but the present invention is not limited to this. The maximum number of merge candidate blocks may be transmitted in a slice header (or a tile group header), but the present invention is not limited to this. After finding the merge candidate blocks, the encoder may generate a merge candidate list and select the merge candidate block with the smallest cost as the final merge candidate block.

[0153] An embodiment of the present specification provides various embodiments of merge candidate blocks that make up the merge candidate list.

[0154] The merge candidate list may include, for example, five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate may be used. As a specific example, for spatial merge candidates, the blocks shown in FIG. 12 may be used as spatial merge candidates.

[0155] FIG. 13 is a flowchart illustrating a method for constructing a merge candidate list according to an embodiment of the present invention.

[0156] Referring to FIG. 13, a coding apparatus (encoder / decoder) searches for spatially neighboring blocks of a current block and inserts derived spatial merge candidates into a merge candidate list (S1301). For example, the spatial neighboring blocks may include a lower left corner neighboring block, a left neighboring block, a right upper corner neighboring block, an upper neighboring block, and an upper left corner neighboring block of the current block. However, this is merely an example, and additional neighboring blocks such as a right neighboring block, a lower neighboring block, and a lower right neighboring block may also be used as the spatial neighboring blocks. The coding apparatus may search the spatial neighboring blocks based on a priority to detect usable blocks and derive motion information of the detected blocks as the spatial merge candidates. For example, the encoder and decoder may search the five blocks shown in FIG. 12 in the order of A1, B1, B0, A0, and B2, and sequentially index usable candidates to form a merge candidate list.

[0157] The coding apparatus searches for temporal neighboring blocks of the current block and inserts derived temporal merge candidates into the merge candidate list (S1302). The temporal neighboring blocks may be located on a reference picture that is a different picture from the current picture in which the current block is located. The reference picture in which the temporal neighboring blocks are located may be referred to as a collocated picture or col picture. The temporal neighboring blocks may be searched for in the order of a lower right corner neighboring block and a lower right center block of a co-located block of the current block on the col picture.

[0158] Meanwhile, when motion data compression is applied, specific motion information can be stored as representative motion information for each fixed storage unit in the col picture. In this case, it is not necessary to store motion information for all blocks within the fixed storage unit, thereby achieving the effect of motion data compression. In this case, the fixed storage unit may be predetermined, for example, in units of 16x16 samples or 8x8 samples, or size information of the fixed storage unit may be signaled from the encoder to the decoder. When motion data compression is applied, motion information of the temporally neighboring blocks may be replaced with representative motion information of the fixed storage unit in which the temporally neighboring blocks are located.

[0159] That is, in this case, in terms of implementation, the temporal merge candidate may be derived based on motion information of a prediction block that is arithmetically shifted right by a certain value based on the coordinates (upper left sample position) of the temporal neighboring block, rather than based on the coordinates of the temporal neighboring block, and then arithmetically shifted left. For example, if the certain storage unit is a 2nx2 sample unit, and the coordinates of the temporal neighboring block are (xTnb, yTnb), then motion information of a prediction block located at the modified position ((xTnb >> n)<< n), (yTnb >> n)<< n)) is used for the temporal merge candidate.

[0160] Specifically, for example, if the constant storage unit is a 16x16 sample unit and the coordinates of the temporal neighboring block are (xTnb, yTnb), motion information of a predicted block located at a modified position ((xTnb >> 4) << 4), (yTnb >> 4) << 4) is used for the temporal merge candidate. Alternatively, for example, if the constant storage unit is an 8x8 sample unit and the coordinates of the temporal neighboring block are (xTnb, yTnb), motion information of a predicted block located at a modified position ((xTnb >> 3) << 3), (yTnb >> 3) << 3) is used for the temporal merge candidate.

[0161] The coding device may check whether the number of current merging candidates is less than the maximum number of merging candidates (S1303). The maximum number of merging candidates may be predefined or may be signaled from the encoder to the decoder. For example, the encoder may generate and encode information regarding the maximum number of merging candidates and transmit the information to the decoder in the form of a bitstream. If the maximum number of merging candidates is reached, the process of adding subsequent candidates may not proceed.

[0162] If the number of current merge candidates is less than the maximum number of merge candidates as a result of the check, the coding apparatus inserts additional merge candidates into the merge candidate list (S1304). The additional merge candidates may include, for example, ATMVPs, combined bi-predictive merge candidates (if the slice type of the current slice is type B), and / or zero vector merge candidates.

[0163] If the check results in the number of current merge candidates being not less than the maximum number of merge candidates, the coding apparatus may terminate construction of the merge candidate list. In this case, the encoder may select an optimal merge candidate from among the merge candidates constituting the merge candidate list based on a rate-distortion (RD) cost, and may signal selection information (e.g., a merge index) indicating the selected merge candidate to the decoder. The decoder may select the optimal merge candidate based on the merge candidate list and the selection information.

[0164] As described above, the motion information of the selected merge candidate is used as the motion information of the current block, and the predicted sample of the current block can be derived based on the motion information of the current block. The encoder can derive residual samples of the current block based on the predicted sample and signal residual information related to the residual sample to a decoder. The decoder can generate reconstructed samples based on the residual samples derived based on the residual information and the predicted sample, and generate a reconstructed picture based on the reconstructed samples.

[0165] When skip mode is applied, the motion information of the current block can be derived in the same way as when merge mode is applied previously, except that when skip mode is applied, the residual signal of the block is omitted, and therefore the predicted sample is immediately used for the reconstructed sample.

[0166] MVP Mode

[0167] FIG. 14 is a flowchart illustrating a method for constructing a merge candidate list according to an embodiment of the present invention.

[0168] When the Motion Vector Prediction (MVP) mode is applied, a motion vector predictor (mvp) candidate list can be generated using motion vectors of reconstructed spatially surrounding blocks (which may be the surrounding blocks described in 12 above) and / or motion vectors corresponding to temporally surrounding blocks (or Col blocks). That is, the motion vectors of reconstructed spatially surrounding blocks and / or motion vectors corresponding to temporally surrounding blocks are used as motion vector predictor candidates.

[0169] The prediction information may include selection information (e.g., an MVP flag or an MVP index) indicating an optimal motion vector predictor candidate selected from the motion vector predictor candidates included in the list. In this case, the prediction unit may select a motion vector predictor for the current block from the motion vector predictor candidates included in the motion vector candidate list using the selection information. The prediction unit of the encoding device may obtain a motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, encode the MVD, and output it in the form of a bitstream. That is, the MVD may be obtained by subtracting the motion vector predictor from the motion vector of the current block. In this case, the prediction unit of the decoding device may obtain the motion vector difference included in the prediction information and derive the motion vector of the current block by adding the motion vector difference and the motion vector predictor. The prediction unit of the decoding device may obtain or derive a reference picture index, etc., indicating a reference picture, from the prediction information. For example, the motion vector predictor candidate list may be configured as shown in FIG. 14.

[0170] Affine motion prediction

[0171] FIG. 15 shows examples of motion models according to an embodiment of the present invention.

[0172] Conventional video compression technologies (e.g., high efficiency video coding (HEVC)) use one motion vector to represent the motion of a coding block. Although using one motion vector per block can represent optimal motion for each block, it may not be optimal for each pixel. Therefore, determining an optimal motion vector for each pixel can improve coding efficiency. Next, an embodiment of the present invention describes a motion prediction method for encoding or decoding a video signal using multiple motion models. In particular, motion vectors of two to four control points can be used to represent motion vectors for each pixel of a block or for each sub-block. Such a prediction method using motion vectors of multiple control points can be referred to as affine motion prediction, affine prediction, etc.

[0173] An affine motion model according to an embodiment of the present invention can express four motion models as shown in Fig. 15. An affine motion model that expresses three motions (translation, scale, rotate) DMF among the motions that can be expressed by the affine motion model is referred to as a similarity (or simplified) affine motion model, and for convenience of explanation, the description of an embodiment of the present invention will be based on the similarity (or simplified) affine motion model, but the present invention is not limited thereto.

[0174] FIG. 16 shows an example of motion vectors of control points for affine motion prediction according to an embodiment of the present invention.

[0175] As shown in Figure 16, affine motion prediction can determine the motion vector of a pixel position (or sub-block) containing a block using two control point motion vector (CPMV) pairs, v_0 and v_1. In this case, the set of motion vectors can be referred to as an affine motion vector field (MVF). In this case, the affine motion vector field can be determined using the following Equation 1:

[0176]

number

[0177] In Equation 1, v_0 (v_0={v_0x,v_0y}) indicates the motion vector (CPMV0) of the first control point located at the upper left side of the current block 1300, and v_1 (v_1={v_1x,v_1y}) indicates the motion vector (CPMV1) of the second control point located at the upper right side of the current block 1300. In addition, w indicates the width of the current block 1300. v (v={v_x,v_y}) indicates the motion vector at the {x,y} position. A motion vector in sub-block (or pixel) units can be derived using Equation 1. In one embodiment, the accuracy of the motion vector can be rounded to 1 / 16 accuracy.

[0178] FIG. 17 illustrates an example of motion vectors for each sub-block of a block to which affine motion prediction is applied according to an embodiment of the present invention.

[0179] 17, during encoding or decoding, an affine motion vector field (MVF) can be determined in units of pixels or blocks. That is, in affine motion prediction, the motion vector of a current block can be derived in units of pixels or sub-blocks.

[0180] When an affine motion vector field is determined in units of pixels, a motion vector is obtained based on the value of each pixel, and when it is determined in units of blocks, a motion vector of the block can be obtained based on the value of the center pixel of the block. In this document, it is assumed that an affine motion vector field (MVF) is determined in units of 4*4 blocks as shown in Figure 17. However, this is for convenience of explanation and is not limited to embodiments of the present invention. Figure 17 shows an example in which a coding block is composed of 16*16 samples and an affine motion vector field (MVF) is determined in units of 4*4 size blocks.

[0181] Affine motion prediction can include affine merge mode (AF_MERGE) and affine inter mode (AF_INTER). The AF_INTER mode can include AF_4_INTER mode, which uses a four-parameter-based motion model, and AF_6_INTER mode, which uses a six-parameter-based motion model.

[0182] Affine merge mode

[0183] AF_MERGE determines the control point motion vector (CPMV) based on the affine motion model of neighboring blocks coded as affine motion prediction. Affine-coded neighboring blocks in search order are used for AF_MERGE. When one or more neighboring blocks are coded as affine motion prediction, the current block can be coded as AF_MERGE.

[0184] That is, when the affine merge mode is applied, the CPMV of the current block can be derived using the CPMV of the neighboring blocks. In this case, the CPMV of the neighboring blocks can be used as the CPMV of the current block as is, or the CPMV of the neighboring blocks can be modified based on the size of the neighboring blocks and the size of the current block, and then used as the CPMV of the current block.

[0185] FIG. 18 shows an example of neighboring blocks used for affine motion prediction in affine merge mode according to an embodiment of the present invention.

[0186] In the affine merge (AF_MERGE) mode, the encoder can perform the encoding process as follows:

[0187] Step 1: Scan the neighboring blocks A to E (1810, 1820, 1830, 1840, 1850) of the current coding block 1800 in alphabetical order, and determine the block coded in the first affine prediction mode based on the scanning order as a candidate block for affine merge (AF_MERGE).

[0188] Step 2: Determine an affine motion model using the control point motion vectors (CPMVs) of the determined candidate blocks.

[0189] Step-3: Based on the affine motion model of the candidate block, the control point motion vector (CPMV) of the current block 1800 is determined, and the MVF of the current block 1800 is determined.

[0190] FIG. 19 illustrates an example of a block on which affine motion prediction is performed using neighboring blocks to which affine motion prediction is applied according to an embodiment of the present invention.

[0191] For example, as shown in FIG. 19, if block A 1920 is coded in affine mode, after determining block A 1920 as a candidate block, an affine motion model can be derived using the control point motion vectors (CPMVs) (e.g., v2, v3) of block A 1920, and then control point motion vectors (CPMVs) v0 and v1 of current block 1900 can be determined. Based on the control point motion vectors (CPMVs) of current block 1900, an affine motion vector field (MVF) of current block 1900 can be determined and coded.

[0192] FIG. 20 is a diagram illustrating a method for generating a merge candidate list using neighboring affine-coded blocks according to an embodiment of the present invention.

[0193] 20, when affine merge candidates are used to determine CPMV pairs, candidates such as those shown in Fig. 20 are used. In Fig. 20, it is assumed that the scan order of the candidate list is set to A, B, C, D, and E. However, the present invention is not limited to this, and various orders can be set in advance.

[0194] As an example, if the number of candidates (hereinafter referred to as affine candidates) coded in affine mode (or affine prediction) used in the neighboring blocks (i.e., A, B, C, D, and E) is zero, the affine merge mode of the current block may be skipped. If only one affine candidate is used (e.g., A), the motion model of that candidate is used to derive motion vectors (CPMV_0 and CPMV_1) of the control points of the current block. In this case, an index indicating that candidate may not be required (or coded). If multiple affine candidates are used, two candidates in scan order may be configured in an AF_MERGE candidate list. In this case, candidate selection information, such as an index indicating the selected candidate in the candidate list, may be signaled. The selection information may be a flag or index information, and may be designated as AF_MERGE_flag, AF_merge_idx, etc.

[0195] In embodiments of the present specification, motion compensation for the current block can be performed based on the size of the sub-blocks. In this case, the sub-block size of the affine block (i.e., the current block) is derived. If the width and height of the sub-blocks are all greater than four luma samples, a motion vector for each sub-block can be derived, and DCT-IF-based motion compensation (1 / 16 fel for luma and 1 / 32 for chrominance) can be performed on the sub-blocks. Otherwise, enhanced bi-linear interpolation filter-based motion compensation can be performed on the entire affine block.

[0196] In the present embodiment, when the merge / skip flag is true and the width and height of a CU are all equal to or greater than 8, an affine flag at the CU level is signaled via the bitstream to indicate that the affine merge mode is used. When a CU is coded as AF_MERGE, a merge candidate index with a maximum value of '5' is signaled to specify which motion information candidate from the affine merge candidate list is used for the CU.

[0197] 21 and 22 are diagrams illustrating a method for constructing an affine merge candidate list using neighboring blocks coded using affine prediction according to an embodiment of the present invention.

[0198] Referring to FIG. 21, the affine merge candidate list is constructed as the next step.

[0199] 1) Insert model-based affine candidates

[0200] Model-based affine candidates mean that the candidates are derived from valid surrounding reconstructed blocks coded in affine mode. As shown in Figure 21, the scanning order of candidate blocks is left (A), top (B), top right (C) and bottom left (D) to top left (E).

[0201] When the neighboring lower-left block (A) is coded in the 6-parameter affine mode, it will obtain motion vectors (v_4, v_5, v_6) for the upper-left corner, upper-right corner, and lower-left corner of the CU containing block (A). The motion vectors (v_0, v_1, v_2) for the upper-left corner of the current block are calculated according to the motion vectors (v_4, v_5, and v_6) in the 6-parameter affine model.

[0202] If the neighboring lower-left block (A) is coded in the 4-parameter affine mode, it will have motion vectors (v_4, v_5) for the upper-left and upper-right corners of the CU containing block (A). The motion vector (v_0, v_1) for the upper-left corner of the current block is calculated according to the motion vectors (v_4, v_5) in the 4-parameter affine model.

[0203] 2) Insert control point-based affine candidates

[0204] Referring to FIG. 21, the control point-based candidate means that the candidate is constructed by combining the motion information around each control point.

[0205] Motion information for a control point is first derived from the designated spatial and temporal neighboring blocks shown in Figure 21. CP_k (k = 1, 2, 3, 4) indicates the kth control point. Also, A, B, C, D, E, F, and G are spatial positions for predicting CP_k (k = 1, 2, 3), and H is the temporal position for predicting CP_k.

[0206] The coordinates of CP_1, CP_2, CP_3 and CP_4 are (0,0), (W,0), (H,0) and (W,H), respectively, where W and H are the width and height of the current block.

[0207] The motion information for each control point is obtained according to the following priority:

[0208] For CP_1, the checking priority is A->B->C, if A is used, then A is used. Otherwise, if B is used, then B is used. If both A and B are not used, then C is used. If all three candidates are not used, then the motion information of CP1 cannot be obtained.

[0209] For CP_2, the checking priority is E->D.

[0210] For CP_3, the checking priority is G->F.

[0211] For CP_4, H is used.

[0212] Second, a combination of control points is used to construct a motion model.

[0213] The motion vectors of two control points are required to calculate the transformation parameters in a four-parameter affine model. The two control points can be selected from one of the following six combinations: {CP_1, CP_4}, {CP_2, CP_3}, {CP_1, CP_2}, {CP_2, CP_4}, {CP_1, CP_3}, {CP_3, CP_4}. For example, using the CP_1 and CP_2 control points to construct a four-parameter affine motion model is denoted as "Affine(CP_1, CP_2)".

[0214] The motion vectors of three control points are required to calculate the transformation parameters in a six-parameter affine model. The three control points can be selected from one of the following four combinations: {CP_1, CP_2, CP_4}, {CP_1, CP_2, CP_3}, {CP_2, CP_3, CP_4}, {CP_1, CP_3, CP_4}. For example, using CP_1, CP_2, and CPv3 control points to construct a six-parameter affine motion model is represented as "Affine(CP_1, CP_2, CP_3)."

[0215] Also, in the present embodiment, in the affine merge mode, if an affine merge candidate exists, it may always be considered as a six-parameter affine mode.

[0216] Affine inter mode

[0217] FIG. 23 shows an example of neighboring blocks used for affine motion prediction in affine inter mode according to an embodiment of the present invention.

[0218] Referring to FIG. 23, affine motion prediction can include affine merge mode (AF_MERGE) and affine inter mode (AF_INTER). In affine inter mode (AF_INTER), after determining the motion vector prediction (CPMVP) and CPMV of two control points, a control point motion vector difference (CPMVD) corresponding to the difference can be transmitted from the encoder to the decoder. A specific encoding process for affine inter mode (AF_INTER) can be as follows:

[0219] Step 1: Determine two CPMVP pair candidates

[0220] Step-1.1: Determine combinations of up to 12 CPMVP candidates (see Equation 2 below)

[0221]

number

[0222] In Equation 2, v_0 is the motion vector (CPMV0) at the upper left control point 2310 of the current block 2300, v_1 ​​is the motion vector (CPMV1) at the upper right control point 2311 of the current block 2300, v_2 is the motion vector (CPMV2) at the lower left control point 2312 of the current block 2300, v_A is the motion vector of neighboring block A 2320 adjacent to the upper left side of the upper left control point 2310 of the current block 2300, v_B is the motion vector of neighboring block B 2322 adjacent to the upper left side of the upper left control point 2310 of the current block 2300, and v_C is the motion vector of the current block 2300. v_F indicates the motion vector of peripheral block F 2330 adjacent to the left of the lower left control point 2312 of the current block 2300; and v_G indicates the motion vector of peripheral block G 2332 adjacent to the left of the lower left control point 2312 of the current block 2300.

[0223] Step 1.2: Sort the CPMVP candidate combinations by the smallest difference value (DV) and use the top two candidates (see Equation 3 below).

[0224]

number

[0225] v_0x is the x-axis element of the motion vector (V0 or CPMV0) of the upper left control point 2310 of the current block 2300, v_1x is the x-axis element of the motion vector (V1 or CPMV1) of the upper right control point 2311 of the current block 2300, v_2x is the x-axis element of the motion vector (V_2 or CPMV_2) of the lower left control point 2312 of the current block 2300, and v_0y is the x-axis element of the motion vector (V1 or CPMV1) of the upper left control point 2310 of the current block 2300. v_1y is the y-axis element of the motion vector (V_0 or CPMV_0), v_1y is the y-axis element of the motion vector (V_1 or CPMV_1) of the upper right control point 2311 of the current block 2300, v_2y is the y-axis element of the motion vector (V_2 or CPMV_2) of the lower left control point 2312 of the current block 2300, w is the width of the current block 2300, and h is the height of the current block 2300.

[0226] Step 2: If the number of control point motion vector predictor (CPMVP) pair candidates is less than 2, use the AMVP candidate list.

[0227] Step 3: For each of the two candidates, determine the control point motion vector predictor (CPMVP), compare the RD cost, and optimally select the candidate with the smallest RD cost.

[0228] Step 4: Transfer the index corresponding to the best candidate and the control point motion vector difference (CPMVD)

[0229] In the present embodiment, AF_INTER provides a CPMVP candidate configuration process. Similar to AMVP, the number of candidates is two, and an index indicating the position in the candidate list is signaled.

[0230] The procedure for setting up the CPMVP candidate list is as follows:

[0231] 1) Scan the surrounding blocks to see if they can be coded as affine motion prediction. If the scanned block is coded as affine prediction, derive the motion vector pair for the current block from the affine motion models of the scanned surrounding blocks until the number of candidates becomes 2.

[0232] 2) If the number of candidates is less than two, a candidate construction process is performed. Also, in this embodiment of the present invention, a four-parameter (two-control point) affine inter-mode is used to predict the zoom-in / out and rotation motion model and content. As shown in Figure 16, the affine motion field of a block is described by the motion vectors of two control points.

[0233] The motion vector field (MVF) of a block is described by Equation 1 above.

[0234] In the prior art, advanced motion vector prediction (AMVP) mode is required to signal motion vector prediction (MVP) indices and motion vector differences (MVDs). When AMVP mode is applied to the present invention, affine_flag is signaled to indicate whether affine prediction is used. When affine prediction is applied, the syntax of inter_dir, ref_idx, mvp_index, and two MVDs (mvd_x and mvd_y) is signaled. An affine MVP pair candidate list containing two affine MVP pairs is generated. The signaled mvp_index is used to select one of them. An affine MVP pair is generated by two types of affine MVP candidates. One is a spatial inherited affine candidate candidate, and the other is a corner-derived affine candidate candidate. When surrounding CUs are coded in affine mode, spatial inherited affine candidates can be generated. The affine motion model of the peripherally affine coded block is used to generate a two-control-point MVP pair of motion vectors. The MV of the two-control-point MVP pair of the spatial inheritance affine candidate is derived by using the following formula:

[0235]

number

[0236]

number

[0237] If V_B0, V_B1, and V_B2 can be replaced with the upper left MV, upper right MV, and lower left MV of any reference / surrounding CU, (posCurCU_X, posCurCU_Y) is the position of the upper left sample of the current CU relative to the upper left sample of the frame, and (posRefCU_X, posRefCU_Y) is the position of the upper left sample of the reference / surrounding CU relative to the upper left sample of the frame.

[0238]

number

[0239]

number

[0240] FIG. 24 shows an example of neighboring blocks used for affine motion prediction in affine inter mode according to an embodiment of the present invention.

[0241] Referring to FIG. 24, if the number of MVP pairs is less than two, corner-derived affine candidates are used. The surrounding motion vectors are used to derive affine MVP pairs as shown in FIG. 24. For the first corner-derived affine candidate, the first available MV in set A (A0, A1, and A2) and the first available MV in set B (B0 and B1) are used to form the first MVP pair. For the second corner-derived affine candidate, the first available MV in set A and the first available MV in set C (C0 and C1) are used to calculate the MV of the upper right side control point. The first available MV in set A and the calculated upper right side control point MV form the second MVP pair.

[0242] In an embodiment of the present invention, two candidate sets containing two (three) candidates {mv_0, mv_1} ({mv_0, mv_1, mv_2) are used to predict two (three) control points of the affine motion model. Given the motion vector differential (mvd_0, mvd_1, mvd_2) and the control points, they are calculated by using the following formula:

[0243]

number

[0244] 25 and 26 are diagrams illustrating a method for deriving motion vector candidates using motion information of neighboring blocks in an affine inter mode according to an embodiment of the present invention.

[0245] The affine candidate list extends affine motion from spatially neighboring blocks (extrapolated affine candidates) and is appended by a combination of motion vectors from spatially neighboring blocks (virtual affine candidates). The candidate set is set as follows:

[0246] 1. Up to two different affine MV predictor sets are derived from the affine motion of neighboring blocks. Neighboring blocks A0, A1, B0, B1, and B2 are identified as shown in Figure 25. If a neighboring block is coded by an affine motion model and its reference frame is the same as that of the current block, two (for a four-parameter affine model) or three (for a six-parameter affine model) control points of the current block are derived from the affine model of the neighboring block.

[0247] 2. Figure 29 shows the neighboring blocks used to generate the virtual affine candidate set. The neighboring MVs are divided into three groups: S_0 = {mv_A, mv_B, mv_C}, S_1 = {mv_D, mv_E}, S_2 = {mv_F, mv_G}. mv_0 is the first MV in S0 that references the same reference picture as the current block. mv_2 is the first MV in S1 that references the same reference picture as the current block.

[0248] If mv_0 and mv_1 are given, mv_2 can be derived by the following equation 9.

[0249]

number

[0250] In Equation 9, the current block size is WxH.

[0251] If only mv_0 and mv_2 are given, mv_1 can be derived by the following equation 10:

[0252]

number

[0253] In one embodiment of the present specification, affine inter prediction can be performed based on the following sequence:

[0254] Input: Affine motion parameters, reference picture samples

[0255] Output: Predicted block of CU

[0256] process

[0257] - Induce the size of sub-blocks of affine blocks

[0258] - If the sub-block width and width mode are greater than 4 luma samples,

[0259] - For each sub-block

[0260] - Guide sub-block motion vectors

[0261] - DCT-IF based motion compensation (1 / 16 pel for luma, 1 / 32 pel for chroma) invoked on sub-blocks

[0262] Otherwise, an enhanced bi-linear interpolation filter-based compensation is invoked for the entire affine block.

[0263] Also, in one embodiment of the present specification, if the merge / skip flag is false and the width and width of a CU are 8 or more, an affine flag is signaled at the CU level to indicate whether the affine inter mode is used. If a CU is coded as an affine inter mode, a model flag is signaled to indicate whether a 4-parameter or 6-parameter affine model is applied to the CU. If the model flag is true, the AF_6_INTER mode (6-parameter affine model) is applied and three MVDs are parsed; otherwise, the AF_4_INTER mode (4-parameter affine model) is applied and two MVDs are parsed.

[0264] In AF_4_INTER mode, similar to the affine merge mode, extrapolated motion vector pairs from neighboring blocks coded in affine mode are generated and inserted into the candidate list first.

[0265] Hereinafter, if the size of the candidate list is less than four, candidates with motion vector pairs {(v_0, v_1)| v_0 = {v_A, v_B, v_c}, v_1 ​​= {v_D, v_E}} are generated by using neighboring blocks. As shown in FIG. 22, v_0 is selected from the motion vectors of blocks A, B, and C. The motion vectors from neighboring blocks are scaled based on the relationship between the reference list and the neighboring block's POC of reference and the current CU. The approach for selecting v_1 from neighboring blocks D and E is similar. If the candidate list is greater than four, candidates are preferentially sorted based on the consistency of neighboring motion vectors (similar to two motion vectors in a candidate pair), and the first four candidates are saved.

[0266] If the number of candidates in the list is less than four, the list is padded with motion vector pairs by duplicating each AMVP candidate.

[0267] In AF_6_INTER mode, similar to the affine merge mode, extrapolated motion vector triples from neighboring blocks coded in affine mode are generated and preferentially inserted into the candidate list.

[0268] Hereinafter, if the size of the candidate list is less than four, candidates containing a motion vector triple {(v_0, v_1, v_2)| v_0 = {v_A, v_B, v_c}, v_1 ​​= {v_D, v_E}, v_2 = {v_G, v_H}} are generated using neighboring blocks. As shown in FIG. 22, v_0 is selected from the motion vectors of blocks A, B, or C. The motion vectors from neighboring blocks are scaled based on the relationship between the reference list, the POC of the neighboring block's reference, the POC of the current CU, and the POC of the current CU. The approach for selecting v_1 from neighboring blocks D and E is similar to that for selecting v_2 from F and G. If the candidate list is greater than four, the candidates are sorted based on the consistency of neighboring motion vectors (similar to two motion vectors from three candidates), and the first four candidates are kept.

[0269] If the number of candidate lists is less than four, the list can be padded with motion vector triples constructed by duplicating each AMVP candidate.

[0270] After the CPMV of the current CU is derived, the MVF of the current CU is generated based on Equation 11 under the 4-parameter affine model or Equation 12 under the 6-parameter affine model, depending on the number of affine parameters.

[0271]

number

[0272]

number

[0273] Here, the sub-block size MxN is derived from the following equation 13, and MvPre is the precision (1 / 16) of the part of the motion vector.

[0274]

number

[0275] After being derived by Equation 12, M and N should be adjusted downward if necessary to make them divisors of w and h. If M or N is smaller than 8, WIF is applied; otherwise, sub-block-based affine motion compensation is applied.

[0276] FIG. 27 illustrates an example of a method for deriving a sub-block-based affine motion vector field according to an embodiment of the present invention.

[0277] Referring to Figure 27, to derive a motion vector for each MxN sub-block, the motion vector of the center sample of each sub-block as shown in Figure 27 is calculated based on Equation 11 or 12 and rounded to 1 / 16 fractional accuracy. An SHVC upsampling interpolation filter is applied to generate a prediction for each sub-block using the derived motion vector.

[0278] An SHVC upsampling interpolation filter with the same filter length and normalization factor as the HEVC motion compensated interpolation filter is used as the motion compensated interpolation filter for additional fractional pel positions. The chroma component motion vector precision is 1 / 32 sample, and the additional interpolation filter for the 1 / 32 pel fractional positions is derived by using the average of the filters for the two adjacent 1 / 16 pel fractional positions.

[0279] The AF_MERGE mode can be selected on the encoder side in the same way that regular merge mode selection is performed: a candidate list is generated preferentially, and the candidate with the smallest RD-cost is selected for comparison with the RD-costs of the different inter modes. The result of the comparison is a decision whether AF_MERGE is applied or not.

[0280] For AF_4_INTER mode, the RD cost check is used to determine which candidate motion vector pair is selected as the control point motion vector prediction (CPMVP) of the current CU. After the CPMVP of the current affine CU is determined, affine motion estimation is applied to obtain the control point motion vector (CPMV). Then, the difference between the CPMV and the CPMVP is determined.

[0281] On the encoder side, AF_6_INTER mode is only confirmed if AF_MERGE or AF_4_INTER mode is determined as the best mode in the previous mode selection stage.

[0282] In one embodiment of the present invention, the Affine Inter (Affine AMVP) mode can be implemented as follows:

[0283] 1) AFFINE_MERGE_IMPROVE: Instead of searching for the first neighboring block that is in affine mode, the improvement tries to search for the neighboring block with the largest coding unit size as an affine merge candidate.

[0284] 2) AFFINE_AMVL_IMPROVE: Add neighboring blocks that are in affine mode to the affine AMVP candidate list similar to the regular AMVP procedure.

[0285] The detailed process of generating an affine AMVP candidate list is as follows:

[0286] First, it is checked whether the lower left neighboring block has the same reference index as the current reference index using an affine motion model. If not, the left neighboring block is checked in the same way. If not, it is checked whether the lower left neighboring block has a different reference index using an affine motion model. If it does, the scaled affine motion vector is added to the reference picture list. If not, the left neighboring block is checked in the same way.

[0287] Second, the right upper neighbor, the top neighbor, and the left upper neighbor are identified in the same manner.

[0288] After the above process, if two candidates are found, the operation of generating the affine AMVP candidate list is completed. If two candidates are not found, the original operation in the JEM software is executed to generate the affine AMVP candidate list.

[0289] 3) AFFINE_SIX_PARAM: Besides the 4-parameter affine motion model, a 6-parameter affine motion model is added as an additional model.

[0290] A six-parameter affine motion model is derived via Equation 14 below.

[0291]

number

[0292] Since the motion model described above has six parameters, three motion vectors at the top left position MV_0, the top right position MV_1, and the bottom left position MV_2 are required to determine the model. The three motion vectors can be determined in a similar manner to the two motion vectors in a four-parameter affine motion model. Affine model merging is always configured as a six-parameter affine motion model.

[0293] 4) AFFINE_CLIP_REMOVE: Removes motion vector constraints for all affine motion vectors, allowing the motion compensation process to control the motion vector constraints itself.

[0294] Affine motion model

[0295] As described above, various affine motion models can be used or considered in affine inter prediction. For example, the affine motion model can express four motions as shown in FIG. 15. An affine motion model expressing three motions (translation, scale, and rotate) among the motions that can be expressed by the affine motion model can be called a similarity (or simplified) affine motion model. Depending on which of the affine motion models is used, the number of CPMVs derived and / or the method of deriving MVs in sample / sub-block units of the current block may differ.

[0296] In one embodiment of the present invention, adaptive four and six parameter motion models are used. In AF_INTER, a six-parameter motion model is provided in addition to the four parameter motion model present in JEM. The six parameter affine motion model is described as follows in Equation 15:

[0297]

number

[0298] where coefficients a, b, c, d, e, and f are affine motion parameters, and (x, y) and (x', y') are coordinates of pixel locations before and after transformation of the affine motion model. To use the affine motion model for video coding, if CPMV0, CPMV1, and CPMV2 are MVs of CP0 (upper left), CP1 (upper right), and CP2 (lower left), Equation 16 can be expressed as follows:

[0299]

number

[0300] where CPMV_0 = {v_0x, v_0y}, CPMV_1 = {v_1x, v_1y}, CPMV_2 = {v_2x, v_2y}, and w and h are the width and height of the coding block, respectively. Equation 16 is the motion vector field (MVF) of the block.

[0301] A flag is parsed at the CU level to indicate whether a 4-parameter or 6-parameter affine motion model is used when a neighboring block is coded with affine prediction. If there are no neighboring blocks coded with affine prediction, the flag is omitted and the 4-parameter model is used for affine prediction. That is, the 6-parameter model is considered when one or more neighboring blocks are coded with an affine motion model. In relation to the number of CPMVDs, two and three CPMVDs are signaled for the 4-parameter and 6-parameter affine motion models, respectively.

[0302] In one embodiment of the present invention, pattern-matched motion vector refinement can be used. In JEM's pattern-matched motion vector derivation (PMMVD, hereafter abbreviated as PMVD in the description of the JEM encoder), the decoder needs to evaluate several motion vectors (MVs) to determine the starting MV candidate for the CU-level search. In the sub-CU search, several MV candidates are added in addition to the optimal CU-level MV. The decoder needs to evaluate these MV candidates to search for the optimal MV, which requires a lot of memory bandwidth. In the proposed pattern-matched motion vector refinement (PMVR), JEM adopts the concepts of template matching and bilateral matching used in PMVD. To indicate whether PMVR is available, one PMVR_flag is signaled when skip mode or merge mode is selected. To significantly reduce memory bandwidth requirements compared to PMVD, a candidate list of MVs is generated and the starting MV candidate index is explicitly signaled if PMVR is applied.

[0303] Using the merge candidate list generation process, a candidate list is generated, but excluding sub-CU merge candidates, e.g., affine candidates and ATMVP candidates.

[0304] For bilateral matching, only one uni-prediction MV candidate is included. A bi-prediction MV candidate is split into two uni-prediction MV candidates. Similar MV candidates (MV differences less than a predefined threshold) are also removed. For CU-level search, diamond search MV refinement is performed starting from the signaled MV candidate.

[0305] Sub-CU-level search is only available in bilateral matching merge mode. The search window of the sub-CU-level search for all sub-CUs is the same as the search window of the CU-level search. Therefore, no additional bandwidth is required for the sub-CU-level search.

[0306] Template matching is also used to refine the MVP in this mode. In AMVP mode, two MVPs are generated using the HEVC MVP generation process, and one MVP index is signaled to select one of them. The selected MVP is further refined by using template matching in PMVR. When adaptive motion vector resolution (AMVR) is applied, the MVP is rounded to a suitable accuracy before template matching refinement. This refinement process is named pattern-matched motion vector predictor refinement (PMVPR). In the remainder of this document, unless otherwise defined, PMVR includes template matching PMVR, bidirectional matching PMVR, and PMVPR.

[0307] To reduce memory bandwidth requirements, PMVR is disabled for 4x4, 4x8, and 8x4 CUs. To reduce the additional memory bandwidth requirements, the search range for {template matching, bidirectional matching} for CU regions equal to 64 can be reduced to {±2, ±4}, and the search range for {template matching, bidirectional matching} for CU regions greater than 64 can be reduced to {±6, ±8}. By using all of the above methods described in the PMVR section of this document, the memory bandwidth required compared to the worst case in HEVC was reduced from 45.9x for JEM-7.0 PMVD to 3.1x for PMVR.

[0308] Affine technology used in non-QT blocks

[0309] FIG. 28 exemplarily illustrates a method for generating a prediction block and a motion vector in inter prediction using an affine motion model according to an embodiment of the present invention.

[0310] 28, a formula for deriving a motion vector when an affine motion model is applied can be seen. The motion vector can be derived based on the following Formula 17.

[0311]

number

[0312] Here, v_x denotes the x component of the sample-by-sample motion vector of the (x, y) coordinate sample in the current block, and v_y denotes the y component of the sample-by-sample motion vector of the (x, y) coordinate sample in the current block.

[0313] That is, (v_x, v_y) is the sample-based motion vector for the sample at the (x, y) coordinate. Here, a, b, c, d, e, and f represent parameters of a formula for deriving a sample-based motion vector (motion information) at the (x, y) coordinate from the control point (CP) of the current block. The CP may also be expressed as a steering pixel. The parameters may be derived from the motion information of the CP of each PU transmitted in units of PUs. The formula for deriving the sample-based motion vector derived from the motion information of the CP described above may be applied to each sample of a block, and may be derived at the position of the sample in the reference image based on the relative positions of each sample on the x and y axes. The sample-based motion vector may be derived differently depending on the block size, asymmetric or symmetric type, block position, etc. according to the QTBT(TT) block division structure. Specific examples of this will be described with reference to FIGS. 29 to 38.

[0314] FIG. 29 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points according to an embodiment of the present invention.

[0315] Referring to FIG. 29, the current block is assumed to be a 2Nx2N block. For example, the motion vector of the top-left sample in the current block may be v_0. Furthermore, samples of neighboring blocks adjacent to the current block may be used as CPs, and the motion vectors of each CP may be v_1 and v_2. That is, when the width and height of the current block are S and the coordinates of the top-left sample position of the current block are (xp, yp), the coordinates of CP0 among the CPs may be (xp, yp), the coordinates of CP1 may be (xp + S, yp), and the coordinates of CP2 may be (xp, yp + S). The motion vector of CP0 may be v_0, the motion vector of CP1 may be v_1, and the motion vector of CP2 may be v_2. The sample-based motion vectors may be derived using the motion vectors of the CPs. The sample-based motion vectors may be derived according to Equation 18 below.

[0316]

number

[0317] Here, v_x and v_y respectively represent the x and y components of the motion vector of the sample at (x, y) coordinate in the current block, v_x0 and v_y0 respectively represent the x and y components of the motion vector v_0 of CP0, v_x1 and v_y1 respectively represent the x and y components of the motion vector v_1 of CP1, and v_x2 and v_y2 respectively represent the x and y components of the motion vector v_2 of CP2. A motion vector can be derived for each sample in the current block based on its relative position within the current block using the equation for deriving the sample-based motion vector, such as Equation 18.

[0318] FIG. 30 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention.

[0319] FIG. 30 exemplarily illustrates CPs for a block partitioned into Nx2N. A formula for deriving a sample-based motion vector within the current block can be derived using the same method as in the case of the partitioning type 2Nx2N described above. In deriving the formula, a width value that matches the shape of the current block can be used. To derive the sample-based motion vector, three CPs can be derived, and the positions of the CPs can be adjusted as shown in FIG. 30. That is, when the width and height of the current block are S / 2 and S, respectively, and the coordinates of the top-left sample position of the current block are (xp, yp), the coordinates of CP0 among the CPs can be (xp, yp), the coordinates of CP1 can be (xp + S / 2, yp), and the coordinates of CP2 can be (xp, yp + S). The sample-based motion vector can be derived based on Equation 19:

[0320]

number

[0321] Here, vx and vy respectively represent the x and y components of the motion vector of the sample at (x, y) coordinate in the current block, v_x0 and v_y0 respectively represent the x and y components of the motion vector v_0 of the CP0, v_x1 and v_y1 respectively represent the x and y components of the motion vector v_1 of the CP1, and v_x2 and v_y2 respectively represent the x and y components of the motion vector v_2 of the CP2. Equation 3 shows an equation for deriving a sample-based motion vector considering that the width of the current block is S / 2. A motion vector can be derived for each sample in the current block partitioned from the CU based on the partitioning type Nx2N using the equation for deriving the sample-based motion vector, such as Equation 19, based on its relative position within the current block.

[0322] FIG. 31 is a diagram illustrating a method for performing motion compensation based on motion vectors of control points in a non-square block according to an embodiment of the present invention.

[0323] Figure 31 exemplarily illustrates a block divided based on a partitioning type 2NxN. To derive the sample-based motion vector, three CPs can be derived, and the positions of the CPs can be adjusted as shown in Figure 31, and the height can be adjusted to S / 2 according to the shape of the current block shown in Figure 31. That is, when the width and height of the current block are S and S / 2, respectively, and the coordinates of the top-left sample position of the current block are (xp, yp), the coordinates of CP0 among the CPs can be (xp, yp), the coordinates of CP1 can be (xp+S, yp), and the coordinates of CP2 can be (xp, yp+S / 2). The sample-based motion vector can be derived based on Equation 20 as follows:

[0324]

number

[0325] Here, v_x and v_y respectively represent the x and y components of the motion vector of the sample at (x, y) coordinate in the current block, v_x0 and v_y0 respectively represent the x and y components of the motion vector v_0 of the CP0, v_x1 and v_y1 respectively represent the x and y components of the motion vector v_1 of the CP1, and v_x2 and v_y2 respectively represent the x and y components of the motion vector v_2 of the CP2. Equation 4 shows a formula for deriving a sample-based motion vector considering that the height of the current block is S / 2. For each sample in the current block partitioned from the CU based on the partitioning type 2NxN using the formula for deriving a sample-based motion vector, such as Equation 4.18, a motion vector can be derived based on the relative position within the current block.

[0326] 32 to 38 are diagrams illustrating a method for performing motion compensation based on motion vectors of control points in non-square blocks according to an embodiment of the present invention.

[0327] FIG. 32 exemplarily illustrates an asymmetric current block CP. As shown in FIG. 32, the width and height of the asymmetric current block may be W and H, respectively. To derive the sample-based motion vector, three CPs may be derived for each current block, and the coordinates of the CPs may be adjusted based on the width and height along the shape of the current block, as shown in FIG. 32. That is, when the width and height of the current block are W and H, and the coordinates of the top-left sample position of each current block are (xp, yp), the coordinates of CP0 among the CPs may be set to (xp, yp), the coordinates of CP1 may be (xp+W, yp), and the coordinates of CP2 may be (xp, yp+H). In this case, the sample-based motion vector within the current block may be derived according to Equation 21, as follows:

[0328]

number

[0329] where v_x and v_y respectively represent the x and y components of the motion vector of the sample at (x, y) coordinate in the current block, v_x0 and v_y0 respectively represent the x and y components of the motion vector v_0 of CP0, v_x1 and v_y1 respectively represent the x and y components of the motion vector v_1 of CP1, and v_x2 and v_y2 respectively represent the x and y components of the motion vector v_2 of CP2. Equation 21 shows an equation for deriving an asymmetric, sample-based motion vector taking into account the width and height of the current block.

[0330] Meanwhile, according to the present invention, in order to reduce the amount of data of CP motion information indicated in block units, at least one CP motion information prediction candidate can be selected based on motion information of neighboring blocks or neighboring samples of a current block. The motion information prediction candidate can be called an affine motion information candidate or an affine motion vector candidate. The affine motion information candidate can include, for example, the contents disclosed in FIGS. 33 to 38.

[0331] ATMVP simplification

[0332] ATMVP (Advanced Temporal Motion Vector Prediction) is inter-prediction using temporal motion information candidates, and may refer to inter-prediction using motion information candidates proposed to improve existing TMVP. In embodiments of the present specification, the ATMVP is not limited to its name, and may be referred to as a sub-block temporal merge candidate, a sub-block-based temporal merge candidate, a sub-block temporal motion vector predictor, a sub-block-based temporal motion vector predictor, etc.

[0333] Specifically, TMVP, which uses the motion vector of the colPB at the right-bottom block of the current block or the center position of the current block as a temporal motion information candidate, may not reflect the motion within the screen. On the other hand, when the ATMVP is applied, the motion vector of the colPB at the position indicated by the motion vector of the surrounding block is used as the ATMVP candidate.

[0334] As an example of how the ATMVP is applied, the merge candidates are checked in order of construction, and then the motion vector (temporal vector) of the available spatial neighbor block is found first, and the position pointed to by the temporal vector in the reference picture is derived as a col-PB (ATMVP candidate). Also, the motion vector of the corresponding block is used for each sub-block using the temporal vector. In this case, if a specific sub-block does not have an MV, the MV of the block located at the center of the corresponding block is used as the MV of the unavailable sub-block and stored as a representative MV.

[0335] Furthermore, various ATMVP simplification schemes have been proposed that use the ATMVP but further reduce memory usage.

[0336] In one embodiment, the reference pictures (specified in the slice segment header) for finding the reference block are limited to collocated pictures, thereby reducing memory usage. As an example, a maximum of four reference pictures are used. To find the reference block, a temporal vector can be derived from one of the spatial candidates according to the scan order. When the current candidate has the same collocated picture as the reference picture, the search process can end.

[0337] In one embodiment, modifications of ATMVP and / or STMVP sub-block merge modes can be applied.

[0338] 1. Signaling is added at the picture / slice-level so that different sub-block sizes are used for ATMVP / STMVP-based motion information derivation.

[0339] 2. A simplified ATMVP-based motion information derivation is used by ensuring that one same-position picture is used.

[0340] In one embodiment, it is proposed to support picture / slice-level adaptation of the granularity of motion fields derived by ATMVP and STMVP modes. Specifically, the default value of the sub-block size used to derive ATMVP and STMVP motion parameters for a slice can be signaled in a sequence parameter set (sps) that indicates the sps. Additionally, a flag is signaled in the slice's segment header. If this flag is 0, it indicates that the default sub-block size as specified in sps is used for ATMVP / STMVP-based motion derivation in the current slice. Otherwise (i.e., if this flag is set to 1), another syntax element can be signaled in the slice's segment header to specify the ATMVP / STMVP sub-block size to use for the slice.

[0341] In one embodiment, a simple encoder method is used to determine the ATMVP / STMVP sub-block size of the current picture. One of two ATMVP / STMVP sub-block sizes, 4 and 8, is selected based on the average size of the ATMVP / STMVP blocks from the last coded picture in the same temporal layer. Assume that the previously coded picture is the i-th picture in the k-th temporal layer and contains N CUs coded in ATMVP and STMVP modes. Assume also that the sizes of these CUs are S_0, S_1, ..., S_(N-1). The average size of ATMVP / STMVP CUs is calculated as σ^k = (Σ_(i=0)^(N-1)S_i) / N. Then, when coding the (i+1)-th picture in the same k-th temporal layer, its ATMVP / STMVP sub-block size g_(i+1)^k can be determined according to the following Equation 22:

[0342]

number

[0343] In one embodiment, for the first picture in each temporal layer, the size of ATMVP / STMVP sub-block 7 may always be set to four.

[0344] In one embodiment, the ATMVP can be derived using the following two steps.

[0345] (1) If a neighboring CU is available and the MV of this CU is different from the MV from the existing candidate list, the encoder / decoder can add motion vectors from the spatially neighboring CUs in the order of A1, B1, B0, and A0, as shown in the previously described Figure 12. The number of available spatial candidates can be represented by N0.

[0346] (2) The first MV candidate from the N0 spatial candidates is used to determine the location (by adding the displacement due to the MV) to fetch the co-position picture and motion of each sub-block. If N0 is 0, the same location with the co-position picture and zero motion signaled in the slice header is used to fetch the motion of each sub-block.

[0347] Co-located pictures of different CUs in ATMVP may not always be the same when multiple reference pictures are used. Having different co-located pictures derived from ATMVP for different CUs in the current picture means that motion fields of multiple reference pictures need to be fetched, which is undesirable due to increased memory bandwidth. Therefore, in one embodiment, a simplified design is provided that uses the same co-located picture as the co-located picture derived from ATMVP as in HEVC signaled in the slice header. At the block level, if the reference picture of a neighboring block A is different from this co-located picture, the MV of block A is scaled using the HEVC temporal MV scaling method, and the scaled MV of block A is used for ATMVP.

[0348] The motion vector used to fetch the motion field at the co-located picture Rcol is denoted as MVcol. To minimize the impact of MV scaling, the MV from the spatial candidate list used to derive MVcol is selected in this answer in the following way: If the reference picture of the candidate MV is a co-located picture, this MV is selected and used as MVcol without any scaling. Otherwise, the MV with the reference picture closest to the co-located picture is selected to derive MVcol with scaling.

[0349] HMVP (History-based Motion Vector Prediction) general

[0350] Generally, video compression technologies exploit spatial and temporal redundancy as two main techniques. For example, High Efficiency Video Coding (HEVC) and VVC both use two motion compression techniques based on inter-coding: merge motion and advanced motion vector prediction (AMVP). Various modifications have been discussed to improve these two prediction modes. These include increasing the number of candidates, exploring more spatially extended candidates, and testing temporal candidates in non-traditional locations. These two techniques primarily build a list of possible candidates, minimize the rate distortion (RD) cost, and signal the selected candidate from the bitstream.

[0351] In particular, in recent video compression technology, HMVP (History-based Motion Vector Prediction) is being discussed, which stores motion information of previously coded blocks and uses the stored motion information to predict the motion of blocks to be coded subsequently. Such HMVP can be added to a merge list (or merge candidate list) or an AMVP list (or AMVP candidate list).

[0352] The decoder maintains a look-up table (LUT) that operates in a FIFO (first in first out) system (or manner) for HMVP. In this specification, the LUT is not limited to its name and may be referred to as a table, HMVP table, HMVP candidate table, buffer, HMVP buffer, HMVP candidate buffer, HMVP list, HMVP candidate list, etc. Specifically, when a non-affine prediction unit (PU) (or coding unit (CU)) is decoded, its motion information is stored in the LUT, and the decoder decodes the next PU. At this time, the stored motion information may include x (horizontal) and y (vertical) direction motion vectors, reference index information, mode information, etc.

[0353] The decoder can maintain a LUT that stores motion information of progressively decoded non-affine candidates. The size of the LUT can be limited to a predefined number of S candidates. In one embodiment, the LUT can be reset from the start of a slice, the start of a CTU row, or the start of a CTU.

[0354] HMVP can be applied in both merge mode and AMVP mode. A merge list can have B candidates, and an AMVP list can have two candidates. In conventional video compression technology, a merge list consists of the following candidates: i) spatial candidates, ii) temporal candidates, iii) bidirectional prediction (Bi-Pred) candidates, and iv) zero motion candidates. Recently, a method in which ATMVP (advanced motion vector prediction) is considered as an additional candidate has been discussed. For example, ATMVP candidates can be inserted into the merge list before temporal candidates. Candidates in the merge list are added to the merge list until the maximum merge list size is reached. Duplicate candidates may not be added to the merge list.

[0355] The AMVP list can be populated with two candidates. As an example, one of the two candidates can be selected from the available spatial candidates and the second candidate can be selected from the temporal candidates, and if the list is not filled, a zero motion vector candidate can be added.

[0356] HMVP is applied on a FIFO basis, where candidates are removed from the table in the same order as they were entered into the LUT.

[0357] In one embodiment, when HMVP is applied to constructing a merge list, an HMVP candidate can be inserted (or added) in the third position of the list as follows:

[0358] 1. Spatial Candidate

[0359] 2. Temporal Candidate

[0360] 3. Up to S HMVP Candidates for a LUT

[0361] 4. Combined Bi-Pred Candidate

[0362] 5. Zero Motion Vector Candidate

[0363] In one embodiment, when HMVP is applied to constructing an AMVP list, HMVP can be inserted in the third position after the time candidate as follows:

[0364] 1. Spatial Candidate

[0365] 2. Temporal Candidate

[0366] 3. Up to K HMVP Candidates

[0367] 4. Zero Motion Vector Candidate

[0368] FIG. 39 is a flowchart illustrating a method for storing HMVP according to an embodiment of the present specification.

[0369] Referring to FIG. 39, the decoder decodes the current PU (or CU) (S3901).

[0370] The decoder checks whether the current PU is a block coded in non-affine mode (S3902). To facilitate the use of HMVP candidates, if the current PU is a block coded in affine mode, the decoder does not store the motion information of the current PU in a table.

[0371] If the current PU is a block coded in non-affine mode, the decoder stores (or updates) the motion information of the current PU in a table (S3903).

[0372] In the present embodiment, the HMVP table can be updated in two ways: i) unconstrained FIFO; and ii) constrained FIFO. In electronics, duplicate information can exist, but no pruning process is applied. This contributes to reducing the overall process complexity. This will be explained with reference to the following diagram:

[0373] FIG. 40 is a diagram illustrating an HMVP table operating in a non-limiting FIFO manner according to an embodiment of the present specification.

[0374] 40, candidates added to the table are added to the end (right side) of the table, while candidates removed from the table based on a FIFO method are located at the front end (left side, oldest candidate) of the table.

[0375] If the table at index L-1 (i.e., the edge) is not completely filled with a pre-defined maximum number of candidates, a new candidate is added without removing any candidates. On the other hand, if the table is already completely filled, i.e., the maximum number of candidates in the table is reached, the oldest candidate at the front end of the table is removed and a new candidate is added.

[0376] FIG. 41 is a diagram illustrating an HMVP table that operates in a restrictive FIFO manner according to an embodiment of the present specification.

[0377] 41, when a restrictive FIFO is used, pruning is performed if adding a new candidate causes any duplication. For example, if a candidate with duplicate motion information exists in the table, the duplicate candidate in the table can be removed and the motion information of the current candidate can be added.

[0378] Example 1

[0379] For HMVP candidates, in many cases, recent history MVs may overlap with the motion information of spatial candidates (or spatial neighbor candidates). Therefore, in this embodiment, when adding HMVP candidates to the AMVP or merge list, we propose a method to set the addition order of the candidates to be different from the order of the HMVP LUT indexes.

[0380] According to the embodiments of the present specification, by adaptively adjusting HMVP candidates, a candidate list can be efficiently constructed, thereby reducing the number of signaling bins used for binarization and improving coding efficiency.

[0381] That is, the HMVP candidates added to the merge list or AMVP list may not be limited by their index within the HMVP list. As an example, Table 2 below illustrates a method for changing the order in which HMVP candidates are added to the AMVP or merge list.

[0382] [Table 2]

[0383] Referring to Table 2, as mentioned above, the most recently inserted HMVP candidate is likely to have the same motion information as the spatial candidate, so taking this into consideration, the addition order of the HMVP candidates can be predefined regardless of the HMVP index.

[0384] Also, in one embodiment, HMVP candidates can be added to the merge or AMVP list starting from the nth candidate in the list. Table 3 below illustrates a modified order for adding candidates to the AMVP or merge list.

[0385] [Table 3]

[0386] Referring to Table 3, HMVP candidates can be added to the merge list or AMVP list from the second index.

[0387] In one embodiment, information about the order in which HMVP candidates are added within a look-up table (LUT) can be signaled from the encoder to the decoder. For example, such ordering information can be conveyed via a high level syntax (HLS). The high level syntax can be, for example, a data header of a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit, and / or other suitable syntax.

[0388] Table 4 below illustrates a high-level syntactic "syntax" structure to which the methods proposed herein can be applied.

[0389] [Table 4]

[0390] Referring to Table 4, set_HMVP_order_flag being 1 indicates that set_HMVP_order_flag is present in the slice header of a non-IDR picture from the VCS, and set_HMVP_order_flag being 0 indicates that set_HMVP_order_flag is not present in the slice header and adaptive HMVP is not used in the VCS.

[0391] Table 5 below illustrates a slice segment header syntax structure to which the method proposed herein can be applied.

[0392] [Table 5]

[0393] Referring to Table 5, slice_HMVP_idx means the index of the candidate order to be used. For example, slice_HMVP_idx of 0 can represent a basic HMVP order such as 0, 1, 2, 3. Similarly, an index value of 1 is used to represent an HMVP order of 3, 2, 1, 0.

[0394] Example 2

[0395] In an embodiment of the present specification, a method is proposed in which a long-term list is used for motion example prediction in addition to the HMVP LUT. This can increase the number of HMVP candidates that can be maintained. As an example, a two-HMVP table can be considered, where one table stores general HMVP candidates and the other table is used as a long-term list that stores candidates that need further maintenance.

[0396] The following example shows how to initialize and configure a long-term list (or long-term HMVP list).

[0397] After decoding the first CTU in a CTU row, one or more history MVs of subsequent CTUs may be added to the long-term HMVP LUT. Such a long-term HMVP LUT may not be used or updated until the next CTU row.

[0398] At the start of the next CTU row, the long-term HMVP LUT is used to initialize the regular HMVP LUT, because the HMVP candidates for CTUs from the start of the CTU row can be more correlated than the history MVs at the end of the previous CTU row.

[0399] The process can be repeated.

[0400] FIG. 42 is a diagram illustrating an HMVP LUT and a long-term HMVP LUT according to an embodiment of the present specification.

[0401] Referring to Figure 42, there are two LUTs. One is an HMVP LUT (or general HMVP LUT, short-term HMVP LUT) and the other is a long-term HMVP LUT. When an HMVP candidate is added to both of the merged AMVP lists, it can be added from either the HMVP LUT or the long-term LUT, as shown in Figure 42.

[0402] The use of the aforementioned long-term LUT can be indicated through higher-level syntax by introducing new syntax elements, such as those present in the headers of a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit, and / or other appropriate syntax data.

[0403] Example 3

[0404] In the embodiment of this specification, we propose a method that considers the flexibility of decoding when adding HMVP candidates to the HMVP LUT. The encoder / decoder can consider the decision of one or more characteristics of a PU (or CU).

[0405] In one embodiment, the encoder / decoder may consider the following factors when adding an HMVP candidate to the table: The encoder / decoder may consider characteristics such as the mode of the PU (e.g., merge mode, affine mode, AMVP mode, etc.) and / or block size, individually or in combination, to add the candidate. In one embodiment, other characteristics may also be considered. For example, the merge type (e.g., spatial candidate or temporal candidate) for which the HMVP LUT is to be updated, whether the PU is a sub-PU, etc. may be considered as candidate selection criteria. The above selection criteria may be determined to reduce overlap with previous history (or previous HMVP). For example, if a PU is coded in merge mode and the merge type is spatial merge, the decoder may not update the motion information of the PU to the HMVP LUT.

[0406] FIG. 43 is a diagram illustrating an example of a method for updating an HMVP LUT according to an embodiment of the present specification.

[0407] Referring to FIG. 43, the encoder / decoder obtains motion information of a coded candidate (S4301).

[0408] The encoder / decoder evaluates whether to update the motion information of the candidate to the LUT based on predefined criteria (S4302). The criteria may include at least one of the following characteristics: the mode of the candidate (e.g., merge mode, affine mode, AMVP mode, etc.), the block size of the candidate, and / or the merge type of the candidate.

[0409] The encoder / decoder updates the LUT based on the decision criterion (S4303). That is, if the candidate satisfies a predefined decision criterion, the encoder / decoder can add motion information of the candidate to the LUT.

[0410] Example 4

[0411] In the present embodiment, we propose a redundancy check restriction for adding HMVP candidates to the merge list (or AMVP list). The redundancy check restriction can be realized in several different ways.

[0412] In one embodiment, the encoder / decoder may limit the number of pruning checks for a specific number of first candidates in the merge list. The encoder / decoder may limit the number of pruning checks for candidates from the first candidate in the merge list to a specific number of candidates. For example, the encoder / decoder may perform a pruning process for candidates from the first candidate in the merge list to a specific number of candidates, and in this case, the number of HMVP candidates to be subject to the pruning check may be limited to a predefined number.

[0413] In one embodiment, the encoder / decoder may limit the pruning check by performing the pruning check on a specific type of merge candidate in the merge list. For example, the encoder / decoder may perform the pruning check only on spatial candidates in the merge list when adding an HMVP candidate. Alternatively, for example, the encoder / decoder may perform the pruning check on only a portion of spatial candidates in the merge list when adding an HMVP candidate. The portion of spatial candidates may be predefined. For example, the predefined portion of spatial candidates may be at least one of left-side neighboring spatial candidates and / or upper-side neighboring spatial candidates. However, the embodiments of the present specification are not limited thereto, and other types may be combined to limit the targets of the pruning check.

[0414] FIG. 44 is a diagram illustrating a method for limiting the number of HMVP candidates that are subject to pruning checks according to an embodiment of the present specification.

[0415] Referring to FIG. 44, in the embodiment of the present specification, the number of HMVP candidates that are subject to pruning check can be limited to M times.

[0416] When constructing a merge list using HMVP candidates, the encoder / decoder can check whether there is redundancy in motion information between the first M candidates in the HMVP LUT and the merge candidates in the merge list.

[0417] Alternatively, when adding the motion information of the currently decoded PU to the HMVP LUT, the encoder / decoder can check whether there is redundancy between the first M candidates in the HMVP LUT and the motion information of the decoded PU.

[0418] FIG. 45 illustrates an example method for performing a pruning check according to an embodiment of the present disclosure.

[0419] Referring to FIG. 45, the encoder / decoder obtains motion information of the decoded candidate and determines (or decodes) the number of pruning checks (S4501, S4502).

[0420] The encoder / decoder performs pruning checks based on the determined number of pruning checks (S4503).

[0421] In one embodiment, information related to pruning checks can be signaled through higher-level syntax in a manner similar to that described in Tables 4 and 5. In this case, a syntax element transmitted from the encoder to the decoder can be included in any header to indicate the number of pruning checks that need to be performed. The higher-level syntax can be, for example, a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit, and / or other suitable syntax data header.

[0422] Example 5

[0423] In one embodiment of the present specification, we propose an efficient method for selecting HMVP candidates. When inserting a history motion vector candidate (i.e., an HMVP candidate) into a merge list (or AMVP list), a pruning check can be performed to ensure that the HMVP candidate does not overlap with existing merge lists.

[0424] To perform a full redundancy check between an M-sized merge list and an N-sized history LUT, (M-1)xN checks are required. Therefore, in the present embodiment, the number of HMVP candidates may depend on the number of existing parallel merge candidates. For example, the number of HMVP candidates may depend on the number of spatial candidates present in the merge list. Or, for example, the number of HMVP candidates may depend on the number of spatial and temporal candidates present in the merge list.

[0425] If there are more merge candidates in the merge list, the number of HVMP candidates for which pruning checks are performed can be reduced based on specific criteria (or rules) based on the number of merge candidates in the merge list and / or the number of HVMPs, thereby reducing the number of duplicate checks in the worst case.

[0426] For example, for a merge list with a size (or length) of 6, the merge list can contain up to 5 spaces or other merge candidates if the merge list is not full. Inserting an HMVP candidate from 6 HMVP lists can require 30 duplicate checks in the worst case.

[0427] In one embodiment, an example related to limiting the number of HMVPs to be checked is shown in Equation 23 and Table 6 below.

[0428]

number

[0429] [Table 6]

[0430] Referring to Table 6, by limiting the number of HMVPs subject to pruning checks to two, the number of duplicate checks for adding HMVPs can be reduced from 30 to 12 in the worst case.

[0431] Example 6

[0432] In one embodiment of this specification, we propose a method for constructing a merge list using history-based spatial temporal motion vector prediction (H-STMVP). H-STMVP indicates a candidate derived from the average of two history-based spatial HMVPs and TMVPs. The two spatial HMVPs can be obtained from the HMVP buffer, and the TMVP can be obtained from the current merge list. Here, the spatial candidate can be a candidate obtained from the last two coded MVs in decoding order before the current block.

[0433] For example, the last coded MV (referred to herein as MV_L), the penultimate coded MV (referred to herein as MV_(L-1)), and MV_TMVP are used to generate H-STMVP candidates to be inserted into the merge list.

[0434] When all three candidates mentioned above are used, the MV to be added to the merge list can be calculated by the following Equation 24.

[0435]

number

[0436] In one embodiment, if only two of the three candidates are available, the H-STMVP can be generated by averaging only the two candidates. Similarly, if only one candidate is available, only that one candidate is used. If no candidates are available, the H-STMVP is not used to construct the merge list.

[0437] In one embodiment of this specification, a method for obtaining motion vectors of H-STMVP candidates using other methods other than Equation 24 is proposed.

[0438] For example, instead of averaging three or more candidates at once, it may be computationally simpler to average the spatial candidates first and then use this result to re-average two candidates. An example of this is equivalent to the following equation:

[0439]

number

[0440] Alternatively, the average value can be obtained as follows:

[0441]

number

[0442]

number

[0443]

number

[0444] The encoder / decoder can first average two candidates, then use a third candidate, and finally average the resulting values, as in Equations 25 to 27. Alternatively, the encoder / decoder can assign higher importance / weight to a candidate, i.e., MV_L, by applying a shift operation by 2, as in Equation 28. Using the above-described Equations 25 to 28, the average can be derived using only shift operations, without division.

[0445] Example 7

[0446] In one embodiment of the present invention, we propose a method for deriving H-STMVP using an arbitrary number (n) of spatial candidates instead of two history-based spatial candidates. These n candidates do not necessarily have to be in consecutive decoding order. They can be selected based on any or some rules.

[0447] Therefore, the above-mentioned Equation 24 can be expressed in a more general way as the following Equation 29.

[0448]

number

[0449] In another embodiment, assuming five spatial candidates are used, the weights applied to the temporal candidates can be increased to minimize the impact of the increased spatial candidates for generating H-STMVP candidates and appropriately reflect the spatial and temporal candidates.

[0450] Therefore, to achieve this, the spatial candidates are averaged together using the following Equation 30, and then the result is used to average the MV_TMVP, thereby achieving the aforementioned objective.

[0451]

number

[0452] Example 8

[0453] In one embodiment of the present specification, we propose a method for adding weights (or weighting factors) to motion vector candidates used to derive H-STMVP. In this case, the weights can be determined empirically, by considering the time distance to a fixed reference frame, or by considering the position in a history table. For example, a new candidate can have more weight than a previous candidate.

[0454] That is, in this embodiment, the above-mentioned Equation 24 can be expressed as the following Equation 31.

[0455]

number

[0456] In this case, the weights may have the same value or may have unevenly distributed values.

[0457] Example 9

[0458] In one embodiment of this specification, we propose a method for scaling the motion vectors used to derive H-STMVP candidates as a single reference picture.

[0459] FIG. 46 is a diagram illustrating a method for deriving H-STMVP candidates using motion vectors that refer to different reference pictures according to one embodiment of the present specification.

[0460] Referring to Figure 46, it is assumed that the MV_L, MV_L-1, and MV_TMVP candidates each refer to (or point to) different reference pictures. That is, Figure 46 shows that each candidate used to generate the H-STMVP candidate can have a different reference index, and consequently, a different reference frame.

[0461] Since frames with neighboring reference frames may have a greater influence on the motion vectors of H-STMVP, the averages of Equations 24 to 31 may result in uneven results. Therefore, for even comparison and reflection, we propose a method of scaling all motion vectors to a single reference frame.

[0462] In this case, the encoder can perform RD optimization to determine which single frame is best to use as a reference frame. For example, the selected reference frame can be signaled in the slice header, similar to the TMVP array index present in the slice header. For example, it may be possible to generate the reference frame to use using a fixed rule. Alternatively, the reference frame can be scaled to the first available reference frame from L0, or scaled by the current picture order count, for example.

[0463] In one embodiment, to achieve the above-mentioned objectives, an encoder can transmit information about a single fixed picture to a decoder using a higher level syntax (HLS), which can be part of a sequence parameter set, a picture parameter set, a slice header, a coded tree unit, and / or a header of other data. For example, a higher level syntax structure such as that shown in Table 7 and / or Table 8 below can be defined.

[0464] [Table 7]

[0465] Referring to Table 7, set_HSTMVP_ref_pic_flag equals 1 to indicate that set_HSTMVP_idx is present in the slice header of a non-IDR picture from the CVS. set_HSTMVP_ref_pic_flag equals 0 to indicate that set_HSTMVP_idx is not present in the slice header.

[0466] [Table 8]

[0467] Referring to Table 8, slice_HMVP_idx specifies the reference index. As an example, the reference index can be selected for list L0.

[0468] Although the embodiments of the present specification have been described above separately for the sake of convenience, the present invention is not limited to this. That is, the embodiments described above in Examples 1 to 9 may be implemented independently, or one or more of the embodiments may be implemented in combination.

[0469] FIG. 47 is a flowchart illustrating a method for processing a video signal based on inter prediction according to an embodiment to which the present invention is applied.

[0470] Referring to Figure 47, for convenience of explanation, the description will focus on the decoder, but the present invention is not limited thereto, and the inter-prediction based video signal processing method according to the embodiments of this specification can be performed in the same way in the encoder and the decoder.

[0471] The decoder constructs a first merge candidate list for the current block using spatial merge candidates and temporal merge candidates for the current block (S4701).

[0472] The decoder constructs a second merge candidate list by adding history-based merge candidates that indicate motion information of previously coded blocks of the current block to the first merge candidate list (S4702).

[0473] The decoder obtains a merge index indicating a merge candidate in the second merge candidate list that is to be applied to the current block (S4703).

[0474] The decoder generates a prediction block for the current block using motion information of the merge candidate indicated by the merge index (S4704).

[0475] As described above, the history-based merge candidate can be added to the first merge candidate list if it has motion information that does not overlap with predefined merge candidates among the merge candidates included in the first merge candidate list.

[0476] As described above, the history-based merge candidate can be added to the first merge candidate list if it has motion information that does not overlap with a predefined number of merge candidates among the merge candidates included in the first merge candidate list.

[0477] As previously mentioned, the history-based merge candidate may be added to the first merge candidate list if it has motion information that does not overlap with a particular spatial merge candidate included in the first merge candidate list.

[0478] As described above, the history-based merge candidates may be derived from a predefined number of candidates in a history-based buffer that stores the history-based merge candidates. For example, the predefined number may be two.

[0479] As described above, the step of constructing the second merge candidate list may further include the step of adding a history-based spatio-temporal merge candidate, and the history-based spatio-temporal candidate may be derived using the two history-based merge candidates and the temporal merge candidate.

[0480] As described above, the motion vector of the history-based spatiotemporal candidate can be derived by averaging the motion vectors of the two history-based merge candidates and the motion vector of the temporal merge candidate.

[0481] 48 shows an example of a block diagram of an apparatus for processing a video signal according to an embodiment of the present specification. The video signal processing apparatus of FIG. 48 may correspond to the encoding apparatus 100 of FIG. 1 or the decoding apparatus 200 of FIG. 2.

[0482] The image processing device 4800 for processing an image signal includes a memory 4820 for storing an image signal, and a processor 4810 coupled to the memory for processing the image signal.

[0483] The processor 4810 according to the embodiment of the present specification may be configured with at least one processing circuit for processing a video signal, and may process the video signal by executing a command for encoding or decoding the video signal. That is, the processor 4810 may encode original video data or decode an encoded video signal by executing the encoding or decoding method described above.

[0484] FIG. 49 shows a video coding system to which the present invention is applied.

[0485] A video coding system may include a source device and a receiving device. The source device may transmit encoded video / image information or data to the receiving device via a digital storage medium or a network in the form of a file or streaming.

[0486] The source device may include a video source, an encoding apparatus, and a transmitter. The receiving device may include a receiver, a decoding apparatus, and a renderer. The encoding apparatus may be called a video / video encoding apparatus, and the decoding apparatus may be called a video / video decoding apparatus. The transmitter may be included in the encoding apparatus. The receiver may be included in the decoding apparatus. The renderer may include a display unit, which may be a separate device or an external component.

[0487] A video source can acquire video / images through a video / image capture, synthesis, or generation process. A video source can include a video / image capture device and / or a video / image generation device. A video / image capture device can include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device can include, for example, a computer, a tablet, or a smartphone, and can (electronically) generate video / images. For example, a virtual video / image can be generated through a computer, in which case the video / image capture process can replace the process by which the associated data is generated.

[0488] An encoding device can encode input video / images. The encoding device can perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0489] The transmitter can transmit the encoded video / image information or data output in the form of a bitstream to a receiver of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter can include elements for generating a media file in a predetermined file format and elements for transmission via a broadcasting / communication network. The receiver can extract the bitstream and transmit it to a decoding device.

[0490] The decoding device can decode the video / image by performing a series of steps such as inverse quantization, inverse transform, and prediction that correspond to the operations of the encoding device.

[0491] The renderer can render the decoded video / image, and the rendered video / image can be displayed via a display unit.

[0492] FIG. 50 shows a structural diagram of a content streaming system as an embodiment to which the present invention is applied.

[0493] Referring to FIG. 50, a content streaming system to which the present invention is applied can broadly include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0494] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, or video camera into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, or video camera directly generates a bitstream, the encoding server can be omitted.

[0495] The bitstream can be generated by an encoding method or a bitstream generating method to which the present invention is applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0496] The streaming server transmits multimedia data to a user device via a web server based on a user request, and the web server acts as an intermediary that informs the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, which then transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, which controls commands and responses between devices in the content streaming system.

[0497] The streaming server can receive content from a media storage and / or an encoding server. For example, if content is received from the encoding server, the content can be received in real time. In this case, the streaming server can store the bitstream for a certain period of time to provide a smooth streaming service.

[0498] Examples of the user device include a mobile phone, a smartphone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, a head mounted display (HMD)), a digital TV, a desktop computer, and a digital signage.

[0499] Each server in the content streaming system can be operated as a distributed server, and in this case, data received by each server can be processed in a distributed manner.

[0500] As described above, the embodiments of the present invention may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in the drawings may be implemented and executed on a computer, processor, microprocessor, controller, or chip.

[0501] In addition, the decoder and encoder to which the present invention is applied may be included in a multimedia broadcasting transmitting / receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a video camera, a video-on-demand (VoD) service providing device, an over-the-top (OTT) video device, an internet streaming service providing device, a three-dimensional (3D) video device, an image telephone video device, a medical video device, etc., and are used to process video signals or data signals. For example, over-the-top (OTT) video devices may include a game console, a Blu-ray player, an internet-connected TV, a home theater system, a smartphone, a tablet PC, a digital video recorder (DVR), etc.

[0502] Furthermore, a processing method according to the present invention can be produced in the form of a computer-executable program and stored in a computer-readable recording medium. Multimedia data having a data structure according to the present invention can also be stored in a computer-readable recording medium. The computer-readable recording medium includes any type of storage device or distributed storage device in which computer-readable data is stored. The computer-readable recording medium can include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable recording medium can also include media realized in the form of a carrier wave (e.g., transmission over the Internet). The bitstream generated by the encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0503] Furthermore, the embodiments of the present invention may be realized as a computer program product by program code, which can be executed by a computer according to the embodiments of the present invention. The program code may be stored on a computer-readable carrier.

[0504] The above-described embodiments are combinations of elements and features of the present invention in a predetermined form. Each element or feature should be considered optional unless otherwise explicitly stated. Each element or feature may be implemented without being combined with other elements or features. It is also possible to combine some elements and / or features to form an embodiment of the present invention. The order of operations described in the embodiments of the present invention may be changed. Some elements or features of any embodiment may be included in other embodiments, or may be replaced with corresponding elements or features of other embodiments. It is obvious that claims that do not have an explicit reference relationship in the claims may be combined to form an embodiment, or may be included as new claims by amendment after filing.

[0505] Embodiments of the present invention may be implemented by various means, such as hardware, firmware, software, or a combination thereof. In a hardware implementation, an embodiment of the present invention may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0506] In the case of a firmware or software implementation, an embodiment of the present invention may be implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above. The software code may be stored in a memory and driven by a processor. The memory may be located inside or outside the processor and may transmit and receive data to and from the processor by various means known in the art.

[0507] It will be apparent to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential characteristics of the present invention. Therefore, the above detailed description should not be construed as limiting in all respects, but should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention. [Industrial Applicability]

[0508] The above-described preferred embodiments of the present invention have been disclosed for illustrative purposes, and those skilled in the art will appreciate that various other embodiments may be improved, modified, substituted, or added within the technical spirit and scope of the present invention as disclosed in the appended claims.

Claims

1. A step of generating a first merge candidate list for a current block based on a plurality of spatial candidates including a lower left corner peripheral spatial candidate, a left side peripheral spatial candidate, a top right corner peripheral spatial candidate, an upper peripheral spatial candidate, and an upper left corner peripheral spatial candidate, and a temporal candidate for the current block, wherein the plurality of spatial candidates precede the temporal candidate in the first merge candidate list; generating a second merge candidate list by adding history-based candidates including motion information for blocks decoded before the current block to the first merge candidate list; purging merge indexes in the second merge candidate list that indicate merge candidates to be applied to the current block; generating a prediction block of the current block based on motion information of the merge candidate indicated by the merge index; generating a residual block of the current block from a bitstream of a video signal; generating a reconstruction block of the current block based on the predicted block and the residual block; The step of generating the second merge candidate list comprises: checking motion information of the history-based candidate with motion information of the left surrounding space candidate and the top surrounding space candidate included in the first merge candidate list; adding the history-based candidate only if the checking indicates that the history-based candidate has motion information that does not overlap with the motion information of the left surrounding spatial candidate and the top surrounding spatial candidate included in the first merge candidate list; the checking step is not performed on the lower left corner surrounding space candidate, the upper right corner surrounding space candidate, and the upper left corner surrounding space candidate included in the first merge candidate list; The method, wherein the checking step is performed between a predefined number of history-based candidates included in a history-based merge candidate list and only the left surrounding spatial candidate and the top surrounding spatial candidate.

2. A step of generating a first merge candidate list for a current block based on a plurality of spatial candidates including a lower left corner peripheral spatial candidate, a left peripheral spatial candidate, a top right corner peripheral spatial candidate, an upper peripheral spatial candidate, and an upper left corner peripheral spatial candidate, and a temporal candidate for the current block, wherein the plurality of spatial candidates precede the temporal candidate in the first merge candidate list; generating a second merge candidate list by adding history-based candidates including motion information for blocks decoded before the current block to the first merge candidate list; generating a merge index indicating a merge candidate to be applied to the current block in the second merge candidate list; generating a prediction block of the current block based on motion information of the merge candidate indicated by the merge index; generating a residual block of the current block based on the predicted block; The step of generating the second merge candidate list comprises: checking motion information of the history-based candidate with motion information of the left surrounding space candidate and the top surrounding space candidate included in the first merge candidate list; adding the history-based candidate only if the checking indicates that the history-based candidate has motion information that does not overlap with the motion information of the left surrounding spatial candidate and the top surrounding spatial candidate included in the first merge candidate list; the checking step is not performed on the lower left corner surrounding space candidate, the upper right corner surrounding space candidate, and the upper left corner surrounding space candidate included in the first merge candidate list; The method, wherein the checking step is performed between a predefined number of history-based candidates included in a history-based merge candidate list and only the left surrounding spatial candidate and the top surrounding spatial candidate.

3. obtaining a bitstream for an image; transmitting data including the bitstream; The bitstream comprises: generating a first merge candidate list for the current block based on a plurality of spatial candidates, including a lower left corner surrounding spatial candidate, a left surrounding spatial candidate, a top right corner surrounding spatial candidate, an upper surrounding spatial candidate, and an upper left corner surrounding spatial candidate, and a temporal candidate for the current block, wherein the plurality of spatial candidates precede the temporal candidate in the first merge candidate list; generating a second merge candidate list by adding history-based candidates including motion information for blocks decoded before the current block to the first merge candidate list; generating a merge index indicating a merge candidate to be applied to the current block in the second merge candidate list; generating a prediction block of the current block based on motion information of the merge candidate indicated by the merge index; generating a residual block of the current block based on the predicted block; The step of generating the second merge candidate list comprises: checking motion information of the history-based candidate with motion information of the left surrounding space candidate and the top surrounding space candidate included in the first merge candidate list; adding the history-based candidate only if the checking indicates that the history-based candidate has motion information that does not overlap with the motion information of the left surrounding spatial candidate and the top surrounding spatial candidate included in the first merge candidate list; the checking step is not performed on the lower left corner surrounding space candidate, the upper right corner surrounding space candidate, and the upper left corner surrounding space candidate included in the first merge candidate list; The method of transmitting, wherein the checking step is performed between a predefined number of history-based candidates included in a history-based merge candidate list and only the left surrounding spatial candidate and the top surrounding spatial candidate.

Images