Method and apparatus for processing video signals based on interpretation

Abstract

Providing a method for processing video signals. [Solution] The method includes the steps of: constructing a first merge candidate list for the current block using spatial merge candidates and temporal merge candidates for the current block; constructing a second merge candidate list by adding history-based merge candidates indicating motion information of previously encoded blocks for the current block to the first merge candidate list; obtaining a merge index in the second merge candidate list that indicates a merge candidate to be applied to the current block; and generating a predicted block for the current block using the motion information of the merge candidate indicated by the merge index, wherein a history-based merge candidate is added to the first merge candidate list if it has motion information that does not overlap with predefined merge candidates from among the merge candidates included in the first merge candidate list.

Description

Technical Field

[0001] The embodiments of this specification relate to a method and an apparatus for processing a video signal using inter prediction, and more particularly, to a method and an apparatus for performing inter prediction using history-based motion vector prediction.

Background Art

[0002] Compression encoding refers to a series of signal processing techniques for transmitting digitized information over a communication line or storing it in a form suitable for a storage medium. Media such as video, images, and audio can be the target of compression encoding, and in particular, the technique of performing compression encoding on video is called video compression.

[0003] Next-generation video content will come to have features called high spatial resolution, high frame rate, and high dimensionality of scene representation. To process such content, it brings a significant increase in terms of memory storage, memory access rate, and processing power.

[0004] Therefore, it is necessary to design coding tools for more efficiently processing next-generation video content.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The purpose of the embodiments of this specification is to propose a limit on redundancy checking for adding HMVP candidates to a merge list (or an AMVP list).

[0006] The technical problems that the embodiments herein seek to solve are not limited to those mentioned above, and other different technical problems not mentioned can be clearly understood by those skilled in the art to which the invention pertains from the following description. [Means for solving the problem]

[0007] One aspect of this embodiment provides a method for processing a video signal based on interpretation, comprising the steps of: constructing a first merge candidate list for the current block using spatial and temporal merge candidates for the current block; constructing a second merge candidate list by adding history-based merge candidates indicating motion information of previously coded blocks of the current block to the first merge candidate list; obtaining a merge index in the second merge candidate list that indicates a merge candidate to be applied to the current block; and generating a predicted block of the current block using the motion information of the merge candidate indicated by the merge index, wherein the history-based merge candidate may be added to the first merge candidate list if it has motion information that does not overlap with predefined merge candidates from among the merge candidates included in the first merge candidate list.

[0008] Preferably, the history-based merge candidates can be added to the first merge candidate list if they have motion information that does not overlap with a predetermined number of merge candidates included in the first merge candidate list.

[0009] Preferably, the history-based merge candidate may be added to the first merge candidate list if it has motion information that does not overlap with a specific spatial merge candidate included in the first merge candidate list.

[0010] Preferably, the history-based merge candidates can be derived from a predefined number of candidates within a history-based buffer that stores the history-based merge candidates.

[0011] Preferably, the step of constructing the second merge candidate list further includes the step of adding a history-based space-time merge candidate, the history-based space-time candidate can be derived using two of the history-based merge candidates and the time merge candidate.

[0012] Preferably, the motion vector of the spatial-time candidate of the history base can be guided to the average of the motion vectors of the two merge candidate history bases and the average of the motion vector of the time-merge candidate.

[0013] Another aspect of the embodiments of this specification relates to an apparatus for processing a video signal based on interprediction, comprising a memory for storing the video signal and a processor coupled to the memory, wherein the processor comprises a first merge candidate list for the current block using spatial and temporal merge candidates for the current block, a second merge candidate list by adding history-based merge candidates indicating motion information of previously coded blocks for the current block to the first merge candidate list, a merge index indicating a merge candidate to be applied to the current block within the second merge candidate list, and a predicted block for the current block using the motion information of the merge candidate indicated by the merge index, wherein the history-based merge candidate may be added to the first merge candidate list if it has motion information that does not overlap with predefined merge candidates from among the merge candidates included in the first merge candidate list. [Effects of the Invention]

[0014] In the embodiments described herein, the complexity associated with redundancy checks can be reduced and efficiency increased by limiting the redundancy checks for adding to a merge list (or AMVP list).

[0015] The effects obtained by the present invention are not limited to those mentioned above, and any other effects not mentioned will be clearly understood by those with ordinary skill in the art to which the present invention pertains from the following description. [Brief explanation of the drawing]

[0016] The accompanying drawings, included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and illustrate the technical features of the invention together with the detailed description.

[0017] [Figure 1] As an example of a video signal processing device according to an embodiment of the present invention, an example of the functional configuration of an encoder is shown. [Figure 2] As an example to which the present invention is applied, a schematic block diagram of a decoding device that decodes video / image signals is shown. [Figure 3] This figure shows an example of a multi-type tree structure to which the present invention can be applied. [Figure 4] This figure illustrates an example of an embodiment to which the present invention can be applied: a signaling mechanism for partition information in a quadtree with nested multi-type tree structure. [Figure 5] This figure illustrates an example to which the present invention can be applied, in which a CTU is divided into multiple CUs based on a quadtree and nested multi-type tree structure. [Figure 6] This figure illustrates an example of a method to which the present invention can be applied, specifically a method for restricting turnary-tree partitioning. [Figure 7]As an example of an embodiment to which the present invention can be applied, it is a diagram illustrating redundant split patterns that can occur in the split of a binary tree and the split of a ternary tree. [Figure 8] It is a diagram illustrating an inter-prediction-based video / video encoding method according to an embodiment of the present invention and an inter-prediction unit in an encoding device according to an embodiment of the present invention. [Figure 9] It is a diagram illustrating an inter-prediction-based video / video encoding method according to an embodiment of the present invention and an inter-prediction unit in an encoding device according to an embodiment of the present invention. [Figure 10] It is a diagram illustrating a method for decoding an inter-prediction-based video / video according to an embodiment of the present invention and an example of an inter-prediction unit in a decoding device according to an embodiment of the present invention. [Figure 11] It is a diagram illustrating a method for decoding an inter-prediction-based video / video according to an embodiment of the present invention and an example of an inter-prediction unit in a decoding device according to an embodiment of the present invention. [Figure 12] As an example of an embodiment to which the present invention is applied, it is a diagram for explaining peripheral blocks used in the merge mode or skip mode. [Figure 13] It is a flowchart illustrating a method for constructing a merge candidate list according to an embodiment to which the present invention is applied. [Figure 14] It is a flowchart illustrating a method for constructing a merge candidate list according to an embodiment to which the present invention is applied. [Figure 15] Examples of motion models according to embodiments of the present invention are shown. [Figure 16] Examples of motion vectors of control points for affine motion prediction according to embodiments of the present invention are shown. [Figure 17] Examples of motion vectors for each sub-block of a block to which affine motion prediction according to an embodiment of the present invention is applied are shown. [Figure 18] Examples of peripheral blocks used for affine motion prediction in the affine merge mode according to embodiments of the present invention are shown. [Figure 19] An example of a block to which affine motion prediction is performed using a surrounding block to which affine motion prediction according to an embodiment of the present invention is applied is shown. [Figure 20] This figure illustrates a method for generating a merge candidate list using a peripheral affine coding block according to an embodiment of the present invention. [Figure 21] This figure illustrates a method for constructing an affine merge candidate list using peripheral blocks encoded by affine prediction according to an embodiment of the present invention. [Figure 22] This figure illustrates a method for constructing an affine merge candidate list using peripheral blocks encoded by affine prediction according to an embodiment of the present invention. [Figure 23] The present invention shows an example of a peripheral block used for affine motion prediction in an affine intermode according to an embodiment. [Figure 24] The present invention shows an example of a peripheral block used for affine motion prediction in an affine intermode according to an embodiment. [Figure 25] This figure illustrates a method for inducing candidate motion vectors using motion information of surrounding blocks in an affine intermode according to an embodiment of the present invention. [Figure 26] This figure illustrates a method for inducing candidate motion vectors using motion information of surrounding blocks in an affine intermode according to an embodiment of the present invention. [Figure 27] An example of a method for inducing an affine motion vector field at the subblock level according to an embodiment of the present invention is shown. [Figure 28] This invention illustrates a method for generating prediction blocks and motion vectors using an affine motion model applied to interpretation. [Figure 29]This figure illustrates a method for performing motion compensation based on the motion vector of a control point according to an embodiment of the present invention. [Figure 30] This figure illustrates a method for performing motion compensation based on the motion vector of a control point in a non-square block according to an embodiment of the present invention. [Figure 31] This figure illustrates a method for performing motion compensation based on the motion vector of a control point in a non-square block according to an embodiment of the present invention. [Figure 32] This figure illustrates a method for performing motion compensation based on the motion vector of a control point in a non-square block according to an embodiment of the present invention. [Figure 33] This figure illustrates a method for performing motion compensation based on the motion vector of a control point in a non-square block according to an embodiment of the present invention. [Figure 34] This figure illustrates a method for performing motion compensation based on the motion vector of a control point in a non-square block according to an embodiment of the present invention. [Figure 35] This figure illustrates a method for performing motion compensation based on the motion vector of a control point in a non-square block according to an embodiment of the present invention. [Figure 36] This figure illustrates a method for performing motion compensation based on the motion vector of a control point in a non-square block according to an embodiment of the present invention. [Figure 37] This figure illustrates a method for performing motion compensation based on the motion vector of a control point in a non-square block according to an embodiment of the present invention. [Figure 38] This figure illustrates a method for performing motion compensation based on the motion vector of a control point in a non-square block according to an embodiment of the present invention. [Figure 39] This is a flowchart illustrating a method for storing HMVP according to the examples of this specification. [Figure 40] This figure illustrates an HMVP table operating in a non-restrictive FIFO manner according to the embodiments of this specification. [Figure 41]This figure illustrates an HMVP table operating in a restricted FIFO scheme according to the embodiments of this specification. [Figure 42] This figure illustrates the HMVP LUT and Long-Term HMVP LUT according to the embodiments of this specification. [Figure 43] This figure shows an example of a method for updating (updating) an HMVP LUT according to the examples described herein. [Figure 44] This figure illustrates a method for limiting the number of HMVP candidates subject to pruning checks according to the embodiments of this specification. [Figure 45] This figure shows an example of a method for performing a pruning check according to the embodiments of this specification. [Figure 46] This figure illustrates a method for inducing H-STMVP candidates using motion vectors that reference different reference pictures, according to an embodiment of this specification. [Figure 47] This flowchart illustrates a method for processing a video signal based on an inter-prediction according to an embodiment to which the present invention is applied. [Figure 48] An example of a block diagram of an apparatus for processing video signals according to the embodiments of this specification is shown. [Figure 49] This shows a video coding system to which the present invention is applied. [Figure 50] An example of an embodiment to which the present invention is applied is shown in the structure diagram of a content streaming system. [Modes for carrying out the invention]

[0018] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to illustrate exemplary embodiments of the present invention and is not intended to represent only the possible embodiments in which the present invention can be carried out. The following detailed description includes specific details to provide a complete understanding of the present invention. However, those skilled in the art will see that the present invention can be carried out without such specific details.

[0019] In some cases, known structures and devices may be omitted or shown in the form of block diagrams focusing on the core function of each structure and device, in order to avoid ambiguity of the concept of the present invention.

[0020] Furthermore, while the terminology used in this invention has been selected as widely used and common terms whenever possible, in certain cases, the applicant may use terms of their own choosing for explanation. In such cases, the meaning will be clearly described in the detailed explanation of the relevant section, and it is important to clarify that the terminology used in this invention should not be interpreted simply by its name alone, but rather by understanding the meaning of the relevant term.

[0021] The specific terms used in the following description are provided to aid in understanding the present invention, and the use of such specific terms can be modified in other ways without departing from the technical spirit of the invention. For example, terms such as signals, data, samples, pictures, frames, and blocks can be appropriately substituted and interpreted during each coding process.

[0022] Hereafter, in this specification, “processing unit” means a unit on which an encoding / decoding process, such as prediction, transformation, and / or quantization, is performed. Hereafter, for convenience of explanation, a processing unit may also be referred to as a “processing block” or “block.”

[0023] The term "processing unit" can be interpreted as including units for both the luminance (luma) component and the chroma component. For example, a processing unit could be a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0024] Furthermore, a processing unit can be interpreted as including units for either the lumen (luma) component or the chroma component. For example, a processing unit could correspond to a coding tree block (CTB), coding block (CB), prediction block (PU), or transform block (TB) for the lumen (luma) component. Alternatively, it could correspond to a coding tree block (CTB), coding block (CB), prediction block (PU), or transform block (TB) for the chroma component. However, it is not limited to these interpretations, and a processing unit can also be interpreted as including units for both the lumen (luma) and chroma components.

[0025] Furthermore, the processing unit is not necessarily limited to a square block; it can also be configured in the form of a polygon with three or more vertices.

[0026] In this specification, pixels or similar elements will be referred to as samples. Using samples can be interpreted as using pixel values ​​or similar elements.

[0027] Figure 1 shows a schematic block diagram of an encoding device to which the present invention is applied, in which video / image signal encoding is performed.

[0028] Referring to Figure 1, the encoding device 100 may consist of a video splitting unit 110, a subtraction unit 115, a conversion unit 120, a quantization unit 130, an inverse quantization unit 140, an inverse conversion unit 150, an addition unit 155, a filtering unit 160, a memory 170, an inter-prediction unit 180, an intra-prediction unit 185, and an entropy encoding unit 190. The inter-prediction unit 180 and the intra-prediction unit 185 can be collectively referred to as the prediction unit. In other words, the prediction unit can include the inter-prediction unit 180 and the intra-prediction unit 185. The conversion unit 120, quantization unit 130, inverse quantization unit 140, and inverse conversion unit 150 can be included in the residual processing unit. The residual processing unit may further include a subtraction unit 115. As one embodiment, the aforementioned video splitting unit 110, subtraction unit 115, conversion unit 120, quantization unit 130, inverse quantization unit 140, inverse conversion unit 150, addition unit 155, filtering unit 160, inter-prediction unit 180, intra-prediction unit 185, and entropy encoding unit 190 can be configured by a single hardware component (e.g., an encoder or processor). Furthermore, the memory 170 may include a DPB (decoded picture buffer) and may be configured by a digital storage medium.

[0029] The video splitting unit 110 can split the input video (or picture, frame) input to the encoding device 100 into one or more processing units. For example, the processing unit may be referred to as a coding unit (CU). In this case, the coding unit can be recursively split from a coding tree unit (CTU) or the largest coding unit (LCU) based on a QTBT (Quad-tree binary-tree) structure. For example, one coding unit can be split into multiple coding units of deeper depth based on a quad-tree structure and / or a binary tree structure. In this case, for example, the quad-tree structure may be applied first and the binary tree structure later, or the binary tree structure may be applied first. Based on the final coding unit that cannot be further split, the coding procedure according to the present invention can be executed. In this case, based on coding efficiency according to the characteristics of the video, the largest coding unit is immediately used as the final coding unit, or, if necessary, the coding unit is recursively divided into coding units of lower depth, and the coding unit of the optimal size is used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and reconstruction, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit can be divided or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit that derives transformation coefficients and / or a unit that derives a residual signal from transformation coefficients.

[0030] The term "unit" is used interchangeably with terms such as "block" or "area," depending on the context. Generally, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, sometimes representing only the luminance (luma) component pixel / pixel value, or only the chroma component pixel / pixel value. A sample is used as the term corresponding to a single picture (or image) pixel or pel.

[0031] The encoding device 100 can generate a residual signal (remaining blocks, remaining sample array) by subtracting the predicted signal (predicted blocks, predicted sample array) output from the inter-prediction unit 180 or intra-prediction unit 185 from the input video signal (original blocks, original sample array), and the generated residual signal is transmitted to the conversion unit 120. In this case, as shown in the figure, the unit that subtracts the predicted signal (predicted blocks, predicted sample array) from the input video signal (original blocks, original sample array) within the encoder 100 can be referred to as the subtraction unit 115. The prediction unit can predict the block to be processed (hereinafter referred to as the current block) and generate a predicted block that includes the predicted samples of the current block.

[0032] The prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block or CU unit. As described later in the explanation of each prediction mode, the prediction unit can generate various prediction-related information, such as prediction mode information, and transmit it to the entropy encoding unit 190. The prediction-related information can be encoded by the entropy encoding unit 190 and output in bitstream format.

[0033] The intra-prediction unit 185 can predict the current block by referring to a sample in the current picture. The referenced sample may be located in the vicinity (neighbor) of the current block or at a distance, depending on the prediction mode. The prediction mode in intra-prediction can include multiple non-directional modes and multiple directional modes. Non-directional modes may include, for example, DC mode and Planar mode. Directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is an example, and depending on the settings, more or fewer directional prediction modes may be used. The intra-prediction unit 185 can also determine the prediction mode to apply to the current block using the prediction modes applied to the surrounding blocks.

[0034] The interprediction unit 180 can derive a predicted block for the current block based on a reference block (an array of reference samples) identified by motion vectors on the reference picture. In this case, to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between surrounding blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include information on the interprediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, surrounding blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference block and the reference picture containing the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, a colCU, etc., and the reference picture containing the temporal neighboring block may be called a collocated picture (colPic). For example, the inter-prediction unit 180 can construct a motion information candidate list based on surrounding blocks and generate information indicating which candidates to use to derive the motion vector and / or reference picture index of the current block. Inter-prediction can be performed based on various prediction modes; for example, in skip mode and merge mode, the inter-prediction unit 180 can use the motion information of surrounding blocks as the motion information of the current block. In skip mode, unlike merge mode, the residual signal may not be transmitted.In motion vector prediction (MVP) mode, the motion vector of the current block can be indicated by signaling the motion vector difference using the motion vectors of surrounding blocks in a motion vector predictor.

[0035] The prediction signal generated via the interpretation unit 180 or the intrapretation unit 185 is used to generate a restored signal or to generate a residual signal.

[0036] The transformation unit 120 can generate transformation coefficients by applying transformation techniques to the residual signal. For example, the transformation technique may include at least one of the following: DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT refers to a transformation obtained from a graph representing the relationship information between pixels. CNT refers to a transformation obtained by generating a prediction signal using all previously reconstructed pixels. The transformation process may also be applied to pixel blocks of the same size that form a square, or it may be applied to blocks of variable size that do not form a square.

[0037] The quantization unit 130 quantizes the conversion coefficients and transmits them to the entropy encoding unit 190, which can encode the quantized signal (information about the quantized conversion coefficients) and output it as a bitstream. The information about the quantized conversion coefficients can be referred to as residual information. The quantization unit 130 can rearrange the quantized conversion coefficients in block form into a one-dimensional vector form based on the coefficient scan order, and can also generate information about the quantized conversion coefficients based on the quantized conversion coefficients in the form of a one-dimensional vector. The entropy encoding unit 190 can perform various encoding methods, such as exponential Golomb, CAVLC (context-adaptive variable length coding), and CABAC (context-adaptive binary arithmetic coding). In addition to the quantized conversion coefficients, the entropy encoding unit 190 can also encode information necessary for video / image restoration (such as the values ​​of syntax elements) together or separately. Encoded information (e.g., encoded video / image information) can be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The bitstream can be transmitted over a network or stored on a digital storage medium. Here, the network can include broadcast networks and / or other networks, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray®, HDD, SSD, etc. The signal output from the entropy encoding unit 190 can be transmitted by a transmission unit (not shown) and / or stored by a storage unit (not shown) which can be configured as internal / external elements of the encoding device 100, or the transmission unit may be a component of the entropy encoding unit 190.

[0038] The quantized conversion coefficients output from the quantization unit 130 are used to generate a prediction signal. For example, the quantized conversion coefficients can be used to reconstruct the residual signal by applying inverse quantization and inverse transformation via the inverse quantization unit 140 and inverse transformation unit 150 in the loop. The adder 155 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter-prediction unit 180 or intra-prediction unit 185. If there is no residual for the block to be processed, as in the case where skip mode is applied, the predicted block is used as the reconstructed block. The adder 155 can be referred to as the reconstruction unit or the reconstructed block generation unit. The generated reconstructed signal is used for intra-prediction of the next block to be processed in the current picture, and can also be used for inter-prediction of the next picture after filtering, as described later.

[0039] The filtering unit 160 can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 160 can apply various filtering methods to the restored picture to generate a modified restored picture, and the modified restored picture can be stored in the memory 170, specifically in the DPB of the memory 170. The various filtering methods can include, for example, ji-blocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter. The filtering unit 160 can generate various filtering information, as described later in the description of each filtering method, and transmit it to the entropy encoding unit 190. The filtering information can be encoded by the entropy encoding unit 190 and output in bitstream format.

[0040] The corrected restored picture transmitted to memory 170 is used as a reference picture by the interpretation unit 180. This allows the encoding device to avoid prediction mismatches between the encoding device 100 and the decoding device when interpretation is applied, and also improves coding efficiency.

[0041] Memory 170DPB can store the corrected restored picture for use as a reference picture from the inter-prediction unit 180. Memory 170 can store motion information of blocks from which motion information in the current picture was derived (or encoded) and / or motion information of blocks in the already restored picture. The stored motion information can be transmitted to the inter-prediction unit 180 for use as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory 170 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 185.

[0042] Figure 2 shows a schematic block diagram of a decoding device that performs video / image signal decoding, as an example to which the present invention is applied.

[0043] Referring to Figure 2, the decoding device 200 may consist of an entropy decoding unit 210, an inverse quantization unit 220, an inverse transformation unit 230, an addition unit 235, a filtering unit 240, a memory unit 250, an inter-prediction unit 260, and an intra-prediction unit 265. The inter-prediction unit 260 and the intra-prediction unit 265 together may be referred to as the prediction unit. That is, the prediction unit may include an inter-prediction unit 180 and an intra-prediction unit 185. The inverse quantization unit 220 and the inverse transformation unit 230 together may be referred to as the residual processing unit. That is, the residual processing unit may include an inverse quantization unit 220 and an inverse transformation unit 230. The aforementioned entropy decoding unit 210, inverse quantization unit 220, inverse transformation unit 230, addition unit 235, filtering unit 240, inter-prediction unit 260, and intra-prediction unit 265 may be configured by a single hardware component (e.g., a decoder or a processor) depending on the embodiment. Furthermore, memory 170 may include a DPB (decoded picture buffer) and may be composed of a digital storage medium.

[0044] When a bitstream containing video / image information is input, the decoding device 200 can reconstruct the image corresponding to the process by which the video / image information from the encoding device in Figure 1 has been processed. For example, the decoding device 200 can decode using the processing unit applied in the encoding device. Therefore, the decoding processing unit can be, for example, a coding unit, which can thus divide a quad-tree structure and / or binary tree structure from a coding tree unit or a maximum coding unit. The reconstructed video signal decoded and output via the decoding device 200 can then be played back by a playback device.

[0045] The decoding device 200 can receive the signal output from the encoding device in Figure 1 in bitstream form, and the received signal can be decoded via the entropy decoding unit 210. For example, the entropy decoding unit 210 can analyze the bitstream and derive information (e.g., video / image information) necessary for video restoration (or picture restoration). For example, the entropy decoding unit 210 can decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntactic elements necessary for video restoration and the quantized values ​​of conversion coefficients related to the residual. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntactic element from the bitstream, determine a context model using information on the syntactic element to be decoded and the decoded information of the surrounding and decoded blocks, or symbol / bin information decoded in a previous step, predict the probability of bin occurrence based on the determined context model, and perform arithmetic decoding of bins to generate symbols corresponding to the values ​​of each syntactic element. In this case, the CABAC entropy decoding method can update the context model using the decoded symbol / empty information for the next symbol / empty context model after determining the context model. Of the information decoded by the entropy decoding unit 2110, information related to prediction is provided to the prediction unit (inter-prediction unit 260 and intra-prediction unit 265), and the residual values, i.e., quantized conversion coefficients and related parameter information, that have been entropy decoded from the entropy decoding unit 210 can be input to the inverse quantization unit 220. In addition, of the information decoded by the entropy decoding unit 210, information related to filtering can be provided to the filtering unit 240. On the other hand, a receiving unit (not shown) that receives the signal output from the encoding device may be composed of internal / external elements of the decoding device 200, or the receiving unit may be a component of the entropy decoding unit 210.

[0046] The inverse quantization unit 220 can inversely quantize the quantized transformation coefficients and output the transformation coefficients. The inverse quantization unit 220 can rearrange the quantized transformation coefficients in the form of two-dimensional blocks. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit 220 can perform inverse quantization of the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) to obtain the transformation coefficients.

[0047] In the inverse conversion unit 230, the conversion coefficients are inversely converted to obtain a residual signal (residual block, residual sample array).

[0048] The prediction unit can make predictions about the current block and generate a predicted block that includes prediction samples of the current block. Based on the prediction information output from the entropy decoding unit 210, the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block, and can determine a specific intra / inter-prediction mode.

[0049] The intra-prediction unit 265 can predict the current block by referring to a sample in the current picture. The referenced sample may be located in the vicinity (neighbor) of the current block or at a distance, depending on the prediction mode. In intra-prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The intra-prediction unit 265 can also determine the prediction mode to apply to the current block using the prediction modes applied to the surrounding blocks.

[0050] The interprediction unit 260 can derive the predicted block of the current block based on a reference block (an array of reference samples) identified by motion vectors on a reference picture. In this case, to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted on a block, subblock, or sample basis based on the correlation of motion information between surrounding blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include information on the interprediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, surrounding blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the interprediction unit 260 can construct a motion information candidate list based on surrounding blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Interprediction can be performed based on various prediction modes, and the prediction information may include information indicating the interprediction mode of the current block.

[0051] The summing unit 235 can generate a restored signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the predicted signal (predicted block, predicted sample array) output from the inter-prediction unit 260 or intra-prediction unit 265. If there is no residual for the block to be processed, such as when skip mode is applied, the predicted block is used as the restored block.

[0052] The addition unit 235 may be referred to as the restoration unit or the restoration block generation unit. The generated restoration signal is used for intra-prediction of the next block to be processed in the current picture, and can also be used for intra-prediction of the next picture after being filtered as described later.

[0053] The filtering unit 240 can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 240 can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory unit 250, specifically to the DPB of the memory unit 250. The various filtering methods can include, for example, ji-blocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter.

[0054] The restored picture stored (modified) in the DPB of memory 250 is used as a reference picture by the inter-prediction unit 260. Memory 250 can store motion information of blocks from which motion information in the current picture has been derived (or decoded) and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 260 for use as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory 170 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 265.

[0055] In this specification, the embodiments described for the filtering unit 160, inter-prediction unit 180, and intra-prediction unit 185 of the encoding device 100 can be applied identically or in a corresponding manner to the filtering unit 240, inter-prediction unit 260, and intra-prediction unit 265 of the respective decoding device 200.

[0056] Block Partitioning

[0057] The video / image coding methods described herein can be carried out based on various detailed techniques, which are outlined below. It will be obvious to those skilled in the art that the techniques described below can be related to the aforementioned and / or later video / image encoding / decoding procedures, including prediction, residual processing (such as inverse transformation and inverse quantization), coding of syntax elements, filtering, and partitioning.

[0058] The block partitioning procedure described herein is performed in the video splitting unit 110 of the encoding device described above, and the partitioning-related information is processed (encoded) by the entropy encoding unit 190 and transmitted to the decoding device in the form of a bitstream. The entropy decoding unit 210 of the decoding device derives the block partitioning structure of the current picture based on the partitioning-related information obtained from the bitstream, and can then perform a series of procedures for video decoding (e.g., prediction, residual processing, block restoration, in-loop filtering, etc.) based on this.

[0059] Partitioning of picture into CTUs

[0060] A picture can be divided into a sequence of coding tree units (CTUs). A CTU can correspond to a coding tree block (CTB). Alternatively, a CTU can contain two coding tree blocks: one for a luma sample and one for a corresponding chroma sample. That is, for a picture containing three sample arrays, a CTU can contain an NxN block for the luma sample and two corresponding blocks for the chroma sample.

[0061] The maximum allowable size of a CTU for coding and prediction may differ from the maximum allowable size of a CTU for conversion. For example, the maximum allowable size of a luma block within a CTU may be 128x128.

[0062] Partitioning of the CTUs using a tree structure

[0063] A CTU can be divided into CUs based on a quad-tree (QT) structure. A quad-tree structure can also be referred to as a quaternary tree structure because it reflects various local characteristics. However, in this document, a CTU can be divided not only on quad-trees but also on multi-type tree structures that include binary-trees (BT) and ternary-trees (TT). Hereinafter, a QTBT structure can include quad-tree and binary-tree based division structures, and a QTBTTT can include quad-tree, binary-tree, and ternary-tree based division structures. Alternatively, a QTBT structure can also include quad-tree, binary-tree, and ternary-tree based division structures. In coding tree structures, CUs can have a square or rectangular shape. A CTU can first be divided into a quad-tree structure. Subsequently, the leaf nodes of the quad-tree structure can be further divided by multi-type tree structures.

[0064] Figure 3 shows an example of a multi-type tree structure as an embodiment to which the present invention can be applied.

[0065] In one embodiment of this specification, the multi-type tree structure may include four splitting types as shown in Figure 3. These four splitting types may include vertical binary splitting (SPLIT_BT_VER), horizontal binary splitting (SPLIT_BT_HOR), vertical ternary splitting (SPLIT_TT_VER), and horizontal ternary splitting (SPLIT_TT_HOR).

[0066] The leaf nodes of the aforementioned multi-type tree structure can be called CUs. Such CUs are used for prediction and transformation procedures. In general, CUs, PUs, and TUs can have the same block size. However, CUs and TUs can have different block sizes if the maximum supported transform length is smaller than the width or height of the color component of the CU.

[0067] Figure 4 illustrates a signaling mechanism for partition information in a quadtree with nested multi-type tree structure, as an example to which the present invention can be applied.

[0068] Here, the CTU is treated as the root of the quadtree and is partitioned for the first time in the quadtree structure. Each leaf node of the quadtree can subsequently be further partitioned in a multi-type tree structure. In the multi-type tree structure, a first flag (e.g., mtt_split_cu_flag) is signaled to indicate whether the node will be further partitioned. If the node is further partitioned, a second flag (e.g., mtt_split_cu_verticla_flag) can be signaled to indicate the splitting direction. Subsequently, a third flag (e.g., mtt_split_cu_binary_flag) can be signaled to indicate whether the splitting type is binary or ternary. For example, based on the mtt_split_cu_vertical_flag and the mtt_split_cu_binary_flag, the multi-type tree splitting mode (MttSplitMode) of the CU can be derived as shown in Table 1.

[0069] [Table 1]

[0070] Figure 5 illustrates an example of how the present invention can be applied, illustrating a method for dividing a CTU into multiple CUs based on a quadtree and nested multi-type tree structure.

[0071] Here, bold block edges indicate quad-tree partitioning, while the remaining edges indicate multi-type tree partitioning. Quad-tree partitions with multi-type trees can provide a content-adapted coding tree structure. A CU can correspond to a coding block (CB). Alternatively, a CU can contain two coding blocks: one for a lumens sample and one for a corresponding chroma sample. The size of a CU can be as large as a CTU, or as small as 4x4 per lumens sample. For example, in a 4:2:0 color format (or chroma format), the maximum chroma CB size can be 64x64, and the minimum chroma CB size can be 2x2.

[0072] In this document, for example, the maximum allowable luma TB size may be 64x64, and the maximum allowable chroma TB size may be 32x32. If the width or height of a CB divided along the tree structure is greater than the maximum transformation width or height, the CB can be automatically (or implicitly) divided until the horizontal and vertical TB size limits are satisfied.

[0073] On the other hand, for a quad-tree coding tree scheme with a multi-type tree, the following parameters can be defined and identified in the sps syntax element.

[0074] - CTU size: The size of the root node of a quaternary tree.

[0075] - MinQTSize: The minimum allowed quaternary tree leaf node size.

[0076] - MaxBtSize: (Maximum allowed binary tree root node size)

[0077] - MaxTtSize: (Maximum allowed ternary tree root node size)

[0078] - MaxMttDepth: The maximum allowed hierarchy depth of a multi-type tree splitting from a quad-tree leaf.

[0079] - MinBtSize: The minimum allowed binary tree leaf node size.

[0080] - MinTtSize: Minimum allowed ternary tree leaf node size

[0081] As an example of a quad-tree coding tree structure with a multi-type tree, the CTU size can be set to a 64x64 block of 128x128 chroma samples and two corresponding chroma samples (in a 4:2:0 chroma format). In this case, MinOTSize can be set to 16x16, MaxBtSize to 128x128, MaxTtSzie to 64x64, MinBtSize and MinTtSize (for both width and height) to 4x4, and MaxMttDepth to 4. Quote tree partitioning can be applied to the CTU to generate leaf nodes of the quad-tree. Leaf nodes of the quad-tree can be referred to as leaf QT nodes. Leaf nodes of the quad-tree can have sizes ranging from 16x16 (i.e., the MinOTSize) to 128x128 (i.e., the CTU size). If a leaf QT node is 128x128, it may not be further divided into a binary tree / turnerly tree. This is because, even if divided, it would exceed MaxBtsize and MaxTtszie (e.g., 64x64). Otherwise, the leaf QT node can be further divided into a multitype tree. Thus, the leaf QT node is the root node of the multitype tree, and the leaf QT node can have a multitype tree depth (mttDepth) value of 0. If the multitype tree depth reaches MaxMttdepth (e.g., 4), further divisions may not be considered. If the width of a multitype tree node is the same as MinBtSize and less than or equal to 2xMinTtSize, further horizontal divisions may no longer be considered. If the height of a multitype tree node is the same as MinBtSize and less than or equal to 2xMinTtSize, further vertical divisions may no longer be considered.

[0082] Figure 6 illustrates an example of a method to which the present invention can be applied, which involves restricting a turnary-tree partition.

[0083] Referring to Figure 6, TT division can be restricted in certain cases to allow for the design of 64x64 chroma blocks and 32x32 chroma pipelines in the hardware decoder. For example, if the width or height of the chroma coding block is greater than a certain pre-set value (e.g., 32, 64), TT division can be restricted as shown in Figure 6.

[0084] In this document, the coding tree scheme can support the idea that luma and chroma blocks have a tree structure of separate blocks.

[0085] For P and B slices, the luma and chroma CTB within a single CTU can be restricted to having the same coding tree structure. However, for I slices, luma and chroma blocks can have separate block tree structures from each other. If the individual block tree mode is applied, the luma CTB can be divided into CUs based on a specific coding tree structure, and the chroma CTB can be divided into chroma CUs based on other coding tree structures. This means that a CU within an I slice can consist of coding blocks of luma components or coding blocks of two chroma components, while a CU in a P or B slice can consist of blocks of three color components.

[0086] As mentioned above in "Partitioning of the CTUs using a tree structure," we described a quad-tree coding tree structure with a multi-type tree, but the structure through which a CU is partitioned is not limited to this. For example, the structures of BT and TT can be interpreted as concepts included in the structure of a Multiple Partitioning Tree (MPT), and a CU can be interpreted as being partitioned through a QT structure and an MPT structure. In one example of a CU being partitioned through a QT structure and an MPT structure, the partition structure can be determined by signaling a syntax element (e.g., MPT_split_type) that contains information about whether the leaf nodes of the QT structure are partitioned into several blocks, and a syntax element (e.g., MPT_split_mode) that contains information about whether the leaf nodes of the QT structure are partitioned vertically or horizontally.

[0087] In other examples, the CU can be divided in ways other than the QT structure, BT structure, or TT structure. That is, unlike the QT structure in which the lower-depth CU is divided into quarters of the upper-depth CU, or the BT structure in which the lower-depth CU is divided into half the size of the upper-depth CU, or the TT structure in which the lower-depth CU is divided into quarters or half the size of the upper-depth CU, the lower-depth CU can, in some cases, be divided into 1 / 5, 1 / 3, 3 / 8, 3 / 5, 2 / 3, or 5 / 8 the size of the upper-depth CU, and the way in which the CU is divided is not limited to these.

[0088] If a portion of a tree node block exceeds the bottom or right picture boundary, that tree node block can be restricted so that all samples of all coded CUs are located within the picture boundary. In this case, for example, the following partitioning rule can be applied:

[0089] - If a portion of a tree node block exceeds both the bottom and the right picture boundaries,

[0090] If the block is a QT node and its size is larger than the minimum QT size, the block is forced to be split in QT split mode.

[0091] Otherwise, the block is forced to be split into SPLIT_BT_HOR mode.

[0092] - Otherwise, if a portion of a tree node block exceeds the bottom picture boundaries,

[0093] If the block is a QT node, and its size is larger than the minimum QT size, and its size is larger than the maximum BT size, the block is forced to be split in QT split mode.

[0094] Otherwise, if the block is a QT node, and the size of the block is larger than the minimum QT size and the size of the block is smaller than or equal to the maximum BT size, the block is forced to be split in QT split mode or SPLIT_BT_HOR mode.

[0095] Otherwise (the block is a BTT node or the block size is smaller than or equal to the minimum QT size), the block is forced to be split in SPLIT_BT_HOR mode.

[0096] - Otherwise, if a portion of a tree node block exceeds the right-hand boundary of the picture,

[0097] If the block is a QT node, and its size is larger than the minimum QT size, and its size is larger than the maximum BT size, the block is forced to be split in QT split mode.

[0098] Otherwise, if the block is a QT node, and the size of the block is larger than the minimum QT size and the size of the block is smaller than or equal to the maximum BT size, the block is forced to be split in QT split mode or SPLIT_BT_VER mode.

[0099] - Otherwise (the block is a BTT node or the block size is smaller than or equal to the minimum QT size), the block is forced to be split with SPLIT_BT_VER mode.

[0100] On the other hand, the quad-tree coding block structure with the aforementioned multi-type tree can provide a highly flexible block partitioning structure. Due to the partitioning types supported by the multi-type tree, other partitioning patterns may, in some cases, potentially result in the same coding block structure. By limiting the occurrence of such redundant partitioning patterns, the amount of data in the partitioning information can be reduced. This is illustrated with reference to the diagram below.

[0101] Figure 7 illustrates redundant partitioning patterns that can occur in binary tree partitioning and turnary tree partitioning as embodiments to which the present invention can be applied.

[0102] As shown in Figure 7, two levels of consecutive binary splits in one direction have the same coding block structure as the binary split of the central partition after a Turner-Lee split. In such cases, the binary tree split of the central partition of the Turner-Lee split (in the given direction) can be restricted. Such restrictions can be applied to the CU of all pictures. When such a particular split is restricted, the signaling of the corresponding syntactic element can be modified to reflect this restriction, thereby reducing the number of bits signaled for partitioning. For example, when the binary tree split of the central partition of a CU is restricted, as in the example shown in Figure 7, the mtt_split_cu_binary_flag syntactic element, which indicates whether the split is a binary split or a Tenor-Lee split, is not signaled, and its value is 0, which can be inferred by the decoder.

[0103] prediction

[0104] To restore the current processing unit in which decoding is performed, the decoded portion of the current picture or another picture containing the current processing unit can be used.

[0105] A picture (slice) that uses only the current picture for reconstruction, i.e., performs only in-screen prediction, can be called an intra-picture or I-picture (slice); a picture (slice) that uses up to one motion vector and reference index to predict each unit can be called a predictive picture or P-picture (slice); and a picture (slice) that uses up to two motion vectors and reference index can be called a bi-predictive picture or B-picture (slice).

[0106] Intra-prediction refers to a prediction method that derives the current processing block from data elements (e.g., sample values, etc.) of the same decoded picture (or slice). In other words, it refers to a method of predicting the pixel values ​​of the current processing block by referring to the restored region within the current picture.

[0107] The following provides a more detailed explanation of interpretation prediction.

[0108] Interpretation (or screen-to-screen prediction)

[0109] Interpretation refers to a prediction method that derives the current processing block based on data elements (e.g., sample values ​​or motion vectors) from pictures other than the current picture. In other words, it refers to a method of predicting the pixel values ​​of the current processing block by referring to restored regions in other restored pictures other than the current picture.

[0110] Inter-picture prediction (or picture-to-picture prediction) is a technique that removes redundancy between pictures, and is mostly done through motion estimation and motion compensation.

[0111] The embodiments described herein describe the detailed techniques of the interpretation method previously explained in Figures 1 and 2. In the case of a decoder, this can be shown in Figure 10, which will be described later, as an interpretation-based video / image decoding method, and in Figure 11, as an interpretation unit within a decoding device. In addition, in the case of an encoder, this can be shown in Figure 8, which will be described later, as an interpretation-based video / image encoding method, and in Figure 9, as an interpretation unit within an encoding device. Furthermore, the data encoded by Figures 8 and 9 can be stored in the form of a bitstream.

[0112] The prediction unit of the encoding / decoding device can perform interpretation on a block-by-block basis to derive predicted samples. Interpretation can indicate predictions derived in a way that is dependent on data elements (e.g., sample values ​​or motion information, etc.) of pictures other than the current picture. When interpretation is applied to the current block, the predicted block (predicted sample array) of the current block can be derived based on the reference block (array of reference samples) identified by the motion vector on the reference picture pointed to by the index of the reference picture.

[0113] In this case, in order to reduce the amount of motion information transmitted in inter-prediction mode, the motion information of the current block can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between surrounding blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include information on the inter-prediction type (L0 prediction, L1 prediction, Bi prediction, etc.).

[0114] When interpretation is applied, a neighboring block can include a spatial neighboring block that exists within the current picture and a temporal neighboring block that exists in the reference picture. The reference picture containing the reference block and the reference picture containing the temporal neighboring block may be the same or different. The temporal neighboring block may be referred to by names such as a collocated reference block or colCU, and the reference picture containing the temporal neighboring block may be referred to as a collocated picture (colPic).

[0115] For example, a list of motion information candidates may be constructed based on the surrounding blocks of the current block, and flags or index information may be signaled to indicate which candidates are selected (used) to derive the motion vector and / or reference picture index of the current block.

[0116] Interpretation can be performed based on various prediction modes. For example, in skip mode and merge mode, the motion information of the current block can be the same as the motion information of selected surrounding blocks. In skip mode, unlike merge mode, residual signals may not be transmitted. In motion vector prediction (MVP) mode, the motion vector difference can be signaled using the motion vector of selected surrounding blocks with a motion vector predictor. In this case, the motion vector of the current block can be derived using the combination of the motion vector predictor and the motion vector difference.

[0117] Figures 8 and 9 illustrate an interpretation-based video / image encoding method according to an embodiment of the present invention and an example of an interpretation unit within an encoding device according to an embodiment of the present invention.

[0118] Referring to Figures 8 and 9, S801 can be performed by the interprediction unit 180 of the encoding device, and S802 can be performed by the residual processing unit of the encoding device. Specifically, S802 can be performed by the subtraction unit 115 of the encoding device. In S803, the prediction information can be derived by the interprediction unit 180 and encoded by the entropy encoding unit 190. In S803, the residual information can be derived by the residual processing unit and encoded by the entropy encoding unit 190. The residual information is information about the residual sample. The residual information may include information about the quantized conversion coefficients for the residual sample.

[0119] As described above, the residual sample is derived into conversion coefficients via the conversion unit 120 of the encoding device, and the conversion coefficients can be derived into quantized conversion coefficients via the quantization unit 130. Information regarding the quantized conversion coefficients can be encoded in the entropy encoding unit 190 via the residual coding procedure.

[0120] The encoding device performs interpretation of the current block (S801). The encoding device can derive the interpretation prediction mode and motion information of the current block and generate prediction samples for the current block. The procedures for determining the interpretation prediction mode, deriving motion information, and generating prediction samples may be performed simultaneously, or any one of the procedures may be performed before the others. For example, the interpretation prediction unit 180 of the encoding device may include a prediction mode determination unit 181, a motion information derivation unit 182, and a prediction sample derivation unit 183. The prediction mode determination unit 181 determines the prediction mode of the current block, the motion information derivation unit 182 derives the motion information of the current block, and the prediction sample derivation unit 183 derives motion samples for the current block.

[0121] For example, the interpretation unit 180 of the encoding device can search for blocks similar to the current block within a certain area (search area) of the reference picture via motion estimation, and derive reference blocks whose difference from the current block is minimal or below a certain standard. Based on this, it can derive an index of the reference picture that points to the reference picture in which the reference block is located, and derive a motion vector based on the difference in position between the reference block and the current block. The encoding device can determine which mode to apply to the current block from among various prediction modes. The encoding device can determine the optimal prediction mode for the current block by comparing the RD costs for the various prediction modes.

[0122] For example, if skip mode or merge mode is applied to the current block, the encoding device can configure a merge candidate list, as described later, and derive a reference block from among the reference blocks pointed to by the merge candidates included in the merge candidate list whose difference between the current block and the current block is the minimum or below a certain standard. In this case, a merge candidate associated with the derived reference block is selected, merge index information pointing to the selected merge candidate is generated, and this information can be signaled to the decoding device.

[0123] The motion information of the current block can be derived using the motion information of the selected merge candidate.

[0124] As another example, when the (A)MVP mode is applied to the current block, the encoding device can configure the (A)MVP candidate list described later, and use the motion vector of the selected mvp candidate from among the mvp (motion vector predictor) candidates included in the (A)MVP candidate list as the mvp of the current block. In this case, for example, the motion vector pointing to the reference block derived by the motion estimation described above is used as the motion vector of the current block, and the mvp candidate having the motion vector with the smallest difference from the motion vector of the current block among the mvp candidates may become the selected mvp candidate. The MVD (motion vector difference), which is the difference obtained by subtracting the mvp from the motion vector of the current block, can be derived. In this case, information regarding the MVD can be signaled to the decoding device. Also, when the (A)MVP mode is applied, the index value of the reference picture is composed of reference picture index information and can be separately signaled to the decoding device.

[0125] The encoding device can derive a residual sample based on the predicted sample (S802). The encoding device can derive the residual sample by comparing the original sample of the current block with the predicted sample.

[0126] The encoding device encodes video information including prediction information and residual information (S803). The encoding device can output the encoded video information in the form of a bitstream. The prediction information may include information related to the prediction procedure, prediction mode information (e.g., skip flag, merge flag, or mode index) and information related to motion information. The information related to motion information may include candidate selection information (e.g., merge index, mvp flag, or mvp index) which is information for deriving a motion vector. The information related to motion information may also include the aforementioned MVD information and / or index information of a reference picture.

[0127] Furthermore, the motion information may include information indicating whether L0 prediction, L1 prediction, or pair (bi) prediction is applied. The residual information is information about the residual sample. The residual information may include information about the quantized transformation coefficients for the residual sample.

[0128] The output bitstream can be stored on (digital) storage media and transmitted to a decoding device, or it can be transmitted to a decoding device via a network.

[0129] On the other hand, as mentioned above, the encoding device can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on the reference sample and the residual sample. This is because the encoding device can derive the same prediction results as those performed by the decoding device, thereby improving coding efficiency. Therefore, the encoding device can store the reconstructed picture (or reconstructed sample, reconstructed block) in memory and use it as a reference picture for interpretation. As mentioned above, procedures such as in-loop filtering can be further applied to the reconstructed picture.

[0130] Figures 10 and 11 illustrate an interpretation-based video / image decoding method according to an embodiment of the present invention, and an interpretation unit within a decoding device according to an embodiment of the present invention.

[0131] Referring to Figures 10 and 11, the decoding device can perform actions corresponding to the operations performed by the encoding device. Based on the received prediction information, the decoding device can make predictions for the current block and derive prediction samples.

[0132] Steps S1001 to S1003 can be performed by the interprediction unit 260 of the decoding device, and the residual information in S1004 can be obtained from the bitstream by the entropy decoding unit 210 of the decoding device. The residual processing unit of the decoding device can derive the residual sample of the current block based on the residual information. Specifically, the inverse quantization unit 220 of the residual processing unit can perform inverse quantization based on the quantized conversion coefficients derived from the residual information to derive the conversion coefficients, and the inverse transformation unit 230 of the residual processing unit can perform inverse transformation of the conversion coefficients to derive the residual sample of the current block. Step S1005 can be performed by the addition unit 235 or the restoration unit of the decoding device.

[0133] Specifically, the decoding device can determine the prediction mode for the current block based on the received prediction information (S1001). Based on the prediction mode information in the prediction information, the decoding device can determine which interpretation mode is applied to the current block.

[0134] For example, based on the merge flag, it can be determined whether the merge mode is applied to the current block or whether the (A)MVP mode is determined. Alternatively, based on the mode index, one of several inter-prediction mode candidates can be selected. The inter-prediction mode candidates may include skip mode, merge mode and / or (A)MVP mode, or may include various inter-prediction modes as described later.

[0135] The decoding device derives motion information for the current block based on the determined inter-prediction mode (S1002). For example, if a skip mode or merge mode is applied to the current block, the decoding device can configure a merge candidate list, as described later, and select one merge candidate from among the merge candidates included in the merge candidate list. This selection can be performed based on the selection information (merge index) described above. The motion information for the current block can be derived using the motion information of the selected merge candidate. The motion information of the selected merge candidate is used as the motion information for the current block.

[0136] As another example, when the (A)MVP mode is applied to the current block, the decoding device can configure the (A)MVP candidate list described later, and use the motion vector of the selected mvp candidate from among the mvp (motion vector predictor) candidates included in the (A)MVP candidate list as the mvp of the current block. The selection can be performed based on the selection information (mvp flag or mvp index) described above. In this case, the MVD of the current block can be derived based on the information regarding the MVD, and the motion vector of the current block can be derived based on the mvp of the current block and the MVD. Furthermore, the reference picture index of the current block can be derived based on the index information of the reference picture. Within the reference picture list for the current block, the picture pointed to by the index of the reference picture can be derived as the reference picture referenced for interpretation of the current block.

[0137] On the other hand, as will be described later, the motion information of the current block can be derived without constructing a candidate list. In this case, the motion information of the current block can be derived by following the procedure disclosed in the prediction mode described later. In this case, the construction of the candidate list as described above can be omitted.

[0138] The decoding device can generate predicted samples for the current block based on the motion information of the current block (S1003). In this case, the reference picture can be derived based on the reference picture index of the current block, and the predicted samples for the current block can be derived using the sample of the reference block pointed to by the motion vector of the current block on the reference picture. In this case, as will be described later, a filtering procedure can be performed on all or some of the predicted samples of the current block.

[0139] For example, the interpretation unit 260 of the decoding device may include a prediction mode determination unit 261, a motion information derivation unit 262, and a prediction sample derivation unit 263. The prediction mode determination unit 261 determines the prediction mode of the current block based on the prediction mode information received, derives motion information (such as motion vectors and / or reference picture indices) of the current block based on motion information information received from the motion information derivation unit 262, and derives prediction samples of the current block in the prediction sample derivation unit 263.

[0140] The decoding device generates a residual sample of the current block based on the received residual information (S1004). Based on the predicted sample and the residual sample, the decoding device can generate a restored sample of the current block and generate a restored picture based on this (S1005). As previously mentioned, procedures such as in-loop filtering may be further applied to the restored picture thereafter.

[0141] As mentioned above, the interpretation procedure may include an interpretation mode determination step, a motion information derivation step corresponding to the determined prediction mode, and a prediction execution step (creation of prediction samples) based on the derived motion information.

[0142] Interpretation mode determination

[0143] Various interpretation modes can be used to predict the current block within a picture. For example, in merge mode and skip mode, various modes such as MVP mode and affine mode can be used. Additional modes such as DMVR (Decoder side motion vector refinement) mode and AMVR (adaptive motion vector resolution) mode are also used. Affine mode can also be called affine motion prediction mode. MVP mode can also be called AMVP (advanced motion vector prediction) mode.

[0144] Prediction mode information indicating the interprediction mode of the current block can be signaled from the encoding device to the decoding device. This prediction mode information can be received by the decoding device as part of a bitstream. The prediction mode information may include index information indicating one of several candidate modes. Alternatively, the interprediction mode can be indicated via hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags.

[0145] For example, a skip flag can be signaled to indicate whether skip mode should be applied, a merge flag can be signaled if skip mode cannot be applied to indicate whether merge mode should be applied, and if merge mode cannot be applied, MVP mode can be applied. Additional flags can also be signaled for further distinctions. Affine modes can be signaled as independent modes or as dependent modes such as merge mode or MVP mode. For example, an affine mode may be configured as one of the candidates in a merge candidate list or MVP candidate list, as described later.

[0146] Derivation of motion information according to interpretation mode

[0147] Interpretation can be performed using the motion information of the current block. The encoding device can derive the optimal motion information for the current block through a motion estimation procedure. For example, the encoding device can use the original block in the original picture of the current block to search for highly correlated similar reference blocks in fractional pixel units within a defined search range in the reference picture, thereby deriving motion information. Block similarity can be derived based on the difference in phase-based sample values. For example, block similarity can be calculated based on the SAD between the current block (or the template of the current block) and the reference block (or the template of the reference block). In this case, motion information can be derived based on the reference block with the smallest SAD within the search area. The derived motion information can be signaled to the decoding device based on several methods based on the interpretation mode.

[0148] Merge mode and Skip mode

[0149] Figure 12 is a diagram illustrating a peripheral block used in merge mode or skip mode as an embodiment to which the present invention is applied.

[0150] When merge mode is applied, the motion information of the current prediction block is not transmitted directly; instead, the motion information of the surrounding prediction blocks is used to guide the motion information of the current prediction block. Therefore, by transmitting flag information indicating that merge mode has been used and a merge index indicating which surrounding prediction blocks were used, the motion information of the current prediction block can be instructed.

[0151] In one embodiment of this specification,

[0152] An encoder in one embodiment of this specification can search for merge candidate blocks to be used to guide the motion information of the current predicted block in order to perform a merge mode. For example, up to five merge candidate blocks may be used, but the present invention is not limited thereto. The maximum number of merge candidate blocks can be transmitted in the slice header (or tile group header), but the present invention is not limited thereto. After finding the merge candidate blocks, the encoder can generate a merge candidate list and select the merge candidate block with the lowest cost among them as the final merge candidate block.

[0153] One embodiment of this specification provides various embodiments of merge candidate blocks that constitute the merge candidate list.

[0154] The merge candidate list can, for example, use five merge candidate blocks. For example, it can use four spatial merge candidates and one temporal merge candidate. As a specific example, in the case of spatial merge candidates, the blocks shown in Figure 12 can be used as spatial merge candidates.

[0155] Figure 13 is a flowchart illustrating a method for configuring a merge candidate list according to one embodiment to which the present invention is applied.

[0156] Referring to Figure 13, the encoding device (encoder / decoder) searches for spatially surrounding blocks of the current block and inserts the derived spatial merge candidates into the merge candidate list (S1301). For example, the spatially surrounding blocks may include the block around the lower left corner of the current block, the left surrounding block, the upper right corner surrounding block, the upper surrounding block, and the upper left corner surrounding block. However, this is an example, and additional surrounding blocks such as the right surrounding block, the lower surrounding block, and the lower right surrounding block may also be used as spatial surrounding blocks. The encoding device can search for the spatially surrounding blocks based on priority, detect available blocks, and derive the motion information of the detected blocks as the spatial merge candidates. For example, the encoder and decoder can search the five blocks shown in Figure 12 in the order A1, B1, B0, A0, B2, and sequentially index the available candidates to form a merge candidate list.

[0157] The coding device searches for temporally surrounding blocks of the current block and inserts the derived temporal merge candidates into the merge candidate list (S1302). The temporally surrounding blocks can be located on a reference picture that is a different picture from the current picture on which the current block is located. The reference picture on which the temporally surrounding blocks are located can be referred to as a collocated picture or a col picture. The temporally surrounding blocks can be searched for in the order of the lower right corner surrounding block and the lower right center block of the co-located block at the same position as the current block on the col picture.

[0158] On the other hand, when motion data compression is applied, specific motion information can be stored as representative motion information for each certain storage unit in the col picture. In this case, it is not necessary to store motion information for all blocks within the certain storage unit, thereby achieving the motion data compression effect. In this case, the certain storage unit may be predetermined, for example, in units of 16x16 samples or 8x8 samples, or the size information of the certain storage unit may be signaled from the encoder to the decoder. When motion data compression is applied, the motion information of the temporally surrounding blocks can be replaced with the representative motion information of the certain storage unit in which the temporally surrounding blocks are located.

[0159] In other words, in this case, from an implementation standpoint, the temporal merge candidate can be derived not from a predicted block located at the coordinates of the temporal surrounding block, but from the motion information of a predicted block that covers a position obtained by arithmetically shifting to the right by a certain value and then arithmetically shifting to the left based on the coordinates (upper left sample position) of the temporal surrounding block. For example, if the constant storage unit is a 2nx2 sample unit, and the coordinates of the temporal surrounding block are (xTnb, yTnb), then the motion information of a predicted block located at the corrected position ((xTnb >> n)<< n), (yTnb >> n)<< n)) is used for the temporal merge candidate.

[0160] Specifically, for example, if the constant storage unit is 16x16 sample units, and the coordinates of the temporally surrounding block are (xTnb, yTnb), then the motion information of the predicted block located at the corrected position ((xTnb >> 4)<< 4), (yTnb >> 4)<< 4)) is used for the temporal merge candidate. Alternatively, for example, if the constant storage unit is 8x8 sample units, and the coordinates of the temporally surrounding block are (xTnb, yTnb), then the motion information of the predicted block located at the corrected position ((xTnb >> 3)<< 3), (yTnb>> 3)<< 3)) is used for the temporal merge candidate.

[0161] The coding device can check whether the current number of merge candidates is less than the maximum number of merge candidates (S1303). The maximum number of merge candidates can be predefined or signaled from the encoder to the decoder. For example, the encoder can generate and encode information regarding the maximum number of merge candidates and transmit it to the decoder in the form of a bitstream. If the maximum number of merge candidates is reached, the process of adding subsequent candidates may not proceed.

[0162] If the result of the above check is that the number of current merge candidates is less than the number of maximum merge candidates, the coding device inserts additional merge candidates into the merge candidate list (S1304). The additional merge candidates may include, for example, ATMVP, combined bi-predictive merge candidates (if the slice type of the current slice is type B), and / or zero vector merge candidates.

[0163] If, as a result of the above check, the number of current merge candidates is not less than the number of maximum merge candidates, the coding device can terminate the configuration of the merge candidate list. In this case, the encoder can select the optimal merge candidate from among the merge candidates constituting the merge candidate list based on RD (rate-distortion) cost, and can signal selection information (e.g., merge index) pointing to the selected merge candidate to the decoder. The decoder can select the optimal merge candidate based on the merge candidate list and the selection information.

[0164] As previously stated, the motion information of the selected merge candidate is used as the motion information of the current block, and based on the motion information of the current block, a predicted sample of the current block can be derived. The encoder can derive a residual sample of the current block based on the predicted sample and signal the decoder to the residual information associated with the residual sample. As previously stated, the decoder can generate a restored sample based on the residual sample derived based on the residual information and the predicted sample, and generate a restored picture based on this.

[0165] When skip mode is applied, the motion information of the current block can be derived in the same way as when merge mode is applied. However, when skip mode is applied, the residual signal of that block is omitted, and therefore the predicted sample is immediately used as the restored sample.

[0166] MVP Mode

[0167] Figure 14 is a flowchart illustrating a method for configuring a merge candidate list according to one embodiment to which the present invention is applied.

[0168] When MVP (Motion Vector Prediction) mode is applied, a candidate list of motion vector predictors (MVP) can be generated using the motion vectors of the reconstructed spatially surrounding blocks (which may be, for example, the surrounding blocks described in section 12 above) and / or the motion vectors corresponding to the temporally surrounding blocks (or Col blocks). In other words, the motion vectors of the reconstructed spatially surrounding blocks and / or the motion vectors corresponding to the temporally surrounding blocks are used as candidates for motion vector predictors.

[0169] The prediction information may include selection information (e.g., an MVP flag or MVP index) that indicates the optimal motion vector predictor candidate selected from among the motion vector predictor candidates included in the list. At this time, the prediction unit can use the selection information to select the motion vector predictor for the current block from among the motion vector predictor candidates included in the motion vector candidate list. The prediction unit of the encoding device can calculate the difference in motion vectors (MVD) between the motion vector of the current block and the motion vector predictor, encode this, and output it in the form of a bitstream. In other words, the MVD can be obtained by subtracting the motion vector predictor from the motion vector of the current block. At this time, the prediction unit of the decoding device can obtain the difference in motion vectors included in the prediction information and derive the motion vector of the current block by adding the difference in motion vectors and the motion vector predictor. The prediction unit of the decoding device can obtain or derive the index of a reference picture that indicates a reference picture from the prediction information. For example, the motion vector predictor candidate list can be configured as shown in Figure 14.

[0170] Affine motion prediction

[0171] Figure 15 shows an example of a motion model according to an embodiment of the present invention.

[0172] Conventional video compression technologies (e.g., HEVC (high efficiency video coding)) use a single motion vector to represent the motion of a coded block. Even if a method using one motion vector per block can represent the optimal motion at the block level, it may not represent the optimal motion of each actual pixel. Therefore, if the optimal motion vector can be determined at the pixel level, coding efficiency can be improved. The present invention then describes a motion prediction method for encoding or decoding a video signal using multiple motion models. In particular, motion vectors can be represented at the pixel level or sub-block level of a block using motion vectors of two to four control points, and such prediction methods using motion vectors of multiple control points can be referred to as affine motion prediction or affine prediction.

[0173] The affine motion model according to an embodiment of the present invention can represent four motion models as shown in Figure 15. Three of the motions that the affine motion model can represent (translation, scale, rotate) dmf are referred to as the similarity (or simplified) affine motion model. For the sake of convenience, the embodiment of the present invention will be described based on the similarity (or simplified) affine motion model, but the present invention is not limited thereto.

[0174] Figure 16 shows an example of a motion vector of a control point for affine motion prediction according to an embodiment of the present invention.

[0175] As shown in Figure 16, affine motion prediction can determine the motion vector of the pixel position (or subblock) containing a block using a pair of control point motion vectors (CPMV), v_0 and v_1. This set of motion vectors can be referred to as the affine motion vector field (MVF). The affine motion vector field can be determined using the following equation 1.

[0176]

number

[0177] In Equation 1, v_0(v_0={v_0x,v_0y}) represents the motion vector (CPMV0) of the first control point at the upper left position of the current block 1300, and v_1(v_1={v_1x,v_1y}) represents the motion vector (CPMV1) of the second control point at the upper right position of the current block 1300. Also, w represents the width of the current block 1300. v(v={v_x,v_y}) represents the motion vector at position {x,y}. Motion vectors at the subblock (or pixel) level can be derived using Equation 1. In one embodiment, the accuracy of the motion vector can be rounded to an accuracy of 1 / 16.

[0178] Figure 17 shows an example of motion vectors for each subblock of a block to which affine motion prediction according to an embodiment of the present invention has been applied.

[0179] Referring to Figure 17, the affine motion vector field (MVF) can be determined on a pixel-by-pixel or block-by-block basis during the encoding or decoding process. In other words, in affine motion prediction, the motion vector of the current block can be derived on a pixel-by-pixel or sub-block basis.

[0180] When the affine motion vector field is determined on a pixel-by-pixel basis, a motion vector can be obtained based on the value of each pixel. When it is determined on a block basis, the motion vector for that block can be obtained based on the value of the central pixel of that block. In this document, it is assumed that the affine motion vector field (MVF) is determined in 4*4 block units, as shown in Figure 17. However, this is for the sake of explanation and is not limited to the embodiments of the present invention. Figure 17 shows an example where the encoded block consists of 16*16 samples and the affine motion vector field (MVF) is determined in 4*4 block units.

[0181] Affine motion prediction can include affine merge mode (AF_MERGE) and affine inter mode (AF_INTER). AF_INTER mode can include AF_4_INTER mode, which uses a four-parameter-based motion model, and AF_6_INTER mode, which uses a six-parameter-based motion model.

[0182] Affine merge mode

[0183] AF_MERGE determines the control point motion vector (CPMV) based on the affine motion model of surrounding blocks coded as affine motion predictions. Surrounding blocks that are affine-coded in the search order are used for AF_MERGE. When one or more adjacent blocks are coded as affine motion predictions, the current block can be coded as AF_MERGE.

[0184] In other words, when affine merge mode is applied, the CPMV of the current block can be derived using the CPMV of the surrounding blocks. In this case, the CPMV of the surrounding blocks can be used directly as the CPMV of the current block, or the CPMV of the surrounding blocks can be modified based on the size of the surrounding blocks and the size of the current block, etc., and then used as the CPMV of the current block.

[0185] Figure 18 shows an example of a peripheral block used for affine motion prediction in the affine merge mode according to an embodiment of the present invention.

[0186] In affine merge (AF_MERGE) mode, the encoder can perform encoding in the following process:

[0187] Step 1: Scan the surrounding blocks A through E (1810, 1820, 1830, 1840, 1850) of the currently coded block 1800 in alphabetical order. Based on the scan order, determine the first block encoded in affine prediction mode as the candidate block for affine merge (AF_MERGE).

[0188] Step 2: Determine the affine motion model using the motion vectors (CPMV) of the control points of the selected candidate blocks.

[0189] Step 3: Based on the affine motion model of the candidate block, the motion vectors (CPMV) of the control points of the current block 1800 are determined, and the MVF of the current block 1800 is determined.

[0190] Figure 19 shows an example of a block to which affine motion prediction is performed using a surrounding block to which affine motion prediction according to an embodiment of the present invention is applied.

[0191] For example, as shown in Figure 19, if block A1920 is encoded in affine mode, after determining block A1920 as a candidate block, the affine motion model can be derived using the motion vectors (CPMV) of block A1920's control points (e.g., v2, v3), and then the motion vectors (CPMV) v0 and v1 of the current block 1900's control points can be determined. Based on the motion vectors (CPMV) of the current block 1900's control points, the affine motion vector field (MVF) of the current block 1900 can be determined and encoded.

[0192] Figure 20 is a diagram illustrating a method for generating a merge candidate list using peripheral affine coding blocks according to an embodiment of the present invention.

[0193] Referring to Figure 20, when determining CPMV pairs using affine merge candidates, the candidates shown in Figure 20 are used. In Figure 20, it is assumed that the scan order of the candidate list is set to A, B, C, D, E. However, the present invention is not limited to this, and various orders can be pre-set.

[0194] As an example, if the number of candidates encoded with the affine mode (or affine prediction) used in surrounding blocks (i.e., A, B, C, D, E) (hereinafter referred to as affine candidates) is 0, the affine merge mode for the current block may be skipped. If there is only one affine candidate used (for example, A), the motion model of that candidate is used to derive the motion vectors (CPMV_0 and CPMV_1) of the control points of the current block. In this case, an index pointing to that candidate may not be required (or coded). If there are multiple affine candidates used, two candidates in scan order may constitute the candidate list of AF_MERGE. In this case, candidate selection information, such as an index pointing to the selected candidate in the candidate list, may be signaled. The selection information may be a flag or index information, and may be referred to as AF_MERGE_flag, AF_merge_idx, etc.

[0195] In the embodiments herein, motion compensation for the current block can be performed based on the size of the subblocks. In this case, the subblock size of the affine block (i.e., the current block) is derived. If the width and height of all subblocks are greater than four luma samples, the motion vector for each subblock is derived, and DCT-IF-based motion compensation (1 / 16 of luminance and 1 / 32 of chrominance) can be performed on the subblocks. Otherwise, enhanced bi-linear interpolation filter-based motion compensation can be performed on the entire affine block.

[0196] In the embodiments herein, when the merge / skip flag is true and the width and height of the CU are all greater than or equal to 8, the affine flag is signaled at the CU level via a bitstream indicating that the affine merge mode should be used. When the CU is coded as AF_MERGE, a merge candidate index with a maximum value of "5" is signaled to specify that a motion information candidate from the affine merge candidate list should be used for the CU.

[0197] Figures 21 and 22 illustrate a method for constructing an affine merge candidate list using peripheral blocks encoded in an affine prediction according to an embodiment of the present invention.

[0198] Referring to Figure 21, the affine merge candidate list is structured as follows:

[0199] 1) Insert model-based affine candidates

[0200] Model-based affine candidates mean that the candidates are derived from valid peripherally reconstructed blocks coded in affine mode. As shown in Figure 21, the scan order of the candidate blocks is from left (A), top (b), top right (C), and bottom left (D) to top left (E).

[0201] When the peripheral lower-left block (A) is encoded in 6-parameter affine mode, we obtain the motion vectors (v_4, v_5, v_6) for the upper-left, upper-right, and lower-left corners of the CU containing block (A). The motion vector (v_0, v_1, v_2) for the upper-left corner on the current block is calculated according to the motion vectors (v_4, v_5, and v_6) obtained by the 6-parameter affine model.

[0202] When the peripheral lower left block (A) is coded in 4-parameter affine mode, we obtain the motion vectors (v_4, v_5) for the upper left and upper right corners of the CU containing block (A). The motion vector (v_0, v_1) of the current upper left corner on the block is calculated according to the motion vector (v_4, v_5) from the 4-parameter affine model.

[0203] 2) Insert control point-based affine candidates

[0204] Referring to Figure 21, the control point-based candidates are constructed by combining motion information from around each control point.

[0205] Motion information for a control point is first derived from the specified spatial and temporal adjacent blocks shown in Figure 21. CP_k(k = 1, 2, 3, 4) represents the k-th control point. A, B, C, D, E, F, and G are the spatial positions for predicting CP_k(k = 1, 2, 3), and H is the temporal position for predicting CP4.

[0206] The coordinates of CP_1, CP_2, CP_3, and CP_4 are (0,0), (W,0), (H,0), and (W,H), respectively, where W and H are the width and height of the current block.

[0207] Motion information for each control point is obtained according to the following priority order.

[0208] For CP_1, the checking priority is A->B->C. If A is used, then A is used. If B is used instead, then B is used. If neither A nor B are used, then C is used. If none of the three candidates are used, then motion information for CP1 cannot be obtained.

[0209] For CP_2, the checking priority is E->D.

[0210] For CP_3, the checking priority is G->F.

[0211] H is used for CP_4.

[0212] Secondly, combinations of control points are used to construct a motion model.

[0213] The motion vectors of two control points are necessary to calculate the transformation parameters in a 4-parameter affine model. The two control points can be selected from one of the following six combinations: {CP_1, CP_4}, {CP_2, CP_3}, {CP_1, CP_2}, {CP_2, CP_4}, {CP_1, CP_3}, {CP_3, CP_4}. For example, using control points CP_1 and CP_2 to construct a 4-parameter affine motion model is denoted as "Affine(CP_1, CP_2)".

[0214] The motion vectors of the three control points are necessary to calculate the transformation parameters in the 6-parameter affine model. The three control points can be selected from one of the following four combinations: {CP_1, CP_2, CP_4}, {CP_1, CP_2, CP_3}, {CP_2, CP_3, CP_4}, {CP_1, CP_3, CP_4}. For example, using control points CP_1, CP_2, and CPv3 to construct a 6-parameter affine motion model is written as "Affine(CP_1, CP_2, CP_3)".

[0215] Furthermore, in the embodiments described herein, if an affine merge candidate exists in the affine merge mode, it may always be considered as a 6-parameter affine mode.

[0216] Affine inter mode

[0217] Figure 23 shows an example of a peripheral block used for affine motion prediction in the affine inter mode according to an embodiment of the present invention.

[0218] Referring to Figure 23, affine motion prediction can include affine merge mode (AF_MERGE) and affine inter mode (AF_INTER). In affine inter mode (AF_INTER), after determining the motion vector predictions (CPMVP) and CPMV of two control points, the control point motion vector difference (CPMVD) corresponding to the difference can be transmitted from the encoder to the decoder. The specific encoding process for affine inter mode (AF_INTER) can be the same as the following.

[0219] Step 1: Determine two CPMVP pair candidates.

[0220] Step 1.1: Determine a maximum of 12 combinations of CPMVP candidates (see formula 2 below).

[0221]

number

[0222] In equation 2, v_0 is the motion vector (CPMV0) at the upper left control point 2310 of the current block 2300, v_1 ​​is the motion vector (CPMV1) at the upper right control point 2311 of the current block 2300, v_2 is the motion vector (CPMV2) at the lower left control point 2312 of the current block 2300, v_A is the motion vector of the surrounding block A2320 adjacent to the upper left side of the upper left control point 2310 of the current block 2300, v_B is the motion vector of the surrounding block B2322 adjacent to the upper left side of the upper left control point 2310 of the current block 2300, and vC is the motion vector of the current block 230 v_D is the motion vector of the surrounding block C2324 adjacent to the left of the upper left control point 2310 of block 0, v_E is the motion vector of the surrounding block D2326 adjacent to the upper right control point 2311 of the current block 2300, v_E is the motion vector of the surrounding block E2328 adjacent to the upper right of the upper right control point 2311 of the current block 2300, v_F is the motion vector of the surrounding block F2330 adjacent to the left of the lower left control point 2312 of the current block 2300, and v_G is the motion vector of the surrounding block G2332 adjacent to the left of the lower left control point 2312 of the current block 2300.

[0223] Step 1.2: Sort the CPMVP candidate combinations by the smallest difference value (DV), and use the top two candidates (see formula 3 below).

[0224]

number

[0225] v_0x is the x-axis element of the motion vector (V0 or CPMV0) of the upper left control point 2310 of the current block 2300, v_1x is the x-axis element of the motion vector (V1 or CPMV1) of the upper right control point 2311 of the current block 2300, v_2x is the x-axis element of the motion vector (V_2 or CPMV_2) of the lower left control point 2312 of the current block 2300, v_0y is the x-axis element of the upper left control point 2310 of the current block 2300 v_1y is the y-axis element of the motion vector (V_0 or CPMV_0), v_1y is the y-axis element of the motion vector (V_1 or CPMV_1) of the upper right control point 2311 of the current block 2300, v_2y is the y-axis element of the motion vector (V_2 or CPMV_2) of the lower left control point 2312 of the current block 2300, w is the width of the current block 2300, and h is the height of the current block 2300.

[0226] Step 2: If the number of control point motion vector predictor (CPMVP) pair candidates is less than 2, use the AMVP candidate list.

[0227] Step 3: For each of the two candidates, determine the motion vector predictor (CPMVP) for the control point, compare the RD costs, and optimally select the candidate with the smaller value for the CPMV.

[0228] Step 4: Transmission of the control point motion vector difference (CPMVD) values ​​between the index corresponding to the optimal candidate and the control point.

[0229] In the embodiments described herein, AF_INTER provides a process for configuring CPMVP candidates. Similar to AMVP, there are two candidates, and an index indicating the position in the candidate list is signaled.

[0230] The procedure for setting up the CPMVP candidate list is as follows:

[0231] 1) Scan surrounding blocks to check if they can be coded as affine motion predictions. If the scanned block is coded as an affine prediction, derive motion vector pairs for the current block from the affine motion models of the scanned surrounding blocks until the number of candidates reaches 2.

[0232] 2) If the number of candidates is less than 2, the candidate configuration process is executed. In addition, in the embodiment of the present invention, a 4-parameter (2-control point) affine intermode is used to predict zoom-in / out and rotation motion models and content. As shown in Figure 16, the affine motion field of a block is described by the motion vectors of two control points.

[0233] The motion vector field (MVF) of a block is described by Equation 1 above.

[0234] In the prior art, AMVP (advanced motion vector prediction) mode is required to signal the MVP (motion vector prediction) index and MVDs (motion vector differences). When AMVP mode is applied to the present invention, the affine_flag is signaled to indicate whether affine prediction should be used. When affine prediction is applied, the syntax of inter_dir, ref_idx, mvp_index, and two MVDs (mvd_x and mvd_y) is signaled. An affine MVP pair candidate list containing two affine MVP pairs is generated. The signaled mvp_index is used to select one of these. An affine MVP pair is generated by two types of affine MVP candidates: one is a spatially inherited affine candidate, and the other is a corner-derived affine candidate. When a peripheral CU is coded in affine mode, a spatially inherited affine candidate can be generated. The affine motion model of the surrounding affine-coded block is used to generate motion vectors for two-control-point MVP pairs. The MV of the two-control-point MVP pair of the spatially inherited affine candidate is derived using the following equation.

[0235]

number

[0236]

number

[0237] If V_B0, V_B1, and V_B2 can be replaced by the upper-left MV, upper-right MV, and lower-left MV of any reference / peripheral CU, then (posCurCU_X, posCurCU_Y) is the position of the upper-left sample of the current CU for the upper-left sample of the frame, and (posRefCU_X, posRefCU_Y) is the position of the upper-left sample of the reference / peripheral CU for the upper-left sample of the frame.

[0238]

number

[0239]

number

[0240] Figure 24 shows an example of a peripheral block used for affine motion prediction in an affine inter mode according to an embodiment of the present invention.

[0241] Referring to Figure 24, if the number of MVP pairs is less than 2, the corner-derived affine candidate is used. The surrounding motion vectors are used to derive the affine MVP pair as shown in Figure 24. For the first corner-derived affine candidate, the first available MV in set A (A0, A1 and A2) and the first available MV in set B (B0 and B1) are used to form the first MVP pair. For the second corner-derived affine candidate, the first available MV in set A and the first available MV in set C (C0 and C1) are used to calculate the MV of the upper right side control point. The first available MV in set A and the calculated upper right side control point MV constitute the second MVP pair.

[0242] In an embodiment of the present invention, two candidate sets including two (three) candidates {mv_0, mv_1} ({mv_0, mv_1, mv_2}) are used to predict two (three) control points of an affine motion model. Given motion vector differences (mvd_0, mvd_1, mvd_2) and control points are calculated by using the following equations.

[0243] [Number]

[0244] FIG. 25 and FIG. 26 are diagrams illustrating a method of deriving candidates for motion vectors by using motion information of peripheral blocks in an affine inter mode according to an embodiment of the present invention.

[0245] The affine candidate list extends the affine motion from spatially adjacent blocks (extrapolated affine candidates) and is appended by a combination of motion vectors from spatially adjacent blocks (virtual affine candidates). The candidate set is set as follows:

[0246] 1. Up to two different sets of affine MV predictors are derived from the affine motion of adjacent blocks. Adjacent blocks A0, A1, B0, B1, and B2 are identified as shown in FIG. 25. If an adjacent block is encoded by an affine motion model and its reference frame is the same as the reference frame of the current block, two (for a 4-parameter affine model) or three (for a 6-parameter affine model) control points of the current block are derived from the affine model of the adjacent block.

[0247] 2. Figure 29 shows the adjacent blocks used to generate a virtual affine candidate set. The adjacent MVs are divided into three groups: S_0 = {mv_A, mv_B, mv_C}, S_1 = {mv_D, mv_E}, S_2 = {mv_F, mv_G}. mv_0 is the first MV in S0 that references the same reference picture as the current block. mv_2 is the first MV in S1 that references the same reference picture as the current block.

[0248] If mv_0 and mv_1 are given, mv_2 can be derived by the following Equation 9.

[0249]

Equation

[0250] In Equation 9, the current block size is WxH.

[0251] If only mv_0 and mv_2 are given, mv_1 can be derived by the following Equation 10.

[0252]

Equation

[0253] In one embodiment of the present specification, the affine inter prediction can be performed based on the following sequence.

[0254] Input: Affine motion parameter, reference picture sample

[0255] Output: Predicted block of CU [[ID=4​​​​​​​​​- If the width of the sub-block is greater than the width mode 4 luma samples,

[0259] - About each subblock

[0260] - Induce the motion vector of the sub-block

[0261] - DCT-IF based motion compensation (1 / 16 pel for luma, 1 / 32 pel for chromatic difference) is invoked for sub-blocks.

[0262] - Otherwise, enhanced bi-linear interpolation filter-based compensation is invoked over the entire affine block.

[0263] Furthermore, in one embodiment of this specification, if the merge / skip flag is false and the width and width of the CU are 8 or greater, the affine flag is signaled at the CU level to indicate whether the affine intermode is used. When the CU is coded as the affine intermode, the model flag is signaled to indicate whether a 4-parameter or 6-parameter affine model is applied to the CU. If the model flag is true, the AF_6_INTER mode (6-parameter affine model) is applied and three MVDs are parsed; otherwise, the AF_4_INTER mode (4-parameter affine model) is applied and two MVDs are parsed.

[0264] In AF_4_INTER mode, similar to affine merge mode, extrapolated motion vector pairs from adjacent blocks coded in affine mode are generated and initially inserted into the candidate list.

[0265] Thereafter, when the size of the candidate list is smaller than 4, candidates having a pair of motion vectors {(v_0, v_1) | v0 = {v_A, v_B, v_c}, v_1 = {v_D, v_E}} are generated by using adjacent blocks. As shown in FIG. 22, v_0 is selected from the motion vectors of blocks A, B, and C. The motion vectors from the adjacent blocks are scaled based on the relationship between the reference list, the POC of the reference of the adjacent block, the POC of the reference of the current CU, and the current CU. And the approach for selecting v_1 from the adjacent blocks D and E is similar. If the candidate list is larger than 4, the candidates are preferentially aligned based on the consistency of the adjacent motion vectors (similar to two motion vectors in a candidate pair), and the first 4 candidates are saved.

[0266] If the number of the candidate list is smaller than 4, the list is padded by a motion vector pair by replicating each AMVP candidate.

[0267] In the AF_6_INTER mode, similar to the affine merge mode, extrapolated affine motion vector triples from adjacent blocks coded in the affine mode are generated and preferentially inserted into the candidate list.

[0268] Thereafter, if the size of the candidate list is less than 4, candidates containing the motion vector triple {(v_0, v_1, v_2)| v0 = {v_A, v_B, v_c}, v1 = {v_D, v_E}, v2 = {v_G, v_H}} are generated using the adjacent blocks. As shown in Figure 22, v_0 is selected from the motion vectors of blocks A, B, or C. The motion vectors from adjacent blocks are scaled based on the relationship between the reference list and the POC of the adjacent block references, the POC of the current CU references, and the POC of the current CU. The approach to selecting v_1 from adjacent blocks D and E is similar to selecting v_2 from F and G. If the candidate list is greater than 4, the candidates are sorted based on the consistency of the adjacent motion vectors (similar to 2 motion vectors from 3 candidates), and the first 4 candidates are saved.

[0269] If the number of candidates in the candidate list is less than 4, the list can be padded by motion vector triples that are constructed by duplicating each AMVP candidate.

[0270] After the CPMV of the current CU is induced, the MVF of the current CU is generated based on equation 11 below for the 4-parameter affine model and based on equation 12 below for the 6-parameter affine model, depending on the number of affine parameters.

[0271]

number

[0272]

number

[0273] Here, the sub-block size MxN is derived from equation 13 below, where MvPre is the precision of the motion vector portion (1 / 16).

[0274]

number

[0275] After being derived by Equation 12, M and N should be modified downward if necessary to create the denominators (divisors) of w and h. If M or N is less than 8, WIF is applied; otherwise, sub-block based affine motion compensation is applied.

[0276] Figure 27 shows an example of a method for inducing an affine motion vector field at the subblock level according to an embodiment of the present invention.

[0277] Referring to Figure 27, in order to derive the motion vector for each MxN sub-block, the motion vector of the central sample of each sub-block is calculated based on Equation 11 or Equation 12, as shown in Figure 27, and rounded to a precision of 1 / 16. An SHVC upward sampling interpolation filter is applied using the derived motion vector to generate predictions for each sub-block.

[0278] An SHVC upward sampling interpolation filter, having the same filter length and normalization factor as the HEVC motion-compensated interpolation filter, is used as a motion-compensated interpolation filter for additional fractional pel positions. The accuracy of the motion vector of the chroma component is 1 / 32 sample, and the additional interpolation filter for the 1 / 32 pel position is induced by using the average of the filters for two adjacent 1 / 16 pel positions.

[0279] The AF_MERGE mode can be selected on the encoder side in the same way that normal merge mode selection is performed. A candidate list is preferentially generated, and the candidate with the lowest RD-cost is selected for comparison with the RD-costs of different intermodes. The result of the comparison determines whether or not AF_MERGE is applied.

[0280] For AF_4_INTER mode, RD cost verification is used to determine which candidate pair of motion vectors is selected as the control point motion vector prediction (CPMVP) for the current CU. After the CPMVP of the current affine CU is determined, affine motion estimation is applied to obtain the control point motion vector (CPMV). Then, the difference between the CPMV and the CPMVP is determined.

[0281] On the encoder side, AF_6_INTER mode is only checked when AF_MERGE or AF_4_INTER mode is determined to be the optimal mode in the previous mode selection stage.

[0282] In one embodiment of this specification, the affine inter (affine AMVP) mode can be implemented as follows:

[0283] 1) AFFINE_MERGE_IMPROVE: Instead of searching for the first adjacent block in affine mode, the improvement attempts to find the adjacent block with the largest coding unit size as a candidate for affine merge.

[0284] 2) AFFINE_AMVL_IMPROVE: Adds adjacent blocks in affine mode to the affine AMVP candidate list in a manner similar to a normal AMVP procedure.

[0285] The detailed process for generating the Affine AMVP candidate list is as follows:

[0286] First, the lower left adjacent block is checked using the affine motion model to see if it has the same reference index as the current reference index. If it does not, the left adjacent block is checked in the same way. If it does not exist, the lower left adjacent block is checked using the affine motion model to see if it has a different reference index. If it does, the scaled affine motion vector is added to the reference picture list. If it does not exist, the left adjacent block is checked in the same way.

[0287] Secondly, the adjacent block on the upper right, the adjacent block at the top, and the adjacent block on the upper left are checked in the same way.

[0288] After the process described above, if two candidates are found, the process of generating the affine AMVP candidate list will terminate. If two candidates are not found, the original operation within the JEM software will be performed to generate the affine AMVP candidate list.

[0289] 3) AFFINE_SIX_PARAM: In addition to the 4-parameter affine motion model, a 6-parameter affine motion model is added as an additional model.

[0290] The 6-parameter affine motion model is derived via the following equation 14.

[0291]

number

[0292] Since the aforementioned motion model has 6 parameters, three motion vectors are required to determine the model: MV_0 at the upper left position, MV_1 at the upper right position, and MV_2 at the lower left position. These three motion vectors can be determined in a similar way to two motion vectors in a 4-parameter affine motion model. Affine model merging is always set as a 6-parameter affine motion model.

[0293] 4) AFFINE_CLIP_REMOVE: Removes the constraints on all affine motion vectors. The motion compensation process controls the constraints on the motion vectors themselves.

[0294] Affine motion model

[0295] As mentioned above, various affine motion models can be used or considered in affine inter prediction. For example, an affine motion model can represent four types of motion, as shown in Figure 15 above. An affine motion model that represents three of the motions that an affine motion model can represent (translation, scale, and rotate) can be called a similarity (or simplified) affine motion model. Depending on which of the above affine motion models is used, the number of CPMVs derived and / or the method of deriving the sample / subblock MV for the current block may differ.

[0296] In one embodiment of the present invention, adaptive four-parameter and six-parameter motion models are used. In AF_INTER, a six-parameter motion model is provided in addition to the four-parameter motion model that exists in JEM. The six-parameter affine motion model is described by the following equation 15.

[0297]

number

[0298] Here, the coefficients a, b, c, d, e, and f are affine motion parameters, and (x, y) and (x', y') are the coordinates of the pixel position before and after the transformation of the affine motion model. In order to use the affine motion model for video coding, if CPMV0, CPMV1, and CPMV2 are MVs of CP0 (upper left), CP1 (upper right), and CP2 (lower left), then equation 16 can be explained as follows.

[0299]

number

[0300] Here, CPMV_0 = {v_0x, v_0y}, CPMV_1 = {v_1x, v_1y}, CPMV_2 = {v_2x, v_2y}, and w and h are the width and height of the coding block, respectively. Equation 16 is the motion vector field (MVF) of the block.

[0301] The flag is parsed at the CU level to indicate whether to use a 4-parameter or 6-parameter affine motion model when adjacent blocks are coded with affine prediction. If there are no adjacent blocks coded with affine prediction, the flag is omitted and the 4-parameter model is used for affine prediction. In other words, the 6-parameter model is considered when one or more adjacent blocks are coded with the affine motion model. In relation to the number of CPMVDs, two and three CPMVDs are signaled for the 4-parameter and 6-parameter affine motion models, respectively.

[0302] Furthermore, in one embodiment of the present invention, pattern-matched motion vector refinement can be used. In JEM's pattern-matched motion vector induction (referred to as PMMVD in the description of the JEM encoder, hereinafter abbreviated as PMVD), the decoder needs to evaluate several motion vectors (MVs) to determine a starting MV candidate for CU-level search. In sub-CU-search, several additional MV candidates are added in addition to the optimal CU-level MV. The decoder needs to evaluate such MV candidates to search for the optimal MV, which requires a lot of memory bandwidth. In the proposed pattern-matched motion vector refinement (PMVR), the concepts of template matching and bilateral matching in PMVD in JEM are employed. A PMVR_flag is signaled when skip mode or merge mode is selected to indicate whether PMVR is available. To significantly reduce memory bandwidth requirements compared to PMVD, a list of MV candidate indexes is generated, and if PMVR is applied, the starting MV candidate index is explicitly signaled.

[0303] The merge candidate list generation process generates a candidate list, but sub-CU merge candidates, such as affine candidates and ATMVP candidates, are excluded.

[0304] For bilateral matching, only a single uni-prediction MV candidate is included. The bu-prediction MV candidate is split into two uni-prediction MV candidates. Additionally, similar MV candidates (whose MV difference is less than a predefined threshold) are removed. For CU-level search, diamond search MV refinement is performed starting with the signaled MV candidates.

[0305] Sub-CU-level search is only available in bilateral matching merge mode. The search window for sub-CU-level search for all sub-CUs is the same as the search window for CU-level search. Therefore, no additional bandwidth is required for sub-CU-level search.

[0306] Template matching is also used to refine MVPs in the mode. In AMVP mode, two MVPs are generated using the HEVC MVP generation process, and one MVP index is signaled to select one of them. The selected MVP is further refined using template matching in PMVR. When adaptive motion vector resolution (AMVR) is applied, the MVP is rounded to the corresponding accuracy before template matching refinement. Such a refinement process is named pattern-matched motion vector predictor refinement (PMVPR). Unless otherwise defined in the remainder of this document, PMVR includes template-matched PMVR, bidirectional-matched PMVR, and PMVPR.

[0307] To reduce memory bandwidth requirements, PMVR is disabled for 4x4, 4x8, and 8x4 CUs. To further reduce additional memory bandwidth requirements, the search range for {template matching, bidirectional matching} in the same CU area as 64 can be reduced to {±2, ±4}, and the search range for {template matching, bidirectional matching} in larger CU areas can be reduced to {±6, ±8}. By employing all the methods described above in the PMVR section of this document, the required memory bandwidth was reduced from 45.9x with PMVD in JEM-7.0 to 3.1x with PMVR compared to the worst-case scenario with HEVC.

[0308] Affine is used in non-QT blocks.

[0309] Figure 28 illustrates the method by which prediction blocks are generated and the motion vectors in interpretation using an affine motion model according to an embodiment of the present invention.

[0310] Referring to Figure 28, we can see the formula for deriving the motion vector when an affine motion model is applied. The motion vector can be derived based on the following formula 17.

[0311]

number

[0312] Here, v_x represents the x-component of the sample-unit motion vector for the (x, y) coordinate sample in the current block, and v_y represents the y-component of the sample-unit motion vector for the (x, y) coordinate sample in the current block.

[0313] In other words, (v_x, v_y) is the sample-level motion vector for the sample at the (x, y) coordinates. Here, a, b, c, d, e, and f are parameters of a formula for deriving the sample-level motion vector (motion information) at the (x, y) coordinates from the control point (CP) of the current block. The CP can also be expressed as a control pixel. The parameters can be derived from the motion information of the CP of each PU transmitted in units of PUs. The formula for deriving the sample-level motion vector derived from the motion information of the CP described above can be applied to each sample of the block and can be derived to the position of the sample in the reference image based on the relative position of the x and y axes of each sample. The sample-level motion vector can be derived differently depending on the size of the block along the QTBT(TT) block division structure, whether it is asymmetric or symmetric, the position of the block, etc. Specific examples of this are shown in Figures 29 to 38 described later.

[0314] Figure 29 illustrates a method for performing motion compensation based on the motion vector of a control point according to an embodiment of the present invention.

[0315] Referring to Figure 29, we will explain assuming that the current block is a 2Nx2N block. For example, the motion vector of the top-left sample in the current block can be denoted as v_0. Also, using samples from the surrounding blocks adjacent to the current block as CPs, the motion vectors of each CP can be denoted as v_1 and v_2. That is, if the width and height of the current block are S, and the coordinates of the sample position at the top-left of the current block are (xp, yp), then the coordinates of CP0 can be (xp, yp), the coordinates of CP1 can be (xp + S, yp), and the coordinates of CP2 can be (xp, yp + S). The motion vector of CP0 can be denoted as v_0, the motion vector of CP1 as v_1, and the motion vector of CP2 as v_2. The motion vectors of the samples can be derived using the motion vectors of the CPs. The motion vectors of the samples can be derived based on the following equation 18.

[0316]

number

[0317] Here, v_x and v_y represent the x and y components of the motion vector of the sample at (x, y) coordinates within the current block, respectively; v_x0 and v_y0 represent the x and y components of the motion vector v_0 of CP0, respectively; v_x1 and v_y1 represent the x and y components of the motion vector v_1 of CP1, respectively; and v_x2 and v_y2 represent the x and y components of the motion vector v_2 of CP2, respectively. By using the formula for deriving the sample-level motion vector, such as formula 18 described above, the motion vector of each sample within the current block can be derived based on its relative position within the current block.

[0318] Figure 30 illustrates a method for performing motion compensation based on the motion vectors of control points in a non-square block according to an embodiment of the present invention.

[0319] Figure 30 illustrates the CPs of a block divided into Nx2N. Using the same method as described above for partitioning type 2Nx2N, a formula for deriving the sample-unit motion vector within the current block can be derived. In the process of deriving the formula, a width value that matches the shape of the current block can be used. Three CPs can be derived to derive the sample-unit motion vector, and the positions of the CPs can be adjusted as shown in Figure 30. That is, if the width and height of the current block are S / 2 and S, respectively, and the coordinates of the sample position at the top-left corner of the current block are (xp, yp), then the coordinates of CP0 can be (xp, yp), the coordinates of CP1 can be (xp + S / 2, yp), and the coordinates of CP2 can be (xp, yp + S). The sample-unit motion vector can be derived based on the following formula 19.

[0320]

number

[0321] Here, vx and vy represent the x and y components of the motion vector of a sample at (x, y) coordinates within the current block, respectively; v_x0 and v_y0 represent the x and y components of the motion vector v_0 of CP0, respectively; v_x1 and v_y1 represent the x and y components of the motion vector v_1 of CP1, respectively; and v_x2 and v_y2 represent the x and y components of the motion vector v_2 of CP2, respectively. Equation 3 shows a formula for deriving the sample-level motion vector, taking into account that the width of the current block is S / 2. Based on the relative position within the current block, each sample within the current block, partitioned from the CU based on partitioning type Nx2N, can have its motion vector derived using a formula for deriving the sample-level motion vector, such as Equation 19 described above.

[0322] Figure 31 illustrates a method for performing motion compensation based on the motion vectors of control points in a non-square block according to an embodiment of the present invention.

[0323] Figure 31 illustrates a block divided based on partitioning type 2NxN. Three CPs can be derived to derive the sample-unit motion vector, and the positions of the CPs can be adjusted as shown in Figure 31, with the height adjusted to S / 2 according to the current block shape shown in Figure 31. That is, if the width and height of the current block are S and S / 2, respectively, and the coordinates of the sample position at the top-left edge of the current block are (xp, yp), then the coordinates of CP0 can be (xp, yp), the coordinates of CP1 can be (xp + S, yp), and the coordinates of CP2 can be (xp, yp + S / 2). The sample-unit motion vector can be derived based on the following equation 20.

[0324]

number

[0325] Here, v_x and v_y represent the x and y components of the motion vector of the sample at (x, y) coordinates within the current block, respectively; v_x0 and v_y0 represent the x and y components of the motion vector v_0 of CP0, respectively; v_x1 and v_y1 represent the x and y components of the motion vector v_1 of CP1, respectively; and v_x2 and v_y2 represent the x and y components of the motion vector v_2 of CP2, respectively. Equation 4 shows a formula for deriving the sample-level motion vector, taking into account that the height of the current block is S / 2. Based on the relative position within the current block, each sample within the current block, partitioned from the CU based on partitioning type 2NxN, can have its motion vector derived using the formula for deriving the sample-level motion vector, such as Equation 4.18 described above.

[0326] Figures 32 to 38 illustrate a method for performing motion compensation based on the motion vectors of control points in a non-square block according to an embodiment of the present invention.

[0327] Figure 32 illustrates the CPs of an asymmetrical, current block. As shown in Figure 32, the width and height of the asymmetrical current block can be denoted as W and H, respectively. To derive the sample-unit motion vector, three CPs can be derived for each current block, and the coordinates of the CPs can be adjusted based on the width and height along the shape of the current block, as shown in Figure 32. That is, if the width and height of the current block are W and H, and the coordinates of the sample position at the top-left corner of each current block are (xp, yp), then the coordinates of CP0 can be set to (xp, yp), the coordinates of CP1 to (xp+W, yp), and the coordinates of CP2 to (xp, yp+H). In this case, the sample-unit motion vector within the current block can be derived based on the following equation 21.

[0328]

number

[0329] Here, v_x and v_y represent the x and y components of the motion vector of a sample at (x, y) coordinates within the current block, respectively; v_x0 and v_y0 represent the x and y components of the motion vector v_0 of CP0, respectively; v_x1 and v_y1 represent the x and y components of the motion vector v_1 of CP1, respectively; and v_x2 and v_y2 represent the x and y components of the motion vector v_2 of CP2, respectively. Equation 21 shows a formula for deriving an asymmetric, sample-level motion vector that takes into account the width and height of the current block.

[0330] On the other hand, according to the present invention, in order to reduce the amount of data for motion information of CPs indicated on a block basis, at least one motion information prediction candidate for a CP can be selected based on the motion information of surrounding blocks or surrounding samples of the current block. The motion information prediction candidate may be referred to as an affine motion information candidate or an affine motion vector candidate. The affine motion information candidate may include, for example, the contents disclosed in Figures 33 to 38.

[0331] ATMVP simplification

[0332] ATMVP (Advanced Temporal Motion Vector Prediction) is an interpretation using temporal motion information candidates, but it can also refer to an interpretation using motion information candidates proposed to improve existing TMVPs. In the embodiments herein, ATMVP may be referred to as a subblock time merge candidate, subblock-based time merge candidate, subblock time motion vector predictor, subblock-based time motion vector predictor, etc., without being limited by its name.

[0333] Specifically, TMVP, which uses the motion vector of the colPB at the right-bottom block of the current block or the center position of the current block as a candidate for time motion information, may not reflect the movement within the screen. On the other hand, when ATMVP is applied, the motion vector of the colPB at the position pointed to by the motion vector of the surrounding block is used as the ATMVP candidate.

[0334] As an example of how ATMVP can be applied, by checking the merge candidate configuration in order, the motion vector (temporal vector) of the first available spatial neighbor block can be found, and then the position pointed to by the temporal vector in the reference picture can be derived as col-PB (ATMVP candidate). In addition, the motion vector of the corresponding block can be used for each sub-block using the temporal vector. In this case, if no MV exists for a particular sub-block, the MV of the block located at the center of the corresponding block is used as the MV for the unavailable sub-block and stored as a representative MV.

[0335] Furthermore, various ATMVP simplification strategies have been proposed that utilize ATMVP but further reduce memory usage.

[0336] In one embodiment, memory usage can be reduced because the reference picture (specified in the slice segment header) for finding the reference block is limited to the same collocated picture. As an example, up to four reference pictures are used. To find the reference block, the temporal vector can be guided from one of the spatial candidates in a scan order. If the current candidate has the same collocated picture and the same reference picture, the search process can be terminated.

[0337] In one embodiment, modifications to ATMVP and / or STMVP sub-block merge modes can be applied.

[0338] 1. Signaling is added at the picture / slice level, and various subblock sizes are used for ATMVP / STMVP-based motion information derivation.

[0339] 2. By using one identical position picture, a simplified ATMVP-based motion information guidance system is employed.

[0340] In one embodiment, it is proposed to support picture / slice-level adaptation of the granularity of motion fields derived by ATMVP and STMVP modes. Specifically, the default subblock size used to derive ATMVP and STMVP motion parameters in a slice referring to a sequence parameter set (sps) can be signaled by sps. Additionally, one flag is signaled from the slice's segment header. If this flag is 0, it indicates that the default subblock size, as explicitly stated in sps, will be used for ATMVP / STMVP-based motion derivation in the current slice. Otherwise (i.e., if this flag is set to 1), other syntax elements can be signaled in the slice's segment header to explicitly specify the ATMVP / STMVP subblock size to use for the slice.

[0341] In one embodiment, a simple encoder method is used to determine the ATMVP / STMVP subblock size of the current picture. One of two ATMVP / STMVP subblock sizes, 4 and 8, is selected based on the average size of ATMVP / STMVP blocks from the last picture coded in the same temporal layer. Assume that the previously coded picture is in the k-th temporal layer and is the i-th picture containing N CUs coded in ATMVP and STMVP modes. Also assume that the sizes of these CUs are S_0, S_1, ..., S_(N-1). The average size of ATMVP / STMVP CUs is calculated as σ^k = (Σ_(i=0)^(N-1)S_i) / N. Then, when coding the (i+1)-th picture in the same k-th temporal layer, its ATMVP / STMVP subblock size g_(i+1)^k can be determined based on the following equation 22.

[0342]

number

[0343] In one embodiment, the size of the ATMVP / STMVP subblock 7 for the first picture in each temporal layer can always be set to 4.

[0344] In one embodiment, ATMVP can be induced using the following two steps.

[0345] (1) If a peripheral CU is available and its MV differs from the MV from the existing candidate list, the encoder / decoder can add motion vectors from the spatial peripheral CU in the order A1, B1, B0, and A0, as shown in Figure 12 described earlier. The number of available spatial candidates can be represented by N0.

[0346] (2) The first MV candidate from the N0 spatial candidates is used to determine the position from which to fetch the same position picture and the motion of each subblock (by adding the displacement due to the MV). If N0 is 0, the same position with zero motion and the same position signaled in the slice header is used to fetch the motion of each subblock.

[0347] The same-position pictures of different CUs in ATMVP may not always be the same when used by numerous reference pictures. Having different same-position pictures in ATMVP derivations for different CUs in the current picture means that the motion fields of numerous reference pictures need to be fetched, which is undesirable due to increased memory bandwidth. Therefore, in one embodiment, a simplified design is provided that uses the same-position picture as the same-position picture in ATMVP derivations, such as HEVC signaled in a slice header. At the block level, if the reference picture of a surrounding block A is different from this same-position picture, the MV of block A is scaled using the HEVC temporal MV scaling method, and the scaled MV of block A is used for ATMVP.

[0348] The motion vector used to fetch the motion field at the same location picture Rcol is denoted as MVcol. To minimize the impact of MV scaling, the MVs from the spatial candidate list used to derive MVcol are selected in this answer in the following way: If the reference picture of a candidate MV is the same location picture, this MV is selected and used as MVcol without any scaling. Otherwise, the MV with the reference picture closest to the same location picture is selected to derive MVcol with scaling.

[0349] HMVP (History-based Motion Vector Prediction) general

[0350] Generally, video compression techniques employ two main methods: exploiting spatial and temporal redundancy. For example, both HEVC (High Efficiency Video Coding) and VVC use two motion compression techniques as the basis for their intercoding: one is merge motion, and the other is AMVP (advanced motion vector prediction). Various modifications have been discussed to improve these two prediction modes. These include increasing the number of candidates, exploring more spatially expanded candidates, and examining temporal candidates at non-traditional locations. These two techniques construct a list of initially possible candidates, minimize the RD (rate distortion) cost, and signal the selected candidates from the bitstream.

[0351] In particular, in recent video compression technologies, History-based Motion Vector Prediction (HMVP) is discussed, which stores motion information from previously encoded blocks and uses the stored motion information to predict the motion of blocks to be encoded later. Such HMVPs can be added to a merge list (or merge candidate list) or an AMVP list (or AMVP candidate list).

[0352] The decoder maintains a look-up table (LUT) that operates in a FIFO (first in first out) system (or scheme) for HMVP. In this specification, the LUT may be referred to as a table, HMVP table, HMVP candidate table, buffer, HMVP buffer, HMVP candidate buffer, HMVP list, HMVP candidate list, etc., without being limited to its name. Specifically, when a non-affine prediction unit (PU) (or coding unit (CU)) is decoded, its motion information is stored in the LUT, and the decoder decodes the next PU. The stored motion information may include motion vectors in the x (horizontal) and y (vertical) directions, reference index information, and mode information.

[0353] The decoder can maintain a LUT in which progressively decoded motion information of non-affine candidates is stored. The size of the LUT can be limited to a predefined number of S candidates. In one embodiment, the LUT can be reset from the start of a slice, the start of a CTU row, or the start of a CTU.

[0354] HMVP can be applied in both merge mode and AMVP mode. A merge list can have B candidates, and an AMVP list can have 2 candidates. In conventional video compression techniques, a merge list consists of the following candidates: i) spatial candidates, ii) temporal candidates, iii) bidirectional prediction (Bi-Pred) candidates, and iv) zero motion candidates. Recently, methods for considering ATMVP (advanced motion vector prediction) as an additional candidate have been discussed. As an example, an ATMVP candidate can be inserted into the merge list before the temporal candidates. Candidates in the merge list are added until the maximum merge list size is reached. Duplicate candidates may not be added to the merge list.

[0355] The AMVP list can have two candidates inserted. For example, one of the two candidates may be selected from the available spatial candidates, the second from the temporal candidates, and if the list is not filled, a zero-motion vector candidate may be added.

[0356] HMVP is applied based on a FIFO (First-In, First-Out) process, where candidates are exited from the table in the same order they were entered in the LUT.

[0357] In one embodiment, when HMVP is applied to the merge list configuration, an HMVP candidate can be inserted (or added) to the third position in the list as follows:

[0358] 1. Spatial Candidate

[0359] 2. Temporal Candidate

[0360] 3. Up to S HMVP candidates for a LUT

[0361] 4. Combined Bi-Pred Candidate

[0362] 5. Zero Motion Vector Candidate

[0363] In one embodiment, when HMVP is applied to the configuration of the AMVP list, HMVP can be inserted as the third position after the time candidate, as follows:

[0364] 1. Spatial Candidate

[0365] 2. Temporal Candidate

[0366] 3. Up to K HMVP candidates

[0367] 4. Zero Motion Vector Candidate

[0368] Figure 39 is a flowchart illustrating a method for storing HMVP according to the embodiments of this specification.

[0369] Referring to Figure 39, the decoder decodes the current PU (or CU) (S3901).

[0370] The decoder checks whether the current PU is a block coded in non-affine mode (S3902). To facilitate the use of HMVP candidates, if the current PU is a block coded in affine mode, the decoder does not store the motion information of the current PU in the table.

[0371] If the current PU is a block coded in non-affine mode, the decoder stores (or updates) the motion information of the current PU in a table (S3903).

[0372] In the embodiments described herein, the HMVP table can be updated in two ways: i) unconstrained FIFO and ii) constrained FIFO. Electronically, duplicate information may exist, but the pruning process is not applied. This contributes to reducing the overall process complexity. This will be illustrated with reference to the diagram below.

[0373] Figure 40 is a diagram illustrating an HMVP table operating in a non-restrictive FIFO manner according to the embodiments of this specification.

[0374] Referring to Figure 40, candidates added to the table are added to the edge of the table (right side). On the other hand, candidates discarded from the table based on the FIFO method are located at the front of the table (left side, the oldest candidate).

[0375] If the table is not completely filled with the predefined maximum number of candidates at index L-1 (i.e., the edge), new candidates are added without any candidates being removed. On the other hand, if the table is already completely filled, i.e., it fills the maximum number of candidates in the table, the oldest candidate at the front edge of the table is removed and new candidates are added.

[0376] Figure 41 illustrates an HMVP table operating in a restricted FIFO scheme according to the embodiments of this specification.

[0377] Referring to Figure 41, when a restrictive FIFO is used, pruning is performed if adding a new candidate would cause any duplication. For example, if a candidate with duplicate motion information exists in the table, the duplicate candidate in the table can be removed and the motion information of the current candidate can be added.

[0378] Example 1

[0379] In many cases, recent historical motion data for HMVP candidates can overlap with the motion information of spatial candidates (or spatial neighbor candidates). Therefore, in this embodiment, we propose a method for setting the order in which candidates are added to AMVP or a merge list to be different from the order of the HMVP LUT index.

[0380] In the examples described herein, the candidate list can be efficiently constructed by adaptively adjusting the HMVP candidates, thereby reducing the number of signaling bins used for binarization and improving coding efficiency.

[0381] In other words, HMVP candidates added to the merge list or AMVP list may not be limited by the index within the HMVP list. As an example, Table 2 illustrates how to change the order in which HMVP candidates are added to the AMVP or merge list.

[0382] [Table 2]

[0383] Referring to Table 2, as mentioned above, recently inserted HMVP candidates are highly likely to be identical to the motion information of spatial candidates. Therefore, taking this into consideration, the order in which HMVP candidates are added can be predefined independently of the HMVP index.

[0384] In one embodiment, HMVP candidates starting from the nth candidate in the list can be added to the merge list or AMVP list. Table 3 illustrates the modified order in which candidates are added to the AMVP or merge list.

[0385] [Table 3]

[0386] Referring to Table 3, HMVP candidates can be added to the merge list or AMVP list from the second indexing stage.

[0387] In one embodiment, information about the order in which HMVP candidates are added within a LUT can be signaled from the encoder to the decoder. For example, such order information can be transmitted via high-level syntax (HLS). The high-level syntax may be, for example, a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit, and / or other suitable syntax data headers.

[0388] Table 4 below illustrates higher-level syntax structures to which the methods proposed herein can be applied.

[0389] [Table 4]

[0390] Referring to Table 4, a set_HMVP_order_flag of 1 indicates that set_HMVP_order_flag is present in the slice header of a non-IDR picture from the CVS. A set_HMVP_order_flag of 0 indicates that set_HMVP_order_flag is not present in the slice header and that adaptive HMVP is not used in the VCS.

[0391] Table 5 below illustrates slice segment header syntax structures to which the method proposed herein can be applied.

[0392] [Table 5]

[0393] Referring to Table 5, slice_HMVP_idx represents the index of the candidate order to be used. For example, slice_HMVP_idx = 0 can represent basic HMVP orders such as 0, 1, 2, 3. Similarly, an index value of 1 is used to represent the HMVP order 3, 2, 1, 0.

[0394] Example 2

[0395] In the embodiments described herein, we propose a method for using a long-term list in addition to an HMVP LUT for motion example prediction. This allows for an increase in the number of HMVP candidates that are retained. As an embodiment, a 2-HMVP table can be considered, where one is used as a long-term list to store general HMVP candidates and the other as a list to store additional candidates that require further retention.

[0396] Next, we will illustrate how to initialize and configure a long-term list (or long-term HMVP list).

[0397] - After decoding the first CTU in a CTU line, one or more subsequent CTU history MVs may be added to the long-term HMVP LUT. Such a long-term HMVP LUT may be used until the next CTU line or may not be updated.

[0398] - At the start of the next CTU row, the long-term HMVP LUT is used to initialize the regular HMVP LUT. This is because the HMVP candidates for the CTU can be more co-related from the start of the CTU row than the history MV at the end of the previous CTU row.

[0399] - The above process can be repeated.

[0400] Figure 42 illustrates the HMVP LUT and Long-Term HMVP LUT according to the embodiments of this specification.

[0401] Referring to Figure 42, there are two LUTs. One is the HMVP LUT (or general HMVP LUT, short-term HMVP LUT), and the other is the long-term HMVP LUT. When an HMVP candidate is added to both of the merged AMVP lists, it can be added from either the HMVP LUT or the long-term LUT, as shown in Figure 42.

[0402] The use of the aforementioned long-term LUT can be indicated through higher-level syntax by introducing new syntactic elements. For example, syntactic elements can reside in sequence parameter sets, picture parameter sets, slice headers, coding tree units, coding units, and / or other appropriate syntax data headers.

[0403] Example 3

[0404] In the embodiments described herein, we propose a method for considering flexibility for decoding when an HMVP candidate is added to an HMVP LUT. The encoder / decoder may consider the decision of one or more characteristics of the PU (or CU).

[0405] As an example, when an encoder / decoder adds an HMVP candidate to its table, it may consider the following: The encoder / decoder may add a candidate by considering the PU's mode (e.g., merge mode, affine mode, AMVP mode, etc.) and / or characteristics such as block size, individually or in combination. In one embodiment, other characteristics may also be considered. For example, the merge type (e.g., spatial candidate or temporal candidate) and whether it is a sub-PU may be considered as candidate selection criteria. The aforementioned selection criteria may be determined to reduce duplication with previous history (or previous HMVPs). For example, if a PU is coded in merge mode and the merge type is spatial merge, the decoder may not update the motion information of that PU in the HMVP LUT.

[0406] Figure 43 shows an example of a method for updating an HMVP LUT according to the embodiments described herein.

[0407] Referring to Figure 43, the encoder / decoder acquires the coded candidate motion information (S4301).

[0408] The encoder / decoder evaluates whether to update the candidate motion information in the LUT based on a predefined decision criterion (S4302). The decision criterion may include at least one characteristic of the candidate's mode (e.g., merge mode, affine mode, AMVP mode, etc.), the candidate's block size, and / or the candidate's merge type.

[0409] The encoder / decoder updates the LUT based on the decision criteria (S4303). In other words, if the candidate satisfies the predefined decision criteria, the encoder / decoder can add the motion information of the candidate to the LUT.

[0410] Example 4

[0411] In the embodiments described herein, we propose a limitation on redundancy checks for adding HMVP candidates to a merge list (or AMVP list). This limitation on redundancy checks can be implemented in several different ways.

[0412] In one embodiment, the encoder / decoder can limit the number of pruning checks for a specific number of candidates in the merge list. The encoder / decoder can limit the number of pruning checks for candidates from the first candidate in the merge list up to a specific number of candidates. For example, the encoder / decoder can perform a pruning process for candidates from the first candidate in the merge list up to a specific number of candidates, and in this case, the number of HMVP candidates subject to pruning checks can be limited to a predefined number.

[0413] Furthermore, in one embodiment, the encoder / decoder can limit pruning checks by performing them on specific types of merge candidates within the merge list. For example, when adding an HMVP candidate, the encoder / decoder can perform pruning checks only on the spatial candidates in the merge list. Or, for example, when adding an HMVP candidate, the encoder / decoder can perform pruning checks only on a subset of the spatial candidates in the merge list. The subset of spatial candidates can be predefined. For example, the predefined subset of spatial candidates may be at least one of the left-side adjacent spatial candidates and / or the upper-side adjacent spatial candidates. The embodiments herein are not limited thereto, and other types can be combined to limit the scope of pruning checks.

[0414] Figure 44 illustrates a method for limiting the number of HMVP candidates subject to pruning checks according to the embodiments of this specification.

[0415] Referring to Figure 44, in the embodiments of this specification, the number of HMVP candidates subject to pruning checks can be limited to M times.

[0416] The encoder / decoder, using HMVP candidates, can check whether there is redundancy in motion information between the first M candidates in the HMVP LUT and the merge candidates in the merge list when constructing a merge list.

[0417] Alternatively, the encoder / decoder can check for redundancy between the first M candidates in the HMVP LUT and the decoded PU motion information before adding the currently decoded PU motion information to the HMVP LUT.

[0418] Figure 45 shows an example of a method for performing a pruning check according to the embodiments of this specification.

[0419] Referring to Figure 45, the encoder / decoder obtains the motion information of the decoded candidate and determines (or decodes) the number of pruning checks (S4501, S4502).

[0420] The encoder / decoder performs pruning checks based on the determined number of pruning checks (S4503).

[0421] In one embodiment, information related to pruning checks can be signaled through higher-level syntax in the same manner as in Tables 4 and 5 above. In this case, the syntax elements transmitted from the encoder to the decoder can be included in any header that indicates the number of pruning checks that need to be performed. The higher-level syntax may be, for example, a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit, and / or other appropriate syntax data headers.

[0422] Example 5

[0423] In one embodiment of this specification, an efficient method for selecting HMVP candidates is proposed. When inserting history motion vector candidates (i.e., HMVP candidates) into a merge list (or AMVP list), a pruning check can be performed to ensure that the HMVP candidates do not overlap with existing merge lists.

[0424] Performing a full redundancy check between an M-sized merge list and an N-sized history LUT requires (M-1)xN checks. Therefore, in the embodiments herein, the number of HMVP candidates can depend on existing parallel merge candidates. For example, the number of HMVP candidates can depend on the number of spatial candidates present in the merge list. Alternatively, for example, the number of HMVP candidates can depend on the number of spatial and temporal candidates present in the merge list.

[0425] If there are further merge candidates present in the merge list, the number of HMVP candidates to perform pruning checks on can be reduced based on specific criteria (or rules) derived from the number of merge candidates and / or HVMPs in the merge list. This can reduce the number of duplicate checks in the worst-case scenario.

[0426] For example, in the case of a merge list with a size (or length) of 6, if the merge list is not full, it can contain up to 5 spaces or other merge candidates. Inserting HMVP candidates from a list of 6 HMVPs could, in the worst case, require 30 duplicate checks.

[0427] In one embodiment, an example related to the limitation on the number of HMVPs to check is shown in the following formula 23 and Table 6.

[0428]

number

[0429] [Table 6]

[0430] Referring to Table 6, by limiting the number of HMVPs subject to pruning checks to two, the number of duplicate checks for adding HMVPs in the worst-case scenario can be reduced from 30 to 12.

[0431] Example 6

[0432] In one embodiment of this specification, a method for constructing a merge list using history-based spatial temporal motion vector prediction (H-STMVP) is proposed. The H-STMVP represents a candidate derived from the average of two history-based spatial MVPs and TMVPs. The two spatial HMVPs can be obtained from an HMVP buffer, and the TMVP can be obtained from the current merge list. Here, the spatial candidate may be a candidate obtained from the last two coded MVs in the decoding order prior to the current block.

[0433] For example, the last coded MV (referred to herein as MV_L), the second-to-last coded MV (referred to herein as MV_(L-1)), and MV_TMVP are used to generate H-STMVP candidates to be inserted into the merge list.

[0434] If all three of the aforementioned candidates are used, the MV to be added to the merge list can be calculated by the following formula 24.

[0435]

number

[0436] As an example, if only two of the three candidates mentioned above are available, the H-STMVP can be generated by averaging only those two candidates. Similarly, if only one candidate is available, only that one candidate is used. If no candidates are available, the H-STMVP is not used to construct the merge list.

[0437] In one embodiment of this specification, we propose a method for obtaining the motion vector of an H-STMVP candidate using a method other than the aforementioned formula 24.

[0438] For example, instead of averaging three or more candidates at once, it may be computationally simpler to first average the spatial candidates and then use this result to re-average two candidates. An example of this is equivalent to the following formula.

[0439]

number

[0440] Alternatively, you can obtain the average value as follows:

[0441]

number

[0442]

number

[0443]

number

[0444] The encoder / decoder can first average two candidates, as shown in equations 25-27, and then use a third candidate to finally average the resulting values. Alternatively, the encoder / decoder can assign a higher importance / weight to a candidate, i.e., MV_L, by applying a shift operation of 2, as shown in equation 28. Using the aforementioned equations 25-28, the mean can be derived using only shift operations without division.

[0445] Example 7

[0446] In one embodiment of this specification, we propose a method for inducing H-STMVP using an arbitrary number (n) of spatial candidates instead of two history-based spatial candidates. These n candidates do not necessarily have to be in a consecutive decoding order. They can be selected based on arbitrary or certain rules.

[0447] Therefore, the aforementioned equation 24 can be expressed in a more general way, as shown in the following equation 29.

[0448]

number

[0449] In another embodiment, assuming the use of five spatial candidates, the influence of the increased spatial candidates used to generate the H-STMVP candidate can be minimized by improving the weight applied to the temporal candidate, thereby appropriately reflecting both the spatial and temporal candidates.

[0450] Therefore, to achieve this, the aforementioned objective can be attained by first averaging the spatial candidates together using the following equation 30, and then averaging MV_TMVP using the result.

[0451]

number

[0452] Example 8

[0453] In one embodiment of this specification, a method is proposed for adding weights (or weighting factors) to motion vector candidates used to induce H-STMVP. These weights may be determined empirically, by considering the time distance to a fixed reference frame, or by considering the position in a history table. For example, a new candidate may have more weights than a previous candidate.

[0454] In other words, in this embodiment, the aforementioned formula 24 can be expressed as the following formula 31.

[0455]

number

[0456] In this case, the weights can have the same value, or they can have values ​​that are unevenly distributed.

[0457] Example 9

[0458] In one embodiment of this specification, we propose a method for scaling the motion vector used to guide the H-STMVP candidate as a single reference picture.

[0459] Figure 46 illustrates a method for inducing H-STMVP candidates using motion vectors that reference different reference pictures, according to one embodiment of this specification.

[0460] Referring to Figure 46, we assume that the MV_L, MV_L-1, and MV_TMVP candidates each refer to (or indicate) different reference pictures. That is, Figure 46 shows that each candidate used to generate the H-STMVP candidate can have other reference indices and consequently other reference frames.

[0461] Because frames with nearby reference frames can inherently have a greater impact on the H-STMVP motion vectors, the average of equations 24 through 31 mentioned above can result in uneven values. Therefore, to ensure equal comparison and reflection, we propose a method for scaling all motion vectors to a single reference frame.

[0462] At this point, the encoder can perform RD optimization to determine which single frame is best suited to be used as the reference frame. For example, the selected reference frame can be signaled with a slice header similar to the index of the TMVP sequence present in the slice header. For instance, it may be possible to generate a reference frame to be used using a fixed rule. Alternatively, it could be scaled to the first available reference frame from L0, or scaled by the current picture order count.

[0463] In one embodiment, to achieve the aforementioned objectives, an encoder can transmit information about a single fixed picture to a decoder using a higher-level syntax (HLS) which can be part of a sequence parameter set, a picture parameter set, a slice header, a coded tree unit, and / or other data headers. For example, a higher-level syntactic structure like those shown in Table 7 and / or Table 8 can be defined.

[0464] [Table 7]

[0465] Referring to Table 7, if set_HSTMVP_ref_pic_flag is equal to 1, it indicates that set_HSTMVP_idx is present in the slice header of a non-IDR picture from CVS. If set_HSTMVP_ref_pic_flag is 0, it indicates that set_HSTMVP_idx is not present in the slice header.

[0466] [Table 8]

[0467] Referring to Table 8, slice_HMVP_idx specifies the reference index. In one example, the reference index can be selected for list L0.

[0468] The embodiments described above in this specification have been described separately for the sake of clarity, but the present invention is not limited thereto. In other words, the embodiments described in Examples 1 to 9 above may be carried out independently, or one or more embodiments may be carried out in combination.

[0469] Figure 47 is a flowchart illustrating a method for processing a video signal based on interpretation according to an embodiment to which the present invention is applied.

[0470] Referring to Figure 47, the explanation will focus on the decoder for convenience, but the present invention is not limited thereto, and the interprediction-based video signal processing method according to the embodiments herein can be performed in the same way with an encoder and a decoder.

[0471] The decoder constructs a first merge candidate list for the current block using the spatial merge candidates and temporal merge candidates for the current block (S4701).

[0472] The decoder constructs a second merge candidate list by adding a history-based merge candidate, which indicates the motion information of previously coded blocks of the current block, to the first merge candidate list (S4702).

[0473] The decoder obtains a merge index in the second merge candidate list that indicates the merge candidate to be applied to the current block (S4703).

[0474] The decoder generates a predicted block for the current block using the motion information of the merge candidate indicated by the merge index (S4704).

[0475] As described above, the history-based merge candidates can be added to the first merge candidate list if they have motion information that does not overlap with predefined merge candidates, selected from among the merge candidates included in the first merge candidate list.

[0476] As described above, the history-based merge candidates may be added to the first merge candidate list if they have motion information that does not overlap with a predetermined number of merge candidates included in the first merge candidate list.

[0477] As described above, if the history-based merge candidate has motion information that does not overlap with a specific spatial merge candidate included in the first merge candidate list, it may be added to the first merge candidate list.

[0478] As mentioned above, the history-based merge candidates can be derived from a predefined number of candidates within the history-based buffer that stores the history-based merge candidates. For example, the predefined number could be two.

[0479] As described above, the step of constructing the second merge candidate list further includes the step of adding history-based space-time merge candidates, the history-based space-time candidates can be derived using two of the history-based merge candidates and the time-time merge candidate.

[0480] As described above, the motion vector of the spatial-time candidate of the history base can be derived from the average of the motion vectors of the two merge candidate history bases and the average of the motion vector of the time-merge candidate.

[0481] Figure 48 shows an example block diagram of an apparatus for processing video signals according to an embodiment of this specification. The video signal processing apparatus in Figure 48 can correspond to the encoding apparatus 100 in Figure 1 or the decoding apparatus 200 in Figure 2.

[0482] The video processing device 4800, which processes video signals, includes a memory 4820 for storing video signals and a processor 4810 that processes video signals while being coupled with the memory.

[0483] The processor 4810 according to the embodiments of this specification may consist of at least one processing circuit for processing video signals and can process video signals by executing commands for encoding or decoding video signals. In other words, the processor 4810 can encode original video data or decode encoded video signals by executing the encoding or decoding methods described above.

[0484] Figure 49 shows a video coding system to which the present invention is applied.

[0485] A video coding system may include a source device and a receiving device. The source device can transmit encoded video / image information or data to the receiving device in file or streaming form via a digital storage medium or network.

[0486] The source device may include a video source, an encoding apparatus, and a transmitter. The receiving device may include a receiver, a decoding apparatus, and a renderer. The encoding apparatus may be called a video / image encoding apparatus, and the decoding apparatus may be called a video / image decoding apparatus. The transmitter may be included in the encoding apparatus. The receiver may be included in the decoding apparatus. The renderer may also include a display unit, which may consist of a separate device or external component.

[0487] A video source can acquire video / images through processes such as video / image capture, synthesis, or generation. A video source can include video / image capture devices and / or video / image generation devices. Video / image capture devices can include, for example, one or more cameras, or video / image archives containing previously captured video / images. Video / image generation devices can include, for example, computers, tablets, and smartphones, and can generate video / images (electronically). For example, virtual video / images can be generated through a computer, in which case the video / image capture process can be replaced by the process of generating the associated data.

[0488] An encoding device can encode input video / image data. It can perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0489] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or network in file or streaming form. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit may include elements for generating media files via a predetermined file format and may include elements for transmission via a broadcast / communication network. The receiving unit can extract the bitstream and transmit it to a decoding device.

[0490] A decoding device can decode video / images by performing a series of steps, such as inverse quantization, inverse transformation, and prediction, corresponding to the operation of an encoding device.

[0491] The renderer can render the decoded video / image. The rendered video / image can be displayed via the display unit.

[0492] Figure 50 shows a diagram of a content streaming system structure as an example to which the present invention is applied.

[0493] Referring to Figure 50, the content streaming system to which the present invention applies can broadly include an encoding server, a streaming server, a web server, media storage, a user device, and a multimedia input device.

[0494] The encoding server is responsible for compressing content input from multimedia input devices such as smartphones, cameras, and video cameras into digital data to generate a bitstream, and transmitting this bitstream to the streaming server. In other cases, if a multimedia input device such as a smartphone, camera, or video camera directly generates the bitstream, the encoding server can be omitted.

[0495] The bitstream can be generated by an encoding method or a bitstream generation method to which the present invention is applied, and the streaming server can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0496] The streaming server transmits multimedia data to the user's device via a web server based on the user's request, and the web server acts as an intermediary to inform the user of what services are available. When the user requests a desired service from the web server, the web server transmits this to the streaming server, and the streaming server transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, in which case the control server controls the commands / responses between each device in the content streaming system.

[0497] The streaming server can receive content from media storage and / or encoding servers. For example, if it receives content from the encoding server, it can receive the content in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0498] Examples of user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (such as smartwatches, smart glasses, and HMDs), digital TVs, desktop computers, and digital signage.

[0499] Each server in the aforementioned content streaming system can be operated as a distributed server, in which case the data received by each server can be processed in a distributed manner.

[0500] As described above, the embodiments described in the present invention can be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in each drawing can be implemented and executed on a computer, processor, microprocessor, controller, or chip.

[0501] Furthermore, decoders and encoders to which the present invention applies can include multimedia broadcasting transceivers, mobile communication terminals, home cinema video equipment, digital cinema video equipment, surveillance cameras, video conferencing equipment, real-time communication equipment such as video communication, mobile streaming equipment, storage media, video cameras, video-on-demand (VoD) service providers, OTT video (Over the top video) equipment, internet streaming service providers, 3D video equipment, image-phone video equipment, and medical video equipment, and are used to process video signals or data signals. For example, OTT video (Over the top video) equipment can include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, DVRs (Digital Video Recorders), etc.

[0502] Furthermore, the processing methods to which the present invention is applied can be produced in the form of programs executed on a computer and stored on a computer-readable recording medium. Multimedia data having the data structure according to the present invention can also be stored on a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices on which computer-readable data is stored. The computer-readable recording medium can include, for example, Blu-ray discs (BDs), Universal Serial Bus (USB), ROMs, PROMs, EPROMs, EEPROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer-readable recording medium also includes media implemented in the form of carrier waves (e.g., transmission over the Internet). Furthermore, bitstreams generated by encoding methods can be stored on computer-readable recording media or transmitted over wireless networks.

[0503] Furthermore, embodiments of the present invention can be realized as computer program products using program code, and such program code can be executed by a computer according to embodiments of the present invention. The program code can be stored on a computer-readable carrier.

[0504] The embodiments described above are combinations of the components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless otherwise explicitly mentioned. Each component or feature can be implemented in a form that is not combined with other components or features. It is also possible to combine some components and / or features to constitute embodiments of the present invention. The order of operations described in the embodiments of the present invention can be changed. Some components or features of any embodiment can be included in other embodiments, or can be replaced with components or features corresponding to other embodiments. It is obvious that claims that are not explicitly referenced in the claims can be combined to constitute embodiments, or can be included as new claims by amendments after filing.

[0505] Embodiments of the present invention can be implemented by various means, such as hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention can be implemented by one or more ASICs (application-specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, etc.

[0506] In the case of implementation by firmware or software, one embodiment of the present invention can be implemented in the form of a module, procedure, function, etc., that performs the functions or operations described above. The software code can be stored in memory and driven by a processor. The memory can be located inside or outside the processor and can send and receive data with the processor by various already known means.

[0507] It will be obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential features of the invention. Therefore, the above-mentioned detailed description should not be constrained in any way, but should be considered illustrative. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention. [Industrial applicability]

[0508] The preferred embodiments of the present invention described above are disclosed for illustrative purposes only, and those skilled in the art can improve, modify, substitute, or add various other embodiments within the technical concept and technical scope of the present invention disclosed in the appended claims.

Claims

1. A step of generating a first merge candidate list for the current block based on a plurality of space candidates, including a space candidate around the lower left corner, a space candidate around the left side, a space candidate around the upper right corner, a space candidate around the upper side, and a space candidate around the upper left corner, and a time candidate for the current block, wherein the plurality of space candidates precede the time candidate in the first merge candidate list. The step of generating a second merge candidate list by adding history-based candidates, which include motion information for blocks decoded before the current block, to the first merge candidate list, The steps include: purging the merge index that indicates the merge candidate to be applied to the current block within the second merge candidate list; A step of generating a predicted block for the current block based on the motion information of the merge candidate indicated by the merge index, A step of generating a current block of the current block from a bitstream of the video signal, The step of generating a restored block of the current block based on the predicted block and the residual block, The step of generating the second merge candidate list is, The steps include checking the motion information of the history-based candidate with the motion information of the left peripheral space candidate and the upper peripheral space candidate included in the first merge candidate list, The process includes, only if the checking step indicates that the history-based candidate has motion information that does not overlap with the motion information of the left peripheral space candidate and the upper peripheral space candidate included in the first merge candidate list, adding the history-based candidate, The aforementioned checking step is not performed on the candidate space around the lower left corner, the candidate space around the upper right corner, and the candidate space around the upper left corner, which are included in the first merge candidate list. The method wherein the checking step is performed between a predefined number of history-based candidates included in the history-based merge candidate list and only the left-side peripheral space candidates and the upper-side peripheral space candidates.

2. A step of generating a first merge candidate list for the current block based on a plurality of space candidates, including a space candidate around the lower left corner, a space candidate around the left side, a space candidate around the upper right corner, a space candidate around the upper side, and a space candidate around the upper left corner, and a time candidate for the current block, wherein the plurality of space candidates precede the time candidate in the first merge candidate list. The step of generating a second merge candidate list by adding history-based candidates, which include motion information for blocks decoded before the current block, to the first merge candidate list, A step of generating a merge index that indicates the merge candidate to be applied to the current block in the second merge candidate list, A step of generating a predicted block for the current block based on the motion information of the merge candidate indicated by the merge index, The step of generating a current block based on the predicted block includes, The step of generating the second merge candidate list is, The steps include checking the motion information of the history-based candidate with the motion information of the left peripheral space candidate and the upper peripheral space candidate included in the first merge candidate list, The process includes, only if the checking step indicates that the history-based candidate has motion information that does not overlap with the motion information of the left peripheral space candidate and the upper peripheral space candidate included in the first merge candidate list, adding the history-based candidate, The aforementioned checking step is not performed on the candidate space around the lower left corner, the candidate space around the upper right corner, and the candidate space around the upper left corner, which are included in the first merge candidate list. The method wherein the checking step is performed between a predefined number of history-based candidates included in the history-based merge candidate list and only the left-side peripheral space candidates and the upper-side peripheral space candidates.

3. The step of obtaining a bitstream related to the image, The step of transmitting data including the bitstream, The aforementioned bitstream is A step of generating a first merge candidate list for the current block based on a plurality of space candidates, including a space candidate around the lower left corner, a space candidate around the left side, a space candidate around the upper right corner, a space candidate around the upper side, and a space candidate around the upper left corner, and a time candidate for the current block, wherein the plurality of space candidates precede the time candidate in the first merge candidate list. The step of generating a second merge candidate list by adding history-based candidates, which include motion information for blocks decoded before the current block, to the first merge candidate list, A step of generating a merge index that indicates the merge candidate to be applied to the current block in the second merge candidate list, A step of generating a predicted block for the current block based on the motion information of the merge candidate indicated by the merge index, The steps of generating a current block based on the predicted block and generating the current block are performed, The step of generating the second merge candidate list is, The steps include checking the motion information of the history-based candidate with the motion information of the left peripheral space candidate and the upper peripheral space candidate included in the first merge candidate list, The process includes, only if the checking step indicates that the history-based candidate has motion information that does not overlap with the motion information of the left peripheral space candidate and the upper peripheral space candidate included in the first merge candidate list, adding the history-based candidate, The aforementioned checking step is not performed on the candidate space around the lower left corner, the candidate space around the upper right corner, and the candidate space around the upper left corner, which are included in the first merge candidate list. The aforementioned checking step is performed between a predefined number of history-based candidates included in the history-based merge candidate list and only the left-side peripheral space candidates and the upper-side peripheral space candidates, in a transmission method.

Images