METHOD AND APPARATUS FOR VIDEO DECODING BY INTER PREDICTION IN VIDEO CODING SYSTEM
The method and apparatus for video decoding using inter prediction and bi-directional optical-flow prediction in sub-block units address the inefficiencies of high-resolution video coding, reducing computational complexity and enhancing coding efficiency.
Patent Information
- Application Number
- JP2023115243
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2017-12-14
- Filing Date
- 2023-07-13
- Publication Date
- 2024-12-09
- Estimated Expiration
- 2038-12-13
AI Technical Summary
The increasing demand for high-resolution, high-quality video data leads to higher transmission and storage costs due to increased information bits, necessitating more efficient video coding techniques.
A method and apparatus for video decoding using inter prediction, including deriving refined motion vectors in sub-block units through bi-directional optical-flow prediction, and determining the applicability of BIO prediction to reduce computational complexity.
This approach enhances video coding efficiency by reducing computational complexity and improving overall coding efficiency through refined motion vector derivation in sub-block units.
Smart Images

Figure 0007596457000069 
Figure 0007596457000070 
Figure 0007596457000071
Abstract
Description
[Technical field]
[0001] The present invention relates to a video coding technology, and more particularly, to a method and apparatus for video decoding using inter prediction in a video coding system. [Background technology]
[0002] Recently, the demand for high-resolution, high-quality images such as HD (High Definition) images and UHD (Ultra High Definition) images is increasing in various fields. As the resolution and quality of image data increases, the amount of information or bits to be transmitted increases relatively compared to existing image data, so that when the image data is transmitted using a medium such as an existing wired or wireless broadband line or when the image data is stored using an existing storage medium, the transmission cost and storage cost increase.
[0003] This requires highly efficient video compression techniques to effectively transmit, store, and play back high-resolution, high-quality video information. Summary of the Invention [Problem to be solved by the invention]
[0004] An object of the present invention is to provide a method and apparatus for improving video coding efficiency.
[0005] An object of the present invention is to provide a method and apparatus for improving inter prediction efficiency.
[0006] Another technical object of the present invention is to provide a method and apparatus for deriving a refined motion vector on a sub-block basis by applying BIO prediction.
[0007] Another technical object of the present invention is to provide a method and apparatus for reducing the computational complexity of deriving a refined motion vector by determining whether BIO prediction is applicable. [Means for solving the problem]
[0008] According to an embodiment of the present invention, there is provided a video decoding method performed by a decoding device, the method including the steps of: deriving a reference picture list 0 (L0) and a reference picture list 1 (L1), deriving two motion vectors (MVs) for a current block, the two MVs including MVL0 for the L0 and MVL1 for the L1, determining whether a bi-directional optical-flow (BIO) prediction for deriving a refined motion vector in sub-block units is applied to the current block, deriving the refined motion vector for a sub-block of the current block based on the MVL0 and the MVL1 if the BIO prediction is applied to the current block, and deriving a prediction sample based on the refined motion vector.
[0009] According to another embodiment of the present invention, there is provided a decoding device for performing video decoding, comprising: an entropy decoding unit for acquiring information on inter prediction of a current block through a bitstream; and a prediction unit for deriving a reference picture list 0 (L0) and a reference picture list 1 (L1), deriving two motion vectors (MVs) for the current block, the two MVs including an MVL0 for the L0 and an MVL1 for the L1, determining whether a bi-prediction optical-flow (BIO) prediction for deriving a refined motion vector in sub-block units is applied to the current block, deriving the refined motion vector for a sub-block of the current block based on the MVL0 and the MVL1 if the BIO prediction is applied to the current block, and deriving a prediction sample based on the refined motion vector.
[0010] According to another embodiment of the present invention, there is provided a video encoding method performed by an encoding apparatus, the method including the steps of: deriving a reference picture list 0 (L0) and a reference picture list 1 (L1), deriving two motion vectors (MVs) for a current block, the two MVs including an MVL0 for the L0 and an MVL1 for the L1, determining whether a bi-prediction optical-flow (BIO) prediction for deriving a refined motion vector in sub-block units is applied to the current block, deriving the refined motion vector for a sub-block of the current block based on the MVL0 and the MVL1 if the BIO prediction is applied to the current block, deriving a prediction sample based on the refined motion vector, and entropy encoding information for inter-prediction of the current block.
[0011] According to another embodiment of the present invention, there is provided a video encoding apparatus, comprising: a prediction unit that derives a reference picture list 0 (L0) and a reference picture list 1 (L1), derives two motion vectors (MVs) for a current block, the two MVs including an MVL0 for the L0 and an MVL1 for the L1, determines whether a bi-prediction optical-flow (BIO) prediction for deriving a refined motion vector in sub-block units is applied to the current block, derives the refined motion vector for a sub-block of the current block based on the MVL0 and the MVL1 if the BIO prediction is applied to the current block, and derives a prediction sample based on the refined motion vector, and an entropy encoding unit that entropy encodes information for inter prediction of the current block. Effect of the Invention
[0012] According to the present invention, by determining whether BIO prediction is applicable to the current block, the computational complexity of inter prediction using refined motion vectors derived on a sub-block basis can be reduced, thereby improving overall coding efficiency.
[0013] According to the present invention, by determining whether to apply BIO prediction based on the FRUC mode, the computational complexity of inter prediction using refined motion vectors derived on a sub-block basis can be reduced, thereby improving overall coding efficiency. [Brief description of the drawings]
[0014] [Figure 1] 1 is a diagram illustrating a schematic configuration of a video encoding device to which the present invention can be applied. [Diagram 2] 1 is a diagram illustrating a schematic configuration of a video decoding device to which the present invention can be applied. [Diagram 3] 13 shows an example of a bi-predictive motion vector when it is assumed that the BCC and an object move at a constant speed in a short period of time. [Figure 4] 1 shows an example in which sample-based BIO prediction and block-based BIO prediction are performed. [Diagram 5] An example of determining whether to apply the BIO prediction based on the size of the current block will be described below. [Figure 6] 13 shows an example of deriving motion information via the BM method. [Figure 7] An example of deriving motion information via the TM method is illustratively shown. [Figure 8a] 1 shows an example of an encoding process and a decoding process performed based on FRUC. [Figure 8b] 1 shows an example of an encoding process and a decoding process performed based on FRUC. [Figure 9a]An example of encoding / decoding a current block to which FRUC and / or BIO is applied will be described. [Figure 9b] An example of encoding / decoding a current block to which FRUC and / or BIO is applied will be described. [Figure 10] 2 illustrates a video encoding method using an encoding apparatus according to the present invention. [Figure 11] 1 illustrates a schematic diagram of an encoding device for performing a video encoding method according to the present invention. [Figure 12] 1 illustrates a video decoding method using a decoding apparatus according to the present invention. [Figure 13] 1 illustrates a schematic diagram of a decoding device for performing a video decoding method according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] The present invention can be modified in various ways and can have various embodiments, and a specific embodiment will be illustrated in the drawings and described in detail. However, this does not limit the present invention to the specific embodiment. The terms used in this specification are used only to describe a specific embodiment and are not intended to limit the technical idea of the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise. In this specification, the terms "include" or "have" specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.
[0016] Meanwhile, each component in the drawings described in the present invention is illustrated independently for the convenience of explaining the different characteristic functions, and does not mean that each component is realized by separate hardware or software. For example, two or more components among each component may be combined to form one component, or one component may be divided into multiple components. An embodiment in which each component is integrated and / or separated is also included in the scope of the present invention as long as it does not deviate from the essence of the present invention.
[0017] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following, the same components in the drawings will be designated by the same reference numerals, and duplicated explanations of the same components will be omitted.
[0018] Meanwhile, the present invention relates to video / image coding, for example, the methods / embodiments disclosed in the present invention can be applied to methods disclosed in the versatile video coding (VVC) standard or next generation video / image coding.
[0019] In this specification, a picture generally means a unit indicating one image in a specific time period, and a slice is a unit constituting a part of a picture in coding. One picture may be composed of multiple slices, and pictures and slices may be used interchangeably as necessary.
[0020] A pixel or pel may refer to the smallest unit constituting one picture (or image). Also, a term corresponding to a pixel may be "sample." A sample generally indicates a pixel or a pixel value, and may indicate only a pixel / pixel value of a luminance (luma) component, or may indicate only a pixel / pixel value of a chroma component.
[0021] A unit refers to a basic unit of image processing. A unit may include at least one of a specific region of a picture and information about the corresponding region. A unit may be used in combination with terms such as a block or an area depending on the case. In a general case, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows.
[0022] FIG. 1 is a diagram illustrating a schematic configuration of a video encoding device to which the present invention can be applied.
[0023] 1, the video encoding apparatus 100 may include a picture division unit 105, a prediction unit 110, a residual processing unit 120, an entropy encoding unit 130, an adder 140, a filter unit 150, and a memory 160. The residual processing unit 120 may include a subtraction unit 121, a transform unit 122, a quantization unit 123, a realignment unit 124, an inverse quantization unit 125, and an inverse transform unit 126.
[0024] The picture division unit 105 can divide an input picture into at least one processing unit.
[0025] For example, the processing unit is called a coding unit (CU). In this case, the coding unit may be recursively divided from the largest coding unit (LCU) according to a quad-tree binary-tree (QTBT) structure. For example, one coding unit may be divided into a plurality of coding units of a deeper depth based on a quad-tree structure and / or a binary tree structure. In this case, for example, the quad-tree structure may be applied first, and then the binary tree structure may be applied. Alternatively, the binary tree structure may be applied first. A coding procedure according to the present invention may be performed based on the final coding unit that is not further divided. In this case, the largest coding unit may be used as the final coding unit based on coding efficiency according to image characteristics, or the coding unit may be recursively divided into coding units of a lower depth as necessary, and a coding unit of an optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later.
[0026] As another example, the processing unit may include a coding unit (CU), a prediction unit (PU), or a transform unit (TU). The coding unit may be split into coding units of deeper depths from a largest coding unit (LCU) according to a quad tree structure. In this case, the largest coding unit may be used as the final coding unit based on coding efficiency according to image characteristics, or the coding unit may be recursively split into coding units of lower depths as necessary, and a coding unit of an optimal size may be used as the final coding unit. When a smallest coding unit (SCU) is set, the coding unit may not be split into coding units smaller than the smallest coding unit. Here, the final coding unit refers to a coding unit that is a basis for partitioning or splitting into prediction units or transform units. The prediction unit is a unit that is partitioned from the coding unit and is a unit of sample prediction. In this case, the prediction unit may be divided into subblocks. A transform unit may be divided from a coding unit according to a quad tree structure, and is a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients. Hereinafter, a coding unit is also called a coding block (CB), a prediction unit is also called a prediction block (PB), and a transform unit is also called a transform block (TB). A prediction block or a prediction unit refers to a specific region in a block form within a picture, and may include an array of prediction samples.Also, a transform block or transform unit refers to a specific region in a picture in the form of a block, and may include an array of transform coefficients or residual samples.
[0027] The prediction unit 110 may perform prediction on a current block to be processed and generate a predicted block including a prediction sample for the current block. The unit of prediction performed by the prediction unit 110 is a coding block, a transform block, or a prediction block.
[0028] The prediction unit 110 may determine whether intra prediction or inter prediction is applied to the current block. For example, the prediction unit 110 may determine whether intra prediction or inter prediction is applied to the current block on a CU basis.
[0029] In the case of intra prediction, the prediction unit 110 may derive a prediction sample for the current block based on a reference sample outside the current block in a picture to which the current block belongs (hereinafter, the current picture). In this case, the prediction unit 110 may (i) derive a prediction sample based on an average or an interpolation of neighboring reference samples of the current block, and (ii) derive the prediction sample based on a reference sample that exists in a specific (prediction) direction with respect to the prediction sample among the neighboring reference samples of the current block. The case (i) is called a non-directional mode or a non-angular mode, and the case (ii) is called a directional mode or an angular mode. Prediction modes in intra prediction may include, for example, 33 directional prediction modes and at least two or more non-directional modes. The non-directional modes may include a DC prediction mode and a planar mode. The prediction unit 110 may also determine a prediction mode to be applied to the current block using a prediction mode applied to a neighboring block.
[0030] In the case of inter prediction, the prediction unit 110 may derive a prediction sample for the current block based on a sample specified by a motion vector on a reference picture. The prediction unit 110 may derive a prediction sample for the current block by applying any one of a skip mode, a merge mode, and a motion vector prediction (MVP) mode. In the case of the skip mode and the merge mode, the prediction unit 110 may use motion information of a neighboring block as motion information of the current block. In the case of the skip mode, unlike the merge mode, a difference (residual) between a prediction sample and an original sample is not transmitted. In the case of the MVP mode, a motion vector of the current block may be derived by using a motion vector of a neighboring block as a motion vector predictor.
[0031] In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in a reference picture. The reference picture including the temporal neighboring blocks is also called a collocated picture (colPic). The motion information may include a motion vector and a reference picture index. Information such as prediction mode information and motion information may be (entropy) encoded and output in the form of a bitstream.
[0032] When motion information of a temporally neighboring block is used in skip mode and merge mode, the top picture on a reference picture list may be used as a reference picture. Reference pictures included in a reference picture list may be sorted based on a picture order count (POC) difference between the current picture and the corresponding reference picture. The POC corresponds to the display order of pictures and may be distinguished from the coding order.
[0033] The subtractor 121 generates a residual sample, which is the difference between the original sample and the predicted sample, and does not generate a residual sample when the skip mode is applied, as described above.
[0034] The transform unit 122 transforms the residual samples in units of transform blocks to generate transform coefficients. The transform unit 122 may perform the transform according to the size of the corresponding transform block and a prediction mode applied to a coding block or a prediction block that spatially overlaps with the corresponding transform block. For example, if intra prediction is applied to the coding block or the prediction block that overlaps with the transform block and the transform block is a 4x4 residual array, the residual samples may be transformed using a Discrete Sine Transform (DST) transform kernel, and otherwise, the residual samples may be transformed using a Discrete Cosine Transform (DCT) transform kernel.
[0035] The quantization unit 123 may quantize the transform coefficients to generate quantized transform coefficients.
[0036] The rearrangement unit 124 rearranges the quantized transform coefficients. The rearrangement unit 124 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form through a coefficient scanning method. Here, the rearrangement unit 124 has been described as a separate configuration, but may be a part of the quantization unit 123.
[0037] The entropy encoding unit 130 may perform entropy encoding on the quantized transform coefficients. The entropy encoding may include encoding methods such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit 130 may also encode information required for video restoration (e.g., syntax element values, etc.) together with or separately from the quantized transform coefficients by entropy encoding or a preset method. The encoded information may be transmitted or stored in network abstraction layer (NAL) unit units in the form of a bitstream.
[0038] The inverse quantization unit 125 inversely quantizes the values (quantized transformation coefficients) quantized by the quantization unit 123, and the inverse transform unit 126 inversely transforms the values inversely quantized by the inverse quantization unit 125 to generate residual samples.
[0039] The adder 140 reconstructs a picture by adding the residual sample and the prediction sample. The residual sample and the prediction sample may be added in block units to generate a reconstructed block. Although the adder 140 has been described as a separate configuration, it may be a part of the prediction unit 110. Meanwhile, the adder 140 may also be called a reconstruction module or a reconstructed block generator.
[0040] The filter unit 150 may apply a deblocking filter and / or a sample adaptive offset to the reconstructed picture. Through the deblocking filtering and / or the sample adaptive offset, artifacts at block boundaries in the reconstructed picture and distortion in the quantization process may be corrected. The sample adaptive offset may be applied on a sample basis and may be applied after the deblocking filtering process is completed. The filter unit 150 may also apply an adaptive loop filter (ALF) to the reconstructed picture. The ALF may be applied to the reconstructed picture after the deblocking filter and / or the sample adaptive offset are applied.
[0041] The memory 160 may store a reconstructed picture (a decoded picture) or information required for encoding / decoding. Here, a reconstructed picture is a reconstructed picture that has been filtered by the filter unit 150. The stored reconstructed picture may be used as a reference picture for (inter) prediction of another picture. For example, the memory 160 may store (reference) pictures used for inter prediction. In this case, the pictures used for inter prediction may be specified by a reference picture set or a reference picture list.
[0042] FIG. 2 is a diagram illustrating a schematic configuration of a video decoding device to which the present invention can be applied.
[0043] 2, the video decoding apparatus 200 may include an entropy decoding unit 210, a residual processing unit 220, a prediction unit 230, an adder 240, a filter unit 250, and a memory 260. Here, the residual processing unit 220 may include a reordering unit 221, an inverse quantization unit 222, and an inverse transform unit 223.
[0044] When a bitstream including video information is input, the video decoding apparatus 200 can restore the video corresponding to the process in which the video information was processed in the video encoding apparatus.
[0045] For example, the video decoding apparatus 200 may perform video decoding using a processing unit applied in a video encoding apparatus. Thus, a processing unit block of the video decoding may be a coding unit as an example, and may be a coding unit, a prediction unit, or a transform unit as another example. The coding unit may be divided from the maximum coding unit according to a quad tree structure and / or a binary tree structure.
[0046] A prediction unit and a transform unit may also be used in some cases, where the prediction block is a block derived or partitioned from the coding unit and is a unit of sample prediction. In this case, the prediction unit may be divided into sub-blocks. The transform unit may be divided from the coding unit by a quadtree structure and is a unit that derives transform coefficients or a unit that derives a residual signal from the transform coefficients.
[0047] The entropy decoding unit 210 may parse the bitstream and output information required for video reconstruction or picture reconstruction. For example, the entropy decoding unit 210 may decode information in the bitstream based on a coding method such as Exponential Golomb Coding, CAVLC, or CABAC, and output values of syntax elements required for video reconstruction and quantized values of transform coefficients for residuals.
[0048] More specifically, the CABAC entropy decoding method receives BINs corresponding to each syntax element in a bitstream, determines a context model using information on the syntax element to be decoded and decoding information on adjacent and blocks to be decoded or information on symbols / BINs decoded in a previous step, predicts the occurrence probability of BINs according to the determined context model, and performs arithmetic decoding of BINs to generate symbols corresponding to the values of each syntax element. In this case, the CABAC entropy decoding method can update the context model using information on the decoded symbols / BINs for the context model of the next symbol / BIN after determining the context model.
[0049] Among the information decoded by the entropy decoding unit 210, information regarding prediction is provided to the prediction unit 230, and the residual values on which entropy decoding is performed by the entropy decoding unit 210, i.e., the quantized transform coefficients, can be input to the reordering unit 221.
[0050] The rearrangement unit 221 may rearrange the quantized transform coefficients in a two-dimensional block form. The rearrangement unit 221 may perform rearrangement in response to coefficient scanning performed in the encoding apparatus. Here, the rearrangement unit 221 may be a part of the inverse quantization unit 222, although it has been described as a separate configuration.
[0051] The inverse quantization unit 222 may inverse quantize the quantized transform coefficients based on the (inverse) quantization parameter to output the transform coefficients. At this time, information for deriving the quantization parameter may be signaled from an encoding device.
[0052] The inverse transform unit 223 can inversely transform the transform coefficients to derive residual samples.
[0053] The prediction unit 230 may perform prediction on a current block and generate a predicted block including prediction samples for the current block. The unit of prediction performed by the prediction unit 230 may be a coding block, a transform block, or a prediction block.
[0054] The prediction unit 230 may determine whether to apply intra prediction or inter prediction based on the information on the prediction. At this time, the unit for determining whether to apply intra prediction or inter prediction is different from the unit for generating a prediction sample. In addition, the unit for generating a prediction sample is also different in inter prediction and intra prediction. For example, whether to apply inter prediction or intra prediction may be determined in units of CU. In addition, for example, in inter prediction, a prediction mode may be determined in units of PU to generate a prediction sample, and in intra prediction, a prediction mode may be determined in units of PU to generate a prediction sample in units of TU.
[0055] In the case of intra prediction, the prediction unit 230 may derive a prediction sample for the current block based on a neighboring reference sample in the current picture. The prediction unit 230 may derive a prediction sample for the current block by applying a directional mode or a non-directional mode based on the neighboring reference sample of the current block. In this case, a prediction mode to be applied to the current block may be determined using the intra prediction mode of the neighboring block.
[0056] In the case of inter prediction, the prediction unit 230 may derive a prediction sample for the current block based on a sample specified on the reference picture by a motion vector on the reference picture. The prediction unit 230 may derive a prediction sample for the current block by applying any one of a skip mode, a merge mode, and an MVP mode. In this case, motion information required for inter prediction of the current block provided by the video encoding apparatus, for example, information on a motion vector, a reference picture index, etc., may be obtained or induced based on information on the prediction.
[0057] In the case of skip mode and merge mode, motion information of neighboring blocks may be used as motion information of the current block, where the neighboring blocks may include spatial neighboring blocks and temporal neighboring blocks.
[0058] The prediction unit 230 may construct a merge candidate list using motion information of available neighboring blocks, and may use information indicated by a merge index on the merge candidate list as a motion vector of the current block. The merge index may be signaled from an encoding device. The motion information may include a motion vector and a reference picture. When motion information of a temporally neighboring block is used in skip mode and merge mode, the top picture on the reference picture list may be used as a reference picture.
[0059] In skip mode, unlike merge mode, the difference (residual) between the predicted sample and the original sample is not transmitted.
[0060] In the MVP mode, the motion vector of the current block can be derived by using the motion vector of the neighboring block as a motion vector predictor, where the neighboring block can include a spatial neighboring block and a temporal neighboring block.
[0061] For example, when a merge mode is applied, a merge candidate list may be generated using the motion vectors of the reconstructed spatial neighboring blocks and / or the motion vector corresponding to the Col block, which is a temporal neighboring block. In the merge mode, a motion vector of a candidate block selected from the merge candidate list is used as the motion vector of the current block. The prediction information may include a merge index indicating a candidate block having an optimal motion vector selected from among the candidate blocks included in the merge candidate list. In this case, the prediction unit 230 may derive a motion vector of the current block using the merge index.
[0062] As another example, when a Motion Vector Prediction (MVP) mode is applied, a motion vector predictor candidate list may be generated using the motion vector of the restored spatial neighboring block and / or the motion vector corresponding to the Col block, which is a temporal neighboring block. That is, the motion vector of the restored spatial neighboring block and / or the motion vector corresponding to the Col block, which is a temporal neighboring block, may be used as a motion vector candidate. The prediction information may include a predicted motion vector index indicating an optimal motion vector selected from the motion vector candidates included in the list. In this case, the prediction unit 230 may select a predicted motion vector of the current block from the motion vector candidates included in the motion vector candidate list using the motion vector index. A prediction unit of the encoding device may obtain a motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, encode the MVD, and output it in the form of a bitstream. That is, the MVD is obtained by subtracting the motion vector predictor from the motion vector of the current block. In this case, the prediction unit 230 may obtain a motion vector differential included in the information on the prediction, and derive the motion vector of the current block by adding the motion vector differential and the motion vector predictor. In addition, the prediction unit may obtain or induce a reference picture index indicating a reference picture from the information on the prediction.
[0063] The adder 240 may reconstruct a current block or a current picture by adding a residual sample and a prediction sample. The adder 240 may also reconstruct a current picture by adding a residual sample and a prediction sample in block units. When a skip mode is applied, the residual is not transmitted, so that the prediction sample may become a reconstructed sample. Here, the adder 240 has been described as a separate configuration, but may be a part of the prediction unit 230. Meanwhile, the adder 240 may also be referred to as a reconstruction module or a reconstructed block generation module.
[0064] The filter unit 250 may apply deblocking filtering, sample adaptive offset, and / or ALF to the reconstructed picture. In this case, the sample adaptive offset may be applied on a sample basis or may be applied after deblocking filtering. The ALF may be applied after deblocking filtering and / or sample adaptive offset.
[0065] The memory 260 may store a reconstructed picture (a decoded picture) or information required for decoding. Here, the reconstructed picture is a reconstructed picture for which a filtering procedure by the filter unit 250 has been completed. For example, the memory 260 may store a picture used in inter prediction. At this time, the picture used in inter prediction may be specified by a reference picture set or a reference picture list. The reconstructed picture may be used as a reference picture for another picture. In addition, the memory 260 may output the reconstructed picture according to an output order.
[0066] As described above, one or two reference picture lists may be used for inter prediction of the current block. That is, reference picture list 0 or reference picture list 1 may be used for inter prediction of the current block, or both reference picture list 0 and reference picture list 1 may be configured. For example, when the slice type of the slice including the current block is B (B slice), at least one of the two reference picture lists may be used, and when the slice type of the slice including the current block is P (P slice), only reference picture list 0 may be used. The reference picture list 0 is called L0 (List0), and the reference picture list 1 is called L1 (List1). When performing prediction for the current block, motion information for inter prediction is bi-prediction motion information or uni-prediction motion information. Motion information for the current block may be derived based on neighboring blocks of the current block. Here, the bi-prediction motion information may include an L0 reference picture index and an L0 motion vector (L0 motion information), an L1 reference picture index and an L1 motion vector (L1 motion information), and the uni-prediction motion information may include an L0 reference picture index and an L0 motion vector (L0 motion information), or an L1 reference picture index and an L1 motion vector (L1 motion information). The L0 indicates a reference picture list L0 (List0), and the L1 indicates a reference picture list L1 (List1). Inter prediction performed based on the L0 motion information is called LO prediction, inter prediction performed based on the L1 motion information is called L1 prediction, and inter prediction performed based on the L0 motion information and L1 motion information, i.e., bi-prediction motion information, is called bi-prediction. Meanwhile, separate motion vectors may be used for each of the L0 prediction and the L1 prediction.That is, for example, the L0 motion vector (motion vector L0, MVL0) for L0 prediction and the L1 motion vector (motion vector L1, MVL1) for L1 prediction for the current block may be derived separately. In this case, for example, if the MVL0 indicates a first reference area in an L0 reference picture in L0 and the MVL1 indicates a second reference area in an L1 reference picture in L1, a prediction sample of the current block may be derived through a weighted sum of a first predictor obtained from a reconstructed sample of the first reference area and a second predictor obtained from a reconstructed sample of the second reference area. Here, the weighted sum may be performed based on a first temporal distance between the current picture and the first reference picture and a second temporal distance between the current picture and the second reference picture. Here, the temporal distance may indicate a POC (Picture Order Count) difference. That is, the difference between the POC value of the current picture and the POC value of the L0 reference picture is the first temporal distance, and the difference between the POC value of the current picture and the POC value of the L1 reference picture is the second temporal distance.
[0067] Meanwhile, according to the present invention, when the bi-prediction is applied, a sample-based motion vector or a block-based motion vector can be obtained based on MVL0 and MVL1 for the current block and a first predictor and a second predictor, thereby deriving a prediction sample with improved prediction performance. This is called BIO (Bi-prediction Optical flow) prediction, and the prediction sample is called a refined prediction sample or a BIO prediction sample by distinguishing it from an existing prediction sample. Also, the sample-based motion vector or the block-based motion vector is called a refined motion vector or a BIO motion vector. When the block-based motion vector is derived, the size of a target block from which the block-based motion vector is derived may be preset to Bw×Bh. For example, the size of the target block may be preset to 4×4. In this case, the refined motion vector for a target block of 4×4 size included in the current block may be derived. Meanwhile, the target block of the current block may be referred to as a sub-block of the current block.
[0068] The refined motion vector and refined prediction samples can be derived in the following manner.
[0069] Assuming that there is no change in the sample (pixel) value of an object in consecutive frames due to the Brightness Constancy Constraint (BCC), the movement of an object due to optical flow (OF) can be expressed as the following equation.
[0070]
number
[0071] Here, I(x,y,t) is the sample value at the (x,y) sample position and time t, and Δ is the amount of change. Assuming small movements, the right-hand term of Equation 1 is a first-order Taylor series equation, which can be expressed as follows:
[0072]
number
[0073] Here, if we divide Equation 2 into t and rearrange it, it can be expressed as follows.
[0074]
number
[0075] Here, V x = Δx / Δt, and V y is Δy / Δt.
[0076] The above equation contains two unknowns: the motion and the spatial derivative of the signal. Therefore, the spatial derivative is needed for the motion analysis.
[0077] According to the present invention, it is possible to obtain a refined motion vector and a reference sample value in a sample (pixel) unit or a specific block unit without transmitting an additional motion vector by using an OF (optical flow) characteristic. For example, if it is assumed that an object moves at a constant speed in a short time in addition to the BCC, MVL0 and MVL1 can be expressed by symmetrical values of the same magnitude. That is, the magnitude of the x components of MVL0 and MVL1 are the same and the signs are different, and the magnitude of the y components of the MVL0 and MVL1 are the same and the signs are different. For example, if the x and y components of the MVL0 are Vx and Vy, respectively, the x and y components of the MVL1 are -Vx and -Vy, respectively.
[0078] FIG. 3 exemplarily illustrates a bi-predictive motion vector when it is assumed that the BCC and an object move at a constant speed in a short period of time.
[0079] Referring to FIG. 3, assuming that BCC and objects move at a constant speed in a short time, in bi-prediction for a current block in a current picture 300, MVL0 indicates a motion vector pointing to a first reference block in a first reference picture (Reference0) 310 corresponding to the current block, and MVL1 indicates a motion vector pointing to a second reference block in a second reference picture (Reference1) 320 corresponding to the current block. The first reference picture is one of the reference pictures included in the L0, and the second reference picture is one of the reference pictures included in the L1. In this case, the representative position of each block may be the top-left sample position of each block. In this case, the MVL0 and the MVL1 may be expressed as symmetrical values.
[0080] 3, a first reference sample value at a position 311 according to MVL0 and a second reference sample value at a position 321 according to MVL1 can be derived based on a target sample 301 in a current block. Here, the first reference sample value is called a first predictor or L0 predictor, and the second reference sample value is called a second predictor or L1 predictor. A difference between the first reference sample value and the second reference sample value can be expressed as the following equation.
[0081]
number
[0082] Where: TIFF0007596457000005.tif1038 indicates a sample value (i.e., a first reference sample value) at sample position 311 of a first reference picture (Reference0) 310, TIFF0007596457000006.tif1038 indicates a sample value (i.e., a second reference sample value) at sample position 321 of second reference picture (Reference1) 320. The sample value is represented by the following formula.
[0083]
number
[0084] Then, by substituting the above-mentioned Equation 5 into the above-mentioned Equation 4 and rearranging it, the following Equation 6 can be obtained.
[0085]
number
[0086] Here, I (0) [i,j] denotes the L0 reference sample value, and I (1) [i,j] denotes the L1 reference sample value, and I x (k) [i,j], I y (k) [i,j] indicate the x-axis and y-axis change, i.e., the gradient. TIFF0007596457000009.tif1124 and TIFF0007596457000010.tif1221 are x-axis and y-axis partial differential values at the position [i,j] of the first reference picture (Reference0) 310 in L0, TIFF0007596457000011.tif1321 and TIFF0007596457000012.tif1119 are x-axis and y-axis partial differential values at the position [i,j] of the second reference picture (Reference1) 320 in L1. The partial differential values are called gradients.
[0087] Meanwhile, in order to improve accuracy and prediction efficiency, the gradient can be expressed as the following Equation 7 based on an interpolation filter. Meanwhile, the position and / or unit to which the filter coefficients of the interpolation filter are assigned can be called a filter tap. The filter coefficients of the interpolation filter can be assigned in 1 / 4 fractional sample units.
[0088]
number
[0089] where 2M is the number of filter taps of the interpolation filter, TIFF0007596457000014.tif1115, TIFF0007596457000015.tif1214 indicates the decimal part of the x and y components of the motion vector, TIFF0007596457000016.tif1124, TIFF0007596457000017.tif1122 is TIFF0007596457000018.tif911, The filter coefficients for the n-th filter tap for TIFF0007596457000019.tif1213 are shown below. TIFF0007596457000020.tif1436 indicates a reconstructed sample value at the [i+n,j] position after bi-prediction, i.e., a reference sample value at the [i+n,j] position of a reference picture. The value of the fractional part of the x component and / or the y component of the motion vector is one of 0, 1 / 4, 2 / 4, and 3 / 4.
[0090] specifically, TIFF0007596457000021.tif1213, TIFF0007596457000022.tif1413 indicates the decimal parts of the x and y components of MVL0, respectively. TIFF0007596457000023.tif1124, TIFF0007596457000024.tif1325 is TIFF0007596457000025.tif1314, TIFF0007596457000026.tif1415 shows the filter coefficients for the nth filter tap, TIFF0007596457000027.tif1230 indicates a reference sample value at the [i+n,j] position of the first reference picture (Reference0) 310. TIFF0007596457000028.tif1313, TIFF0007596457000029.tif1214 indicates the decimal parts of the x and y components of MVL1, TIFF0007596457000030.tif1123, TIFF0007596457000031.tif1022 is TIFF0007596457000032.tif1313, TIFF0007596457000033.tif1215 shows the filter coefficients for the nth filter tap, TIFF0007596457000034.tif1125 indicates the reference sample value at position [i+n,j] of the second reference picture (Reference1) 320.
[0091] As an example, when the number of filter taps is 6 (ie, 2M is 6), the filter tap coefficients of the interpolation filter for the fractional parts of the x and y components of the motion vector may be defined as shown in the following table.
[0092] [Table 1]
[0093] Based on the above Equation 6 The sample-wise motion vector v that makes TIFF0007596457000036.tif1020 have the minimum value x [i,j], v y For example, we can assume that samples in a particular region, window Ω, centered at [i,j], have locally steady motion, where window Ω can contain (2M+1)×(2M+1) samples. In this case, the sample positions in window Ω are It can be expressed as TIFF0007596457000037.tif914. In this case, In TIFF0007596457000038.tif1013, iM≦i′≦i+M and jM≦j′≦j+M are satisfied. Based on this, TIFF0007596457000039.tif1427. In this case, given the assumption that the samples in the window Ω have locally unchanged motion, we can calculate the motion vector that minimizes TIFF0007596457000040.tif1117 is represented by the following formula.
[0094]
number
[0095] Where: TIFF0007596457000042.tif954, TIFF0007596457000043.tif1057, TIFF0007596457000044.tif1254. Here, P (0) [i′,j′] and P (1) [i', j'] indicate the L0 predictor and the L1 predictor, respectively. (0) [i′,j′] and the P (1) [i′,j′] are I 0 [i′,j′] and I 1 Corresponds to [i′,j′].
[0096] Formula 7 is V x , V y When each of these is partially differentiated and rearranged, we get the following.
[0097]
number
[0098] Where: TIFF0007596457000046.tif1128, TIFF0007596457000047.tif1036, TIFF0007596457000048.tif930, TIFF0007596457000049.tif1131, Assuming that the image is TIFF0007596457000050.tif1136, substituting s1, s2, s3, and s4 into the above-mentioned formula 9 and rearranging, the following formulas 10 and 11 can be obtained.
[0099]
number
[0100]
number
[0101] Vx and Vy can be rearranged based on Equations 10 and 11 as follows:
[0102]
number
[0103] That is, Vx and Vy can be respectively arranged as follows:
[0104]
number
[0105] Therefore, a refined predictor for a target sample, i.e., a refined predicted sample, can be calculated using Vx and Vy as follows:
[0106]
number
[0107] Based on the above method, a refined motion vector and a refined predicted sample for each sample can be obtained, where P represents a refined predicted sample for the target sample, and Vx and Vy represent the x and y components of the refined motion vector for the target sample, respectively.
[0108] Meanwhile, the above-mentioned method assumes that two motion vectors (i.e., MVL0 and MVL1) that are symmetrical and have the same magnitude are constant motion. That is, the above-mentioned refined predicted sample derivation method assumes that a first temporal distance between a current picture and a first reference picture associated with the MVL0 is the same as a second temporal distance between the current picture and a second reference picture associated with the MVL1. The difference between the POC value of the current picture and the POC value of the first reference picture is the first temporal distance, and the difference between the POC value of the second reference picture and the POC value of the second reference picture is the second temporal distance.
[0109] A method of deriving the refined predicted sample by considering the two motion vectors having a certain motion when the first temporal distance and the second temporal distance are not equal will be described below.
[0110] For example, when the first time distance and the second time distance have different values, Equation 14 can be recalculated as follows:
[0111]
number
[0112] where P denotes a refined predictor for the target sample, P(0) and P(1) denote the first and second predictors for the current sample, respectively, and I x (0) , I y (0) indicate the x-axis and y-axis gradients in the first predictor, respectively, and I x (1) , I y (1) denote the x-axis and y-axis gradients in the second predictor, respectively, Vx and Vy denote the x-component and y-component of the refinement motion vector for the target sample, respectively, and τ 0 denotes the first time distance, and τ 1 indicates the second temporal distance. The first temporal distance and the second temporal distance may indicate a distance between the current picture (or the current frame) and a reference picture (or a reference frame).
[0113] The first time distance and the second time distance can be derived based on the following formula:
[0114]
number
[0115] Here, POC(current) indicates a POC (picture order count) value of the current picture, POC(Ref0) indicates a POC value of the first reference picture, and POC(Ref1) indicates a POC value of the second reference picture.
[0116] Meanwhile, in order to reduce the computational complexity in the method of deriving the refined predicted sample, an approximation of Equation 13 can be used. The approximation of Equation 13 can be derived as follows.
[0117] The numerator and denominator of Vx in Equation 13 can be divided into s5 to derive the following equation.
[0118]
number
[0119] When the value of s5 in Equation 17 is sufficiently large, s2*s6 / s5 in the denominator and s2*s4 / s5 in the numerator of Equation 17 can be approximated to 0. Therefore, Equation 17 can be expressed as follows.
[0120]
number
[0121] When Vx derived through Equation 18 is substituted into Vy of Equation 11, Vy can be derived based on the following equation.
[0122]
number
[0123] Vy can be rearranged based on Equation 19 as follows:
[0124]
number
[0125] The computational complexity for deriving Vx and Vy based on Equations 18 and 20 can be reduced, thereby reducing the overall computational complexity of the method for deriving refined prediction samples.
[0126] Meanwhile, as described above, although BIO prediction for deriving a refined motion vector in a sample unit may be applied, BIO prediction for deriving a refined motion vector in a block unit may be proposed. BIO prediction for performing refinement in a block unit may reduce computational complexity compared to BIO prediction for performing refinement in a sample unit.
[0127] For example, if the width Bw and height Bh of the target block on which the refinement is performed are preset to 4, the predictor may be refined in units of blocks of 4×4 size. In the BIO prediction that performs refinement in units of samples, a gradient map may be generated for each sample as described above, an offset for each sample may be derived, and the refined predicted sample may be derived by adding the offset to the predictor for each sample according to Equation 14. However, in the BIO prediction that performs refinement in units of blocks (which may also be referred to as block based BIO prediction), an offset for a target block of 4×4 size may be derived, and the same offset may be added to the predictor of a target sample included in the target block. That is, the same refined motion vector may be applied to the target block.
[0128] FIG. 4 shows an example in which sample-based BIO prediction and block-based BIO prediction are performed. Referring to FIG. 4, when the size of the target block on which the refinement is performed is preset to Bw×Bh (for example, 4×4), x-axis, y-axis, and time-axis gradient maps of reference sample values of the target block of size w×h can be derived based on MVL0 and MVL1 derived through the encoding / decoding process. Then, an offset for a target sample included in the target block can be derived based on the gradient map for the target block. An offset for the target block can be derived based on a Bw×Bh kernel, and a refined prediction sample for the target sample can be derived by adding the offset to a predictor for the target sample.
[0129] Specifically, the block-based refine motion vector may be derived based on the gradient of a target sample included in a target block of the current block (i.e., a gradient map of the target block). The block-based refine motion vector may be derived in units of the target block of the current block, and the block-based refine motion vector may be expressed as a refine motion vector for the target block of the current block. For example, s of a target sample included in the target block of the current block may be derived based on the gradient of the target sample included in the target block of the current block. n The sum of the s n The s n s n、bk For example, s 1、bk s 2、bk s 3、bk , s 4、bk s 5、bk s 6、bk can be derived as follows:
[0130]
number
[0131] Where: TIFF0007596457000063.tif1164, TIFF0007596457000064.tif1267, The file is TIFF0007596457000065.tif1168.
[0132] Hereinafter, in the above-mentioned Equation 13, Equation 18, and Equation 20, s n Instead of (s n、bk For example, if the size of the target block from which the block-based motion vector is derived is 4×4, the refined motion vector for the target block may be derived by inserting s into Equation 13. n Instead of (s n、bk >>16) may be inserted into the equation to derive a refinement motion vector for the current block. If the current block has a size of 4×4, the block-based refinement motion vector may be derived based on the following equation:
[0133]
number
[0134] Alternatively, the block-based refined motion vector may be derived based on an approximated equation. n Instead of (s n、bk >>n) may be inserted into the refinement motion vector for the current block. If the current block has a size of 4×4, the block-based refinement motion vector may be derived based on the following formula:
[0135]
number
[0136] Hereafter, V in the above-mentioned Equation 14 x V x、bk Change to V y V y、bk Instead, a refined prediction sample for the target sample can be derived.
[0137] Meanwhile, the present invention proposes a method for adaptively applying the BIO prediction based on a specific condition for a current block, thereby optimizing the BIO prediction, particularly inter-prediction using block-based BIO prediction, to improve coding efficiency.
[0138] For example, whether or not to apply the BIO prediction may be determined based on the size of the current block, For example, if the size of the current block is W×H, whether or not to apply the prediction may be determined based on the size of the current block and a preset threshold.
[0139] For example, if the width of the current block is equal to the width threshold and the height of the current block is equal to the height threshold, it may be determined that the BIO prediction is not applied to the current block. Alternatively, if the size W×H of the current block is smaller than a preset minimum size N×M, it may be determined that the BIO prediction is not applied to the current block. Alternatively, if the number of samples of the current block is smaller than a preset threshold, it may be determined that the BIO prediction is not applied to the current block.
[0140] Specifically, for example, if the size of the current block is 4×4, the BIO prediction is not applied to the current block.
[0141] FIG. 5 illustrates an example of determining whether to apply the BIO prediction based on the size of the current block.
[0142] The encoding / decoding apparatus may derive a motion vector for the current block (S500). The motion vector may include MVL0 and MVL1.
[0143] The encoding / decoding apparatus may determine whether the size of the current block is 4×4 (S510). If the size of the current block is 4×4, the BIO prediction for the current block is not performed. In this case, inter prediction for the current block may be performed based on the derived motion vector.
[0144] If the size of the current block is not 4×4, the encoding / decoding apparatus may perform block-based BIO prediction on the current block (S520). In this case, the size of the target block from which the block-based motion vector is derived is 4×4. That is, the encoding / decoding apparatus may derive a refinement motion vector for a target block of size 4×4 included in the current block, and may derive refinement prediction samples for target samples included in the target block of the current block based on the refinement motion vector.
[0145] Meanwhile, in the process of deriving the motion information of the current block, Frame Rate Up-Conversion (FRUC) may be applied, and the above-mentioned BIO prediction may be performed based on the motion information derived through the FRUC. The FRUC is also called Pattern Matched Motion Vector Derivation (PMMVD). The FRUC assumes that an object moves at a constant speed in an image and that pixel values (sample values) do not change at this time, and may indicate a method of deriving motion information through a template matching (TM) method or a bilateral matching (BM) method. Specifically, it may be described as follows.
[0146] 6 shows an example of deriving motion information through the BM method. When the FRUC is applied to the current block, MVL0 and MVL1 for the current block are selected one by one, and then a motion vector corresponding to the motion vector in a direction different from the direction associated with the motion vector, among the L0 direction and the L1 direction, can be derived by performing interpolation around the picture to which the current block belongs.
[0147] Specifically, the encoding / decoding apparatus may derive a motion information candidate list for the current block based on neighboring blocks of the current block. The motion vector candidates for the neighboring blocks are bi-predictive motion vector candidates or uni-predictive motion vector candidates. The encoding / decoding apparatus may classify the motion vectors included in the motion vector candidates into L0 direction and L1 direction. That is, the encoding / decoding apparatus may classify the motion vectors included in the motion vector candidates into MVL0 and MVL1.
[0148] Then, the divided motion vectors are sequentially selected one by one, and then interpolation is performed based on the picture to which the current block belongs to, to derive a motion vector corresponding to the motion vector in a direction different from the direction associated with the motion vector, either the L0 direction or the L1 direction.
[0149] For example, when a motion vector candidate of the current block is MVL0, the MVL1 corresponding to the MVL0 may be derived by scaling based on a temporal distance between an L0 reference picture for the MVL0 and a current picture including the current block and a temporal distance between an L1 reference picture and the current picture. That is, the MVL0 may be scaled based on a first temporal distance between an L0 reference picture for the MVL0 and the current picture and a second temporal distance between the L1 reference picture and the current picture. Here, the first temporal distance is a difference value between a POC of the L0 reference picture and a POC of the current picture, and the second temporal distance is a difference value between a POC of the L1 reference picture and a POC of the current picture. Also, the L1 reference picture may be derived from among L1 reference pictures included in the L1. For example, the L1 reference picture may be derived as a reference picture including a reference area with the smallest residual, i.e., a difference value between a reference area pointed to by the MVL0 and a reference area pointed to by the derived MVL1 among the L1 reference pictures included in the L1. The difference value between the reference area pointed to by the MVL0 and the reference area pointed to by the derived MVL1 may be represented as a cost for the motion vector candidate.
[0150] Also, for example, when the motion vector candidate of the current block is MVL1, the MVL0 corresponding to the MVL1 can be derived by scaling based on a temporal distance between an L1 reference picture for the MVL1 and a current picture including the current block and a temporal distance between an L0 reference picture and the current picture. That is, the MVL1 can be derived by scaling the MVL1 based on a first temporal distance between an L1 reference picture for the MVL1 and the current picture and a second temporal distance between the L0 reference picture and the current picture. Here, the first temporal distance is a difference value between a POC of the L1 reference picture and a POC of the current picture, and the second temporal distance is a difference value between a POC of the L0 reference picture and a POC of the current picture. TD0 shown in FIG. 6 indicates the first temporal distance, and TD1 indicates the second temporal distance. Also, the L0 reference picture can be derived from among L0 reference pictures included in the L0. For example, the L0 reference picture may be derived as a reference picture including a reference area with the smallest residual, i.e., a difference value between a reference area pointed to by the MVL1 and a reference area pointed to by the derived MVL0, among the L0 reference pictures included in the L0. The difference value between the reference area pointed to by the MVL1 and the reference area pointed to by the derived MVL0 may be represented as a cost.
[0151] The encoding / decoding apparatus may derive costs for the motion vector candidates, and may derive the motion vector candidate with the smallest cost (bi-predictive motion information including MVL0 and MVL1) as the motion information of the current block.
[0152] Meanwhile, when the FRUC is applied to the current block, motion information of the current block may be derived through the TM method.
[0153] Figure 7 illustrates an example of deriving motion information through the TM method. Referring to Figure 7, a reference block for a template most similar to a template of the current block among the L0 reference blocks included in the L0 reference picture can be derived as an L0 reference block for the current block, and a motion vector pointing to the L0 reference block can be derived as an MVL0 for the current block. That is, a reference block for a template having the smallest cost with the template of the current block among the reference blocks included in the L0 reference picture can be derived as an L0 reference block for the current block, and a motion vector pointing to the L0 reference block can be derived as an MVL0 for the current block.
[0154] The template of the current block may include a left neighboring region and / or an upper neighboring region of the current block, the template of the L0 reference block is a region corresponding to the template of the current block, and the template of the reference block may include a left neighboring region and / or an upper neighboring region of the reference block. The cost for the reference block may be derived as a sum of absolute differences (SAD) between the template of the current block and the template of the reference block.
[0155] The cost can be derived based on the following formula:
[0156]
number
[0157] where i,j denotes the position (i,j) of the sample in the block template, and Cost distortion denotes the cost, and Tem pref indicates the sample value of the template of the reference block, and Temp curindicates a sample value of the template of the current block. Differences between corresponding samples between a template of a reference block and a template of the current block may be accumulated, and the accumulated differences may be used as a cost for the reference block. An L0 reference block having the smallest cost among L0 reference blocks included in an L0 reference picture may be derived as an L0 reference block for the current block, and a motion vector pointing to the L0 reference block may be derived as an MVL0 for the current block. The cost may be derived based on the template of the current block and the template of the L0 reference block as described above.
[0158] 7, a reference block for a template most similar to a template of the current block among L1 reference blocks included in an L1 reference picture may be derived as an L1 reference block for the current block, and a motion vector pointing to the L1 reference block may be derived as an MVL1 for the current block. That is, a reference block for a template having the smallest cost with the template of the current block among reference blocks included in an L1 reference picture may be derived as an L1 reference block for the current block, and a motion vector pointing to the L1 reference block may be derived as an MVL1 for the current block.
[0159] The template of the current block may include a left adjacent region and / or an upper adjacent region of the current block, and the template of the L1 reference block is a region corresponding to the template of the current block, and the template of the reference block may include a left adjacent region and / or an upper adjacent region of the reference block. The cost for the reference block may be derived as a sum of absolute differences (SAD) between the template of the current block and the template of the reference block. An L1 reference block having the smallest cost among the L1 reference blocks included in the L1 reference picture may be derived as the L1 reference block for the current block, and a motion vector pointing to the L1 reference block may be derived as the MVL1 for the current block. The cost may be derived based on the template of the current block and the template of the L1 reference block as described above.
[0160] Meanwhile, the decoding apparatus may obtain information on a matching method for deriving an optimal motion vector from the encoding apparatus, and in this case, the decoding apparatus may derive an optimal motion vector according to the matching method. For example, the decoding apparatus may obtain a flag for selecting the BM method or the TM method via a bitstream. The flag may also be referred to as a BM / TM selection flag. The decoding apparatus may select a matching method based on a value of the flag. For example, if the value of the flag is 1, the decoding apparatus may perform the BM method to derive motion information of the current block, or if the value of the flag is 0, the decoding apparatus may perform the TM method to derive motion information of the current block. Alternatively, if the value of the flag is 0, the decoding apparatus may perform the BM method to derive motion information of the current block, or if the value of the flag is 1, the decoding apparatus may perform the TM method to derive motion information of the current block.
[0161] 8a and 8b show an example of an encoding process and a decoding process performed based on FRUC.
[0162] FIG. 8a shows an example of an encoding process performed based on FRUC.
[0163] 8A, the encoding apparatus derives motion information for a current block by applying the BM mode (S800). Meanwhile, the method of deriving motion information based on the above-mentioned BM method may be referred to as the BM mode.
[0164] For example, the encoding apparatus may derive a motion information candidate list for the current block based on neighboring blocks of the current block. Candidates included in the motion information candidate list may include L0 motion information or L1 motion information.
[0165] Thereafter, the encoding apparatus can derive costs for candidates included in the motion information candidate list through the BM method, and can derive motion information for the current block based on the candidate having the smallest cost.
[0166] The cost for the candidates included in the motion information candidate list may be derived as follows: The encoding apparatus may derive a motion vector corresponding to the motion vector in a direction different from the direction associated with the motion vector among the L0 direction and the L1 direction by interpolating the motion vector of the candidate centered on the current picture, and may derive the SAD between a reference block pointed to by the motion vector of the candidate and a reference block pointed to by the derived motion vector as the cost of the candidate.
[0167] For example, when the motion candidate includes L0 motion information, the encoding apparatus may scale MVL0 for the motion candidate based on a first temporal distance and a second temporal distance to derive MVL1. The encoding apparatus may derive the SAD between the L0 reference block pointed to by the MVL0 and the L1 reference block pointed to by the MVL1 as the cost of the motion candidate. Here, the first temporal distance is a difference value between a POC of an L0 reference picture for the motion candidate and a POC of a current picture, and the second temporal distance is a difference value between a POC of an L1 reference picture for the MVL1 and a POC of the current picture.
[0168] Also, for example, when the motion candidate includes L1 motion information, the encoding apparatus may scale MVL1 for the motion candidate based on a first temporal distance and a second temporal distance to derive MVL0. The encoding apparatus may derive the SAD between the L1 reference block pointed to by the MVL1 and the L0 reference block pointed to by the MVL0 as the cost of the motion candidate. Here, the first temporal distance is a difference value between a POC of an L1 reference picture for the motion candidate and a POC of a current picture, and the second temporal distance is a difference value between a POC of an L0 reference picture for the MVL0 and a POC of the current picture.
[0169] The encoding apparatus may compare the costs of the candidates, and derive motion information (L0 motion information or L1 motion information) of the candidate having the smallest cost and motion information (L1 motion information or L0 motion information) derived based on the candidate as the motion information of the current block. The derived motion information may indicate motion information derived based on the BM mode.
[0170] The encoding apparatus derives motion information for the current block by applying the TM mode (S810). Meanwhile, the method of deriving motion information based on the above-mentioned TM method may be referred to as the TM mode.
[0171] The encoding apparatus may derive, as an MVL0 for the current block, a motion vector pointing to an L0 reference block for a template having the smallest cost with respect to the template of the current block among L0 reference blocks included in an L0 reference picture. Here, the template of the current block may include a left neighboring region and / or an upper neighboring region of the current block, the template of the L0 reference block is a region corresponding to the template of the current block, and the template of the L0 reference block may include a left neighboring region and / or an upper neighboring region of the L0 reference block. The cost for the L0 reference block may be derived as a sum of absolute differences (SAD) between the template of the current block and the template of the L0 reference block.
[0172] The encoding apparatus may also derive, as an MVL1 for the current block, a motion vector pointing to an L1 reference block for a template having the smallest cost with respect to the template of the current block among L1 reference blocks included in an L1 reference picture. Here, the template of the current block may include a left neighboring region and / or an upper neighboring region of the current block, the template of the L1 reference block is a region corresponding to the template of the current block, and the template of the L1 reference block may include a left neighboring region and / or an upper neighboring region of the L1 reference block. The cost for the L1 reference block may be derived as a Sum of Absolute Difference (SAD) between the template of the current block and the template of the L1 reference block.
[0173] The encoding apparatus may derive motion information including L0 motion information and L1 motion information as the motion information of the current block. The L0 motion information may include the MVL0 and a reference picture index pointing to the L0 reference picture, and the L1 motion information may include the MVL1 and a reference picture index pointing to the L1 reference picture.
[0174] The encoding apparatus may perform a rate distortion (RD) check on the motion information derived based on the BM mode and the motion information derived based on the TM mode, and may select an optimal mode for the current block from the BM mode and the TM mode (S820). The encoding apparatus may select a mode to be applied to the current block from the BM mode and the TM mode based on an RD cost for the motion information derived based on the BM mode and the motion information derived based on the TM mode. Meanwhile, the encoding apparatus may generate information indicating the selected mode. For example, the encoding apparatus may generate a flag indicating one of the BM mode and the TM mode, and may signal the flag via a bitstream.
[0175] FIG. 8b shows an example of a decoding process performed based on FRUC.
[0176] Referring to Fig. 8b, a decoding apparatus may determine whether a BM mode is applied to the current block (S830). The decoding apparatus may determine whether a BM mode or a TM mode is applied to the current block. For example, the decoding apparatus may obtain a flag indicating one of the BM mode and the TM mode via a bitstream. The decoding apparatus may derive one of the BM mode and the TM mode indicated by the flag as a mode to be applied to the current block.
[0177] If the BM mode is applied to the current block, the decoding apparatus derives motion information for the current block based on the BM mode (S840).
[0178] For example, the decoding apparatus may derive a motion information candidate list for the current block based on neighboring blocks of the current block. Candidates included in the motion information candidate list may include L0 motion information or L1 motion information.
[0179] Thereafter, the decoding apparatus can derive costs for candidates included in the motion information candidate list through the BM method, and can derive motion information for the current block based on the candidate having the smallest cost.
[0180] The cost for the candidates included in the motion information candidate list can be derived as follows: The decoding apparatus may derive a motion vector corresponding to the motion vector in a direction different from the direction associated with the motion vector among the L0 direction and the L1 direction by interpolating the motion vector of the candidate centered on the current picture, and may derive the SAD between a reference block pointed to by the motion vector of the candidate and a reference block pointed to by the derived motion vector as the cost of the candidate.
[0181] For example, if the motion candidate includes L0 motion information, the decoding apparatus may scale MVL0 for the motion candidate based on a first temporal distance and a second temporal distance to derive MVL1. The decoding apparatus may derive the SAD between the L0 reference block pointed to by the MVL0 and the L1 reference block pointed to by the MVL1 as the cost of the motion candidate. Here, the first temporal distance is a difference value between a POC of an L0 reference picture for the motion candidate and a POC of a current picture, and the second temporal distance is a difference value between a POC of an L1 reference picture for the MVL1 and a POC of the current picture.
[0182] Also, for example, when the motion candidate includes L1 motion information, the decoding apparatus may scale MVL1 for the motion candidate based on a first temporal distance and a second temporal distance to derive MVL0. The decoding apparatus may derive the SAD between the L1 reference block pointed to by the MVL1 and the L0 reference block pointed to by the MVL0 as the cost of the motion candidate. Here, the first temporal distance is a difference value between a POC of an L1 reference picture for the motion candidate and a POC of a current picture, and the second temporal distance is a difference value between a POC of an L0 reference picture for the MVL0 and a POC of the current picture.
[0183] The decoding device can compare the costs of the candidates and derive the motion information (L0 motion information or L1 motion information) of the candidate having the smallest cost and the motion information derived based on the candidate (L1 motion information or L0 motion information) as the motion information of the current block.
[0184] On the other hand, if the TM mode is applied to the current block, the decoding apparatus derives motion information for the current block based on the TM mode (S850).
[0185] The decoding apparatus may derive, as an MVL0 for the current block, a motion vector pointing to an L0 reference block for a template having the smallest cost with respect to the template of the current block among L0 reference blocks included in an L0 reference picture. Here, the template of the current block may include a left neighboring region and / or an upper neighboring region of the current block, the template of the L0 reference block is a region corresponding to the template of the current block, and the template of the L0 reference block may include a left neighboring region and / or an upper neighboring region of the L0 reference block. The cost for the L0 reference block may be derived as a Sum of Absolute Difference (SAD) between the template of the current block and the template of the L0 reference block.
[0186] Also, the decoding apparatus may derive, as an MVL1 for the current block, a motion vector pointing to an L1 reference block for a template having the smallest cost with respect to the template of the current block among L1 reference blocks included in an L1 reference picture. Here, the template of the current block may include a left neighboring region and / or an upper neighboring region of the current block, the template of the L1 reference block is a region corresponding to the template of the current block, and the template of the L1 reference block may include a left neighboring region and / or an upper neighboring region of the L1 reference block. The cost for the L1 reference block may be derived as a Sum of Absolute Difference (SAD) between the template of the current block and the template of the L1 reference block.
[0187] The decoding apparatus may derive motion information including L0 motion information and L1 motion information as the motion information of the current block. The L0 motion information may include the MVL0 and a reference picture index pointing to the L0 reference picture, and the L1 motion information may include the MVL1 and a reference picture index pointing to the L1 reference picture.
[0188] Meanwhile, when the motion information of the current block is derived through FRUC as described above, adaptively applying the BIO based on the mode (BM mode or TM mode) applied to the current block can further improve coding efficiency. Specifically, the BM mode of FRUC and the BIO are prediction methods that take into account assumptions 1 and 2 described below.
[0189] -Assumption 1. Objects move at the same speed over a short period of time.
[0190] - Assumption 2. Sample values for a target object do not change in consecutive frames.
[0191] When the BM mode is applied, the motion vector of each (sub) block can be derived taking into account the assumption, and when the BIO is applied, the offset of each (sub) block (a value derived based on the motion vector of each (sub) block) can be derived taking into account the assumption. Therefore, it is possible to further improve coding efficiency by not applying methods performed based on the same assumptions repeatedly. Therefore, when FRUC is applied to the current block, if motion information is derived based on the TM mode, the BIO is applied, and if motion information is derived based on the BM mode, a method of not applying the BIO can be proposed.
[0192] 9a to 9b show an example of encoding / decoding a current block to which FRUC and / or BIO is applied.
[0193] FIG. 9a illustrates an example of encoding a current block to which FRUC and / or BIO is applied.
[0194] Referring to Fig. 9a, an encoding apparatus may derive motion information of a current block by applying FRUC to the current block (S900). The encoding apparatus may derive motion information for the current block based on a BM mode and a TM mode for the current block. A specific method for deriving motion information for the current block based on the BM mode has been described above. Also, a specific method for deriving motion information for the current block based on the TM mode has been described above.
[0195] The encoding apparatus may determine whether a TM mode is applied to a current block (S910). The encoding apparatus may compare motion information derived based on the BM mode with motion information derived based on the TM mode to select a mode to be applied to the current block.
[0196] When the TM mode is applied to the current block, the encoding apparatus may perform BIO prediction based on motion information for the current block (S920). The encoding apparatus may perform BIO prediction based on the motion information derived based on the TM mode. For example, the motion information may include MVL0 and MVL1 for the current block, and the encoding apparatus may obtain a block-wise refined motion vector for the current block based on MVL0 and MVL1 for the current block, a first predictor, and a second predictor. For example, a refined motion vector for a block of 4×4 size included in the current block may be derived. The encoding apparatus may derive a refined motion vector for a target block of the current block based on the MVL0 and the MVL1 for the current block, the first predictor, and the second predictor, and may derive a refined prediction sample based on the refined motion vector.
[0197] Specifically, the encoding apparatus may derive a first predictor for a position according to the MVL0 and a second predictor for a position according to the MVL1 based on each sample of the target block, and may derive a gradient for the target sample based on the first predictor and the second predictor. Thereafter, the encoding apparatus may derive a refined motion vector for the target block based on the gradient for the sample. For example, the refined motion vector for the target block may be derived based on Equation 16 above. The encoding apparatus may derive a refined prediction sample for the target block based on the refined motion vector.
[0198] Meanwhile, if the BM mode is applied to the current block, the encoding apparatus may perform prediction based on derived motion information without performing BIO prediction on the current block.
[0199] FIG. 9b shows an example of decoding a current block to which FRUC and / or BIO is applied.
[0200] Referring to Fig. 9b, a decoding apparatus may determine whether a TM mode is applied to the current block (S930). The decoding apparatus may determine whether a BM mode or a TM mode is applied to the current block. For example, the decoding apparatus may obtain a flag indicating one of the BM mode and the TM mode via a bitstream. The decoding apparatus may derive one of the BM mode and the TM mode indicated by the flag as a mode to be applied to the current block.
[0201] On the other hand, if the TM mode is applied to the current block, the decoding apparatus derives motion information for the current block based on the TM mode (S940).
[0202] The decoding apparatus may derive, as an MVL0 for the current block, a motion vector pointing to an L0 reference block for a template having the smallest cost with respect to the template of the current block among L0 reference blocks included in an L0 reference picture. Here, the template of the current block may include a left neighboring region and / or an upper neighboring region of the current block, the template of the L0 reference block is a region corresponding to the template of the current block, and the template of the L0 reference block may include a left neighboring region and / or an upper neighboring region of the L0 reference block. The cost for the L0 reference block may be derived as a Sum of Absolute Difference (SAD) between the template of the current block and the template of the L0 reference block.
[0203] Also, the decoding apparatus may derive, as an MVL1 for the current block, a motion vector pointing to an L1 reference block for a template having the smallest cost with respect to the template of the current block among L1 reference blocks included in an L1 reference picture. Here, the template of the current block may include a left neighboring region and / or an upper neighboring region of the current block, the template of the L1 reference block is a region corresponding to the template of the current block, and the template of the L1 reference block may include a left neighboring region and / or an upper neighboring region of the L1 reference block. The cost for the L1 reference block may be derived as a Sum of Absolute Difference (SAD) between the template of the current block and the template of the L1 reference block.
[0204] The decoding apparatus may derive motion information including L0 motion information and L1 motion information as the motion information of the current block. The L0 motion information may include the MVL0 and a reference picture index pointing to the L0 reference picture, and the L1 motion information may include the MVL1 and a reference picture index pointing to the L1 reference picture.
[0205] When the TM mode is applied to the current block, the decoding apparatus may perform BIO prediction based on motion information for the current block (S950). The decoding apparatus may perform BIO prediction based on the motion information derived based on the TM mode. For example, the motion information may include MVL0 and MVL1 for the current block, and the decoding apparatus may obtain a block-wise refined motion vector for the current block based on MVL0 and MVL1 for the current block and a first predictor and a second predictor. For example, a refined motion vector for a block of 4×4 size included in the current block may be derived. The decoding apparatus may derive a refined motion vector for a target block of the current block based on the MVL0 and the MVL1 for the current block and the first predictor and the second predictor, and may derive a refined prediction sample based on the refined motion vector.
[0206] Specifically, the decoding apparatus may derive a first predictor for a position according to the MVL0 and a second predictor for a position according to the MVL1 based on each sample of the target block, and may derive a gradient for the target sample based on the first predictor and the second predictor. Thereafter, the decoding apparatus may derive a refined motion vector for the target block based on the gradient for the sample. For example, the refined motion vector for the target block may be derived based on Equation 16 above. The decoding apparatus may derive a refined prediction sample for the target block based on the refined motion vector.
[0207] On the other hand, if the BM mode is applied to the current block, the decoding apparatus derives motion information for the current block based on the BM mode (S960).
[0208] For example, the decoding apparatus may derive a motion information candidate list for the current block based on neighboring blocks of the current block. Candidates included in the motion information candidate list may include L0 motion information or L1 motion information.
[0209] Thereafter, the decoding apparatus can derive costs for candidates included in the motion information candidate list through the BM method, and can derive motion information for the current block based on the candidate having the smallest cost.
[0210] The cost for the candidates included in the motion information candidate list can be derived as follows: The decoding apparatus may derive a motion vector corresponding to the motion vector in a direction different from the direction associated with the motion vector among the L0 direction and the L1 direction by interpolating the motion vector of the candidate centered on the current picture, and may derive the SAD between a reference block pointed to by the motion vector of the candidate and a reference block pointed to by the derived motion vector as the cost of the candidate.
[0211] For example, if the motion candidate includes L0 motion information, the decoding apparatus may scale MVL0 for the motion candidate based on a first temporal distance and a second temporal distance to derive MVL1. The decoding apparatus may derive the SAD between the L0 reference block pointed to by the MVL0 and the L1 reference block pointed to by the MVL1 as the cost of the motion candidate. Here, the first temporal distance is a difference value between a POC of an L0 reference picture for the motion candidate and a POC of a current picture, and the second temporal distance is a difference value between a POC of an L1 reference picture for the MVL1 and a POC of the current picture.
[0212] Also, for example, when the motion candidate includes L1 motion information, the decoding apparatus may scale MVL1 for the motion candidate based on a first temporal distance and a second temporal distance to derive MVL0. The decoding apparatus may derive the SAD between the L1 reference block pointed to by the MVL1 and the L0 reference block pointed to by the MVL0 as the cost of the motion candidate. Here, the first temporal distance is a difference value between a POC of an L1 reference picture for the motion candidate and a POC of a current picture, and the second temporal distance is a difference value between a POC of an L0 reference picture for the MVL0 and a POC of the current picture.
[0213] The decoding device may compare the costs of the candidates, and derive motion information (L0 motion information or L1 motion information) of the candidate having the smallest cost and motion information (L1 motion information or L0 motion information) derived based on the candidate as the motion information of the current block. The decoding device may perform prediction for the current block based on the motion information.
[0214] Figure 10 illustrates a video encoding method by an encoding device according to the present invention. The method disclosed in Figure 10 may be performed by the encoding device disclosed in Figure 1. Specifically, for example, S1000 to S1040 in Figure 10 may be performed by a prediction unit of the encoding device, and S1050 may be performed by an entropy encoding unit of the encoding device.
[0215] The encoding apparatus derives (S1000) a reference picture list 0 (L0) and a reference picture list 1 (L1). The encoding apparatus may derive the reference picture list 0 (L0) and the reference picture list 1 (L1), where the reference picture list 0 is referred to as L0 (List0) and the reference picture list 1 is referred to as L1 (List1).
[0216] The encoding apparatus derives two motion vectors (MVs) for a current block, the two MVs including MVL0 for the L0 and MVL1 for the L1 (S1010). The encoding apparatus may derive the two MVs when the current block is included in a B slice and bi-prediction is applied to the current block. The two MVs are MVL0 for the L0 and MVL1 for the L1.
[0217] The encoding apparatus may derive the two MVs for the current block based on neighboring blocks of the current block.
[0218] For example, the encoding apparatus may generate a motion information candidate list based on motion information of the neighboring blocks, and may select a specific candidate from among the candidates included in the motion information candidate list based on an RD cost. The encoding apparatus may generate an index indicating the specific candidate. The index may be included in the information for inter prediction described above. The encoding apparatus may derive MVL0 and MVL1 of the specific candidate as the two MVs of the current block.
[0219] Alternatively, the encoding apparatus may derive the two MVs by applying FRUC to the current block.
[0220] Specifically, for example, an encoding apparatus may determine whether a template matching (TM) mode or a bi-lateral matching (BM) mode is applied to the current block. Also, the encoding apparatus may generate a flag indicating one of the TM mode and the BM mode. The information regarding the inter prediction may include the flag.
[0221] When the BM mode is applied to the current block, the encoding apparatus may derive a motion information candidate list for the current block based on neighboring blocks of the current block, and may select a specific candidate from among the candidates included in the motion information candidate list. The encoding apparatus may scale a first motion vector of the specific candidate based on a first temporal distance and a second temporal distance to derive a second motion vector associated with a list different from a list associated with the motion vector among the L0 and the L1, and may derive the first motion vector and the second motion vector as an MV of the current block. Here, the first temporal distance is a difference between a POC (Picture Order Count) of a reference picture for the first motion vector and a POC of a current picture including the current block, and the second temporal distance is a difference between a POC of a reference picture for the second motion vector and a POC of the current picture. Meanwhile, a candidate having the smallest cost among the candidates included in the motion information candidate list may be selected as the specific candidate, and the cost for the specific candidate may be derived as a Sum of Absolute Difference (SAD) between a reference block indicated by the first motion vector and a reference block indicated by the second motion vector. That is, the encoding apparatus may derive a second motion vector of a candidate for the motion information candidate list, derive a cost for the candidate, and compare the costs to select the specific candidate.
[0222] Furthermore, when the TM mode is applied to the current block, the encoding apparatus may derive an L0 reference picture and an L1 reference picture of the current block, derive a motion vector pointing to an L0 reference block having the smallest cost among the L0 reference blocks of the L0 reference picture as the MVL0, and derive a motion vector pointing to an L1 reference block having the smallest cost among the L1 reference blocks of the L1 reference picture as the MVL1. Here, the cost of the L0 reference block may be derived as a Sum of Absolute Difference (SAD) between a template of the current block and a template of the L0 reference block, and the cost of the L1 reference block may be derived as a SAD between a template of the current block and a template of the L1 reference block. Also, the template of the current block may indicate a specific area including adjacent samples of the current block, the template of the L0 reference block may indicate a specific area including adjacent samples of the L0 reference block corresponding to the adjacent samples of the current block, and the template of the L1 reference block may indicate a specific area including adjacent samples of the L1 reference block corresponding to the adjacent samples of the current block. Meanwhile, the cost of the L0 reference block or the L1 reference block may be derived according to Equation 24. Also, the L0 reference picture may be derived as one of the L0 reference pictures included in the L0, and the L1 reference picture may be derived as one of the L1 reference pictures included in the L1. Also, the L0 reference picture and the L1 reference picture may be derived based on motion information of adjacent blocks of the current block.
[0223] Meanwhile, the encoding apparatus may generate a FRUC flag indicating whether FRUC is applied to the current block, and the information on inter prediction may include a FRUC flag for the current block.
[0224] The encoding apparatus determines whether bi-prediction optical-flow (BIO) prediction for deriving a refined motion vector in sub-block units is applied to the current block (S1020). For example, the encoding apparatus may determine whether the BIO prediction is applied to the current block based on the size of the current block. For example, if the size of the current block is smaller than or equal to 4×4 size, the BIO prediction is not applied to the current block, and if the size of the current block is larger than 4×4 size, the BIO prediction may be applied to the current block.
[0225] As another example, the encoding apparatus may determine whether the BIO prediction is applied to the current block based on whether the TM mode is applied. For example, if the BM mode is applied to the current block, the BIO prediction may not be applied to the current block, and if the TM mode is applied to the current block, the BIO prediction may be applied to the current block.
[0226] When the BIO prediction is applied to the current block, the encoding apparatus derives the refinement motion vector for a sub-block of the current block based on the MVL0 and the MVL1 (S1030). Here, the size of the sub-block may be preset. For example, the size of the sub-block may be 4×4. The refinement motion vector may be derived in units of the sub-block of the current block.
[0227] Based on the MVL0 and the MVL1, x-axis, y-axis, and time-axis gradient maps of reference sample values of the sub-block may be derived, and the refined motion vector for the sub-block may be derived based on the gradient map for the sub-block. The refined motion vector for the sub-block may be derived based on Equation 21 or Equation 23.
[0228] The encoding apparatus derives a prediction sample based on the refined motion vector (S1040). The encoding apparatus may derive a prediction sample based on the refined motion vector. The prediction sample for the sub-block of the current block is expressed as V x V x、bk Change to V y V y、bk In addition, when the magnitudes of the first and second time distances are different, the predicted samples for the sub-blocks of the current block can be derived based on the equation (15) changed to V x V x、bk Change to V y V y、bk It can be derived based on the formula changed to:
[0229] On the other hand, even if not shown in FIG. 10, if it is determined that the BIO prediction is not applied, the encoding device can derive a predicted sample for the target sample based on the MVL0 and the MVL1.
[0230] The encoding apparatus entropy-encodes information regarding inter-prediction of the current block (S1050). The encoding apparatus may entropy-encode the information regarding inter-prediction and output the information in the form of a bitstream. The bitstream may be transmitted to a decoding apparatus via a network or a storage medium.
[0231] The information regarding inter prediction may include prediction mode information of the current block. The prediction mode information may indicate an inter prediction mode applied to the current block. In addition, the information regarding inter prediction may include a flag indicating one of the TM mode and the BM mode. In addition, the information regarding inter prediction may include a FRUC flag for the current block.
[0232] Meanwhile, the encoding apparatus may generate a residual sample based on an original sample of an original picture and the generated predicted sample. The encoding apparatus may generate information about the residual based on the residual sample. The information about the residual may include a transform coefficient for the residual sample. The encoding apparatus may derive the reconstructed sample based on the predicted sample and the residual sample. That is, the encoding apparatus may derive the reconstructed sample by adding the predicted sample and the residual sample. Also, the encoding apparatus may encode information about the residual and output it in the form of a bitstream. The bitstream may be transmitted to a decoding apparatus via a network or a storage medium.
[0233] Fig. 11 is a schematic diagram of an encoding device for performing a video encoding method according to the present invention. The method disclosed in Fig. 10 can be performed by the encoding device disclosed in Fig. 11. Specifically, for example, a prediction unit of the encoding device of Fig. 11 can perform S1000 to S1040 of Fig. 10, and an entropy encoding unit of the encoding device can perform S1050 of Fig. 10.
[0234] Figure 12 is a schematic diagram illustrating a video decoding method by a decoding device according to the present invention. The method disclosed in Figure 12 may be performed by the decoding device disclosed in Figure 2. Specifically, for example, S1200 to S1240 in Figure 12 may be performed by a prediction unit of the decoding device.
[0235] The decoding apparatus derives reference picture list 0 (L0) and reference picture list 1 (L1) (S1200). The decoding apparatus may derive reference picture list 0 (L0) and reference picture list 1 (L1), where reference picture list 0 is referred to as L0 (List0) and reference picture list 1 is referred to as L1 (List1).
[0236] The decoding apparatus derives two motion vectors (MVs) for the current block, the two MVs including MVL0 for the L0 and MVL1 for the L1 (S1210). The decoding apparatus may obtain information for inter prediction through a bitstream, and may derive an inter prediction mode to be applied to the current block based on the information for inter prediction.
[0237] When the current block is included in a B slice and bi-prediction is applied to the current block, the decoding apparatus may derive the two MVs, which are MVL0 for the L0 and MVL1 for the L1.
[0238] The decoding apparatus can derive the two MVs for the current block based on the neighboring blocks of the current block.
[0239] For example, the decoding apparatus may generate a motion information candidate list based on motion information of the neighboring blocks, and may select a specific candidate from among the candidates included in the motion information candidate list based on an index indicating the specific candidate. The index may be included in the information for inter prediction described above. The decoding apparatus may derive MVL0 and MVL1 of the specific candidate as the two MVs of the current block.
[0240] Alternatively, the decoding apparatus may derive the two MVs by applying FRUC to the current block.
[0241] Specifically, for example, the decoding apparatus may determine whether a template matching (TM) mode or a bi-lateral matching (BM) mode is applied to the current block. As an example, the decoding apparatus may obtain a flag indicating one of the TM mode and the BM mode, and may determine that the mode indicated by the flag is applied to the current block. The information regarding inter prediction may include the flag.
[0242] When the BM mode is applied to the current block, the decoding apparatus may derive a motion information candidate list for the current block based on neighboring blocks of the current block, and may select a specific candidate from among the candidates included in the motion information candidate list. The decoding apparatus may scale a first motion vector of the specific candidate based on a first temporal distance and a second temporal distance to derive a second motion vector associated with a list different from a list associated with the motion vector among the L0 and the L1, and may derive the first motion vector and the second motion vector as an MV of the current block. Here, the first temporal distance is a difference between a POC (Picture Order Count) of a reference picture for the first motion vector and a POC of a current picture including the current block, and the second temporal distance is a difference between a POC of a reference picture for the second motion vector and a POC of the current picture. Meanwhile, a candidate having the smallest cost among the candidates included in the motion information candidate list may be selected as the specific candidate, and the cost for the specific candidate may be derived as a Sum of Absolute Difference (SAD) between a reference block indicated by the first motion vector and a reference block indicated by the second motion vector. That is, the decoding apparatus may derive a second motion vector of a candidate for the motion information candidate list, derive a cost for the candidate, and compare the costs to select the specific candidate.
[0243] Furthermore, when the TM mode is applied to the current block, the decoding apparatus may derive an L0 reference picture and an L1 reference picture of the current block, derive a motion vector pointing to an L0 reference block having the smallest cost among the L0 reference blocks of the L0 reference picture as the MVL0, and derive a motion vector pointing to an L1 reference block having the smallest cost among the L1 reference blocks of the L1 reference picture as the MVL1. Here, the cost of the L0 reference block may be derived as a Sum of Absolute Difference (SAD) between a template of the current block and a template of the L0 reference block, and the cost of the L1 reference block may be derived as a SAD between a template of the current block and a template of the L1 reference block. Also, the template of the current block may indicate a specific area including adjacent samples of the current block, the template of the L0 reference block may indicate a specific area including adjacent samples of the L0 reference block corresponding to the adjacent samples of the current block, and the template of the L1 reference block may indicate a specific area including adjacent samples of the L1 reference block corresponding to the adjacent samples of the current block. Meanwhile, the cost of the L0 reference block or the L1 reference block may be derived according to Equation 24. Also, the L0 reference picture may be derived as one of the L0 reference pictures included in the L0, and the L1 reference picture may be derived as one of the L1 reference pictures included in the L1. Also, the L0 reference picture and the L1 reference picture may be derived based on motion information of adjacent blocks of the current block.
[0244] Meanwhile, the information on inter prediction may include a FRUC flag for the current block, and the FRUC flag may indicate whether FRUC is applied to the current block.
[0245] The decoding apparatus determines whether bi-prediction optical-flow (BIO) prediction for deriving a refined motion vector in sub-block units is applied to the current block (S1220). For example, the decoding apparatus may determine whether the BIO prediction is applied to the current block based on the size of the current block. For example, if the size of the current block is smaller than or equal to 4×4 size, the BIO prediction is not applied to the current block, and if the size of the current block is larger than 4×4 size, the BIO prediction may be applied to the current block.
[0246] As another example, the decoding apparatus may determine whether the BIO prediction is applied to the current block based on whether the TM mode is applied. For example, if the BM mode is applied to the current block, the BIO prediction may not be applied to the current block, and if the TM mode is applied to the current block, the BIO prediction may be applied to the current block.
[0247] When the BIO prediction is applied to the current block, the decoding apparatus derives the refinement motion vector for a sub-block of the current block based on the MVL0 and the MVL1 (S1230). Here, the size of the sub-block may be preset. For example, the size of the sub-block may be 4×4. The refinement motion vector may be derived in units of the sub-block of the current block.
[0248] Based on the MVL0 and the MVL1, x-axis, y-axis, and time-axis gradient maps of reference sample values of the sub-block may be derived, and the refined motion vector for the sub-block may be derived based on the gradient map for the sub-block. The refined motion vector for the sub-block may be derived based on Equation 21 or Equation 23.
[0249] The decoding apparatus derives a prediction sample based on the refined motion vector (S1240). The decoding apparatus may derive a prediction sample based on the refined motion vector. The prediction sample for the sub-block of the current block is expressed as V x V x、bk Change to V y V y、bk In addition, when the magnitudes of the first and second time distances are different, the predicted samples for the sub-blocks of the current block can be derived based on the equation (15) changed to V x V x、bk Change to V y V y、bk It can be derived based on the formula changed to:
[0250] The decoding apparatus may use the predicted samples as reconstructed samples according to a prediction mode, or may generate reconstructed samples by adding residual samples to the predicted samples.
[0251] If a residual sample for the current block exists, the decoding apparatus may obtain information on the residual for the current block from the bitstream. The information on the residual may include a transform coefficient for the residual sample. The decoding apparatus may derive the residual sample (or a residual sample array) for the current block based on the residual information. The decoding apparatus may generate a reconstructed sample based on the prediction sample and the residual sample, and may derive a reconstructed block or a reconstructed picture based on the reconstructed sample. As described above, the decoding apparatus may then apply an in-loop filtering procedure, such as a deblocking filtering and / or an SAO procedure, to the reconstructed picture to improve subjective / objective picture quality as necessary.
[0252] On the other hand, even if not shown in Fig. 12, if it is determined that the BIO prediction is not applied, the decoding apparatus may derive a predicted sample for the target sample based on the MVL0 and the MVL1. The decoding apparatus may use the predicted sample as a reconstructed sample according to a prediction mode, or may generate a reconstructed sample by adding a residual sample to the predicted sample.
[0253] Figure 13 is a schematic diagram of a decoding device for performing a video decoding method according to the present invention. The method disclosed in Figure 12 can be performed by the decoding device disclosed in Figure 13. Specifically, for example, a prediction unit of the decoding device of Figure 13 can perform S1200 to S1240 of Figure 12. Meanwhile, the information on the inter prediction and / or the information on the residual can be obtained by an entropy decoding unit of the decoding device disclosed in Figure 13.
[0254] According to the present invention described above, it is possible to reduce the computational complexity of inter prediction using refined motion vectors derived on a sub-block basis by determining whether BIO prediction is applicable to the current block, thereby improving overall coding efficiency.
[0255] In addition, according to the present invention, it is possible to reduce the computational complexity of inter prediction using refined motion vectors derived on a sub-block basis by determining whether BIO prediction is applicable based on the FRUC mode, thereby improving overall coding efficiency.
[0256] In the above embodiments, the method is described based on a flow chart with a series of steps or blocks, but the present invention is not limited to the order of steps, and certain steps may occur in a different order or simultaneously with other steps than those described. Also, those skilled in the art will understand that the steps shown in the flow chart are not exclusive, and other steps may be included, or one or more steps of the flow chart may be omitted without affecting the scope of the present invention.
[0257] The above-described method according to the present invention can be embodied in software form, and the encoding device and / or decoding device according to the present invention can be included in a device that performs image processing, such as a TV, a computer, a smartphone, a set-top box, a display device, etc.
[0258] When the embodiments of the present invention are implemented in software, the methods described above may be implemented with modules (processes, functions, etc.) that perform the functions described above. The modules may be stored in a memory and executed by a processor. The memory may be internal or external to the processor and may be coupled to the processor in various well-known ways. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and / or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and / or other storage devices. That is, the embodiments described in the present invention may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in the drawings may be implemented and executed on a computer, processor, microprocessor, controller, or chip.
[0259] In addition, the decoding device and encoding device to which the present invention is applied may be included in a multimedia broadcast transmitting / receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video interactive device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camcorder, a custom video (VoD) service providing device, an over-the-top video (OTT) device, an internet streaming service providing device, a three-dimensional (3D) video device, an image telephone video device, and a medical video device, and may be used to process a video signal or a data signal. For example, an over-the-top video (OTT) device may include a game console, a Blu-ray player, an internet-connected TV, a home theater system, a smartphone, a tablet PC, a digital video recorder (DVR), and the like.
[0260] In addition, the processing method to which the present invention is applied may be produced in the form of a program executed by a computer and may be stored in a computer-readable recording medium. In addition, multimedia data having a data structure according to the present invention may also be stored in a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium may include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable recording medium may include a medium embodied in the form of a carrier wave (e.g., transmission via the Internet). In addition, a bit stream generated by the encoding method may be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network. In addition, an embodiment of the present invention may be embodied in a computer program product by a program code, and the program code may be executed by a computer according to an embodiment of the present invention. The program code may be stored on a computer-readable carrier.
[0261] Moreover, the content streaming system to which the present invention is applied can largely include an encoding server, a streaming server, a web server, a media repository, a user device, and a multimedia input device.
[0262] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, when a multimedia input device such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server can be omitted. The bitstream can be generated by an encoding method or a bitstream generation method to which the present invention is applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.
[0263] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, and in this case, the control server controls commands / responses between devices in the content streaming system.
[0264] The streaming server may receive content from a media repository and / or an encoding server. For example, when receiving content from the encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a certain period of time in order to provide a smooth streaming service.
[0265] Examples of the user devices include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glass, head mounted displays (HMDs)), digital TVs, desktop computers, digital signage, etc. Each server in the content streaming system can be operated as a distributed server, and in this case, data received by each server can be processed in a distributed manner.
Claims
1. 1. A video decoding method performed by a decoding device, comprising: obtaining video information including residual information from a bitstream; deriving a current block based on a division of a current picture, the current block being a non-square block; deriving a reference picture list 0 (L0) and a reference picture list 1 (L1); deriving two motion vectors (MVs) for the current block, the two MVs including MVL0 for the L0 and MVL1 for the L1; determining whether bi-directional optical-flow (BIO) prediction is applied to the current block; deriving a predicted sample of the current block based on the MVL0, the MVL1, and a result of determining to apply the BIO prediction to the current block; generating a residual sample of the current block based on residual information; generating a reconstructed sample of the current block based on the predicted sample and the residual sample; Determining whether the BIO prediction is applied to the current block includes: determining whether to apply the BIO prediction to the current block by comparing a width of the current block to a first threshold and a height of the current block to a second threshold; determining that the BIO prediction is applied to the current block based on a width of the current block being greater than or equal to the first threshold and a height of the current block being greater than or equal to the second threshold; and determining that the BIO prediction is not applied to the current block based on the width of the current block being smaller than the first threshold and the height of the current block being smaller than the second threshold.
2. 1. A video encoding method performed by an encoding device, comprising: deriving a current block based on a division of a current picture, the current block being a non-square block; deriving a reference picture list 0 (L0) and a reference picture list 1 (L1); deriving two motion vectors (MVs) for the current block, the two MVs including MVL0 for the L0 and MVL1 for the L1; determining whether bi-directional optical-flow (BIO) prediction is applied to the current block; deriving a predicted sample of the current block based on the MVL0, the MVL1, and a result of determining to apply the BIO prediction to the current block; generating a residual sample of the current block based on the predicted sample; generating residual information based on the residual samples; encoding image information including information about inter prediction of the current block and the residual information; The step of determining whether the BIO prediction is applied to the current block comprises: determining whether to apply the BIO prediction to the current block by comparing a width of the current block to a first threshold and a height of the current block to a second threshold; determining that the BIO prediction is applied to the current block based on a width of the current block being greater than or equal to the first threshold and a height of the current block being greater than or equal to the second threshold; and determining that the BIO prediction is not applied to the current block based on the width of the current block being smaller than the first threshold and the height of the current block being smaller than the second threshold.
3. A method for transmitting data for a video, comprising the steps of: obtaining a bitstream for the video, the bitstream deriving a current block based on a division of a current picture, the current block being a non-square block; deriving a reference picture list 0 (L0) and a reference picture list 1 (L1); deriving two motion vectors (MVs) for the current block, the two MVs including MVL0 for the L0 and MVL1 for the L1; determining whether bi-directional optical-flow (BIO) prediction is applied to the current block; deriving a predicted sample of the current block based on the MVL0, the MVL1, and a result of determining that the BIO prediction is applied to the current block; generating a residual sample of the current block based on the predicted sample; generating residual information based on the residual samples; and encoding video information including information about inter prediction of the current block and the residual information; transmitting the data including the bitstream; Determining whether the BIO prediction is applied to the current block includes: determining whether to apply the BIO prediction to the current block by comparing a width of the current block to a first threshold and a height of the current block to a second threshold; determining that the BIO prediction is applied to the current block based on a width of the current block being greater than or equal to the first threshold and a height of the current block being greater than or equal to the second threshold; A method for transmitting data for an image, wherein it is determined that the BIO prediction is not applied to the current block based on the width of the current block being smaller than the first threshold and the height of the current block being smaller than the second threshold.
Citation Information
Patent Citations
Image decoding apparatus, image decoding method and image encoding apparatus
WO2013047805A1
Moving image decoding device, moving image encoding device, and prediction image generation device
WO2017134957A1
Motion-compensation prediction based on BI-directional optical flow
WO2019010156A1
