Method and apparatus for encoding or decoding images, and non-transitory computer-readable medium
Through the restricted bidirectional prediction method and inter-frame mode adjustment, motion vectors are generated, which solves the problems of low inter-frame prediction efficiency and large storage bandwidth in the prior art, and realizes efficient video encoding and decoding.
Patent Information
- Application Number
- CN202411372248.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-10-23
- Filing Date
- 2019-09-25
- Publication Date
- 2024-11-15
AI Technical Summary
In the existing video compression technology, inter-frame prediction efficiency is low, memory bandwidth consumption is large, and the block division structure is complex, which affects coding efficiency.
The restricted two-way prediction method is adopted to adjust the motion information based on the predefined inter-frame mode, and generate motion vectors by combining the candidate list and the affine candidate list, limiting the storage bandwidth consumption, and improving the inter-frame prediction efficiency.
It effectively reduces storage bandwidth consumption, improves the efficiency of image encoding and decoding, simplifies the block division structure, and improves video compression performance.
Smart Images

Figure CN118972576A_ABST
Abstract
Description
[0001] This application is a divisional application of invention patent application No. 201980071465.7, whose application date is September 25, 2019 and whose invention name is “Method and device for encoding or decoding an image based on inter-frame mode”. Technical Field
[0002] The present disclosure relates to video encoding / decoding methods and apparatus. Background Art
[0003] With the recent increase in demand for high-resolution and high-definition video, there has been a need for efficient video compression technology for next-generation video services. Based on this demand, ISO / IEC MPEG and ITU-T VCEG, which have unified the standardization of H.264 / AVC and HEVC video compression standards, formed JVET (Joint Video Exploration Team) in October 2015 and conducted research and exploration to establish a new video compression standard. In April 2018, with the evaluation of the response to the new video compression standard CfP (Call for Proposals), new video compression standardization began.
[0004] In video compression technology, a block division structure refers to a unit for performing encoding and decoding and a unit to which main encoding and decoding techniques such as prediction and transformation are applied. With the development of video compression technology, the size of blocks used for encoding and decoding has gradually increased, and more various division types are supported as block division types. In addition, video compression is performed using not only units for encoding and decoding but also units subdivided according to the role of blocks.
[0005] In the HEVC standard, video encoding and decoding are performed using unit blocks subdivided according to a quadtree type block division structure and functions for prediction and transformation. In addition to the quadtree type block division structure, various types of block division structures are proposed, such as QTBT (quadtree plus binary tree) in the form of a combination of quadtree and binary tree and MTT (Multi-Type Tree) in which a ternary tree is combined with QTBT to improve video coding efficiency. By supporting various block sizes and various types of block division structures, a picture is divided into multiple blocks, and information in units of coding units such as coding modes, motion information, and intra-frame prediction direction information corresponding to each block is expressed in various ways, so the number of bits expressing information in units of coding units is significantly increased. Summary of the invention
[0006] Technical issues
[0007] The image encoding / decoding method and apparatus according to the present disclosure provide a restricted bidirectional prediction method.
[0008] An image encoding / decoding method and apparatus according to the present disclosure defines various inter modes for improving the efficiency of inter prediction, and provides a method of deriving motion information according to each inter mode.
[0009] The image encoding / decoding method and apparatus according to the present disclosure provide a method of adaptively limiting consumption of storage bandwidth by using a plurality of motion vectors.
[0010] The image encoding / decoding method and apparatus according to the present disclosure provide a hierarchical division method of a picture.
[0011] Technical Solution
[0012] The image encoding / decoding method and apparatus according to the present disclosure may derive motion information for bidirectional prediction of a current block based on an inter mode predefined in an image decoding apparatus, and perform inter prediction on the current block based on the motion information.
[0013] In the image encoding / decoding method and apparatus according to the present disclosure, motion information for bidirectional prediction may be adjusted to motion information for unidirectional prediction according to a predefined inter mode.
[0014] In the image encoding / decoding method and device according to the contents of the present disclosure, when the predefined inter-frame mode is a merge mode, the motion information used for bidirectional prediction can be adjusted to the motion information for unidirectional prediction, and when the predefined inter-frame mode is an affine mode, the motion information used for bidirectional prediction may not be adjusted to the motion information for unidirectional prediction.
[0015] In the image encoding / decoding method and apparatus according to the present disclosure, when the predefined inter mode is the merge mode, motion information for bidirectional prediction is adjusted to motion information for unidirectional prediction in consideration of the size of the current block.
[0016] In the image encoding / decoding method and apparatus according to the present disclosure, when the predefined inter mode is a merge mode, deriving motion information may include: constructing a merge candidate list of the current block, and deriving motion information of the current block from the merge candidate list.
[0017] In the image encoding / decoding method and apparatus according to the present disclosure, the merge candidate list may include at least one of a spatial merge candidate, a temporal merge candidate, or a combined merge candidate.
[0018] In the image encoding / decoding method and device according to the contents of the present disclosure, when the predefined inter-frame mode is an affine mode, deriving motion information may include: generating a candidate list consisting of affine candidates of the current block; deriving a control point vector of the current block based on the candidate list and the candidate index; and deriving a motion vector of the current block based on the control point vector of the current block.
[0019] In the image encoding / decoding method and apparatus according to the present disclosure, the affine candidate may include at least one of a spatial candidate, a temporal candidate, or a configuration candidate.
[0020] In an image encoding / decoding method and apparatus according to the present disclosure, a motion vector of a current block is derived taking into account the size of a predetermined reference area, and the reference area includes a reference block specified by motion vectors of n sub-blocks and pixels for interpolating fractional pixels of the reference block.
[0021] In the image encoding / decoding method and apparatus according to the present disclosure, when the size of the reference area is less than or equal to a threshold, a motion vector of the current block can be derived for each sub-block, and when the size of the reference area is greater than the threshold, a motion vector can be derived for the current block.
[0022] In the image encoding / decoding method and apparatus according to the present disclosure, the motion vector of the current block may be derived based on the affine motion parameters, and the affine motion parameters may be calculated based on the control point vector of the current block.
[0023] In the image encoding / decoding method and apparatus according to the present disclosure, the current block may belong to a predetermined sub-picture, and the sub-picture may be defined as a group of grids having the same index among a plurality of grids constituting one picture.
[0024] Beneficial Effects
[0025] The present disclosure may allow bidirectional prediction only when a predetermined condition is met, or may reduce the consumption of storage bandwidth by adjusting motion vectors.
[0026] According to the present disclosure, the efficiency of inter prediction may be improved by using a combined merging candidate in addition to a spatial merging candidate / a temporal merging candidate.
[0027] According to the present disclosure, the efficiency of image encoding / decoding can be improved through inter-frame prediction based on an affine motion model.
[0028] According to the present disclosure, the efficiency of image encoding / decoding may be improved by dividing a picture into multi-level segment regions. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1is a block diagram illustrating an image encoding apparatus according to the present disclosure.
[0030] Figure 2 is a block diagram illustrating an image decoding apparatus according to the present disclosure.
[0031] Figure 3 A restricted bidirectional prediction process is shown as an embodiment to which the present disclosure is applied.
[0032] Figure 4 A concept of performing inter-frame prediction using affine transformation prediction in a video encoding method and apparatus according to an embodiment of the present disclosure is shown.
[0033] Figure 5 The concept of affine transformation prediction in sub-block units according to an embodiment of the present disclosure is shown.
[0034] Figure 6 An inter-frame prediction method based on an affine mode is shown as an embodiment to which the present disclosure is applied.
[0035] Figure 7 A concept of limiting motion vectors in consideration of memory bandwidth is shown as an embodiment to which the present disclosure is applied.
[0036] Figure 8 An inter-frame prediction method using mask-based prediction in a video encoding method and apparatus according to an embodiment of the present disclosure is shown.
[0037] Fig. 9 The concept of bidirectional prediction in a video encoding and / or decoding method and apparatus according to an embodiment of the present disclosure is shown.
[0038] Fig.10 A concept of restricted bidirectional prediction is shown as an embodiment to which the present disclosure is applied.
[0039] Fig.11 is a flowchart of a process of decoding prediction directionality according to an embodiment of the present disclosure.
[0040] Fig.12 is a flowchart for determining whether prediction directionality is limited according to an embodiment of the present disclosure.
[0041] Fig.13 is a flowchart for determining whether prediction directionality is limited according to another embodiment of the present disclosure.
[0042] Fig.14 A concept of dividing one picture using units defined as a plurality of layers according to an embodiment of the present disclosure is shown.
[0043] Fig.15 A concept of dividing an image, an image group, or a video into a plurality of tiles and processing the plurality of tiles into three tile groups according to an embodiment of the present disclosure is shown.
[0044] Fig.16 A concept of dividing an image, an image group, or an image into a plurality of tiles and sharing and restoring only a reconstructed image corresponding to each tile according to an embodiment of the present disclosure is shown.
[0045] Fig.17 An example is shown in which an image, an image group, or a video is divided into a plurality of tiles and each tile is divided into sub-tiles according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0046] The image encoding / decoding method and apparatus according to the present disclosure may derive motion information for bidirectional prediction of a current block based on an inter mode predefined in an image decoding apparatus, and perform inter prediction on the current block based on the motion information.
[0047] In the image encoding / decoding method and apparatus according to the present disclosure, motion information for bidirectional prediction may be adjusted to motion information for unidirectional prediction according to a predefined inter mode.
[0048] In the image encoding / decoding method and device according to the contents of the present disclosure, when the predefined inter-frame mode is a merge mode, the motion information used for bidirectional prediction can be adjusted to the motion information for unidirectional prediction, and when the predefined inter-frame mode is an affine mode, the motion information used for bidirectional prediction may not be adjusted to the motion information for unidirectional prediction.
[0049] In the image encoding / decoding method and apparatus according to the present disclosure, when the predefined inter mode is the merge mode, motion information for bidirectional prediction may be adjusted to motion information for unidirectional prediction in consideration of the size of the current block.
[0050] In the image encoding / decoding method and apparatus according to the present disclosure, when the predefined inter mode is a merge mode, deriving motion information may include constructing a merge candidate list of the current block, and deriving motion information of the current block according to the merge candidate list.
[0051] In the image encoding / decoding method and apparatus according to the present disclosure, the merge candidate list may include at least one of a spatial merge candidate, a temporal merge candidate, or a combined merge candidate.
[0052] In the image encoding / decoding method and device according to the contents of the present disclosure, when the predefined inter-frame mode is an affine mode, deriving the motion information may include: generating a candidate list consisting of affine candidates of the current block; deriving a control point vector of the current block based on the candidate list and the candidate index; and deriving the motion vector of the current block based on the control point vector of the current block.
[0053] In the image encoding / decoding method and apparatus according to the present disclosure, the affine candidate may include at least one of a spatial candidate, a temporal candidate, or a configuration candidate.
[0054] In an image encoding / decoding method and apparatus according to the present disclosure, a motion vector of a current block is derived taking into account the size of a predetermined reference area, and the reference area includes a reference block specified by motion vectors of n sub-blocks and pixels for interpolating fractional pixels of the reference block.
[0055] In the image encoding / decoding method and apparatus according to the present disclosure, when the size of the reference area is less than or equal to a threshold, a motion vector of the current block can be derived for each sub-block, and when the size of the reference area is greater than the threshold, a motion vector can be derived for the current block.
[0056] In the image encoding / decoding method and apparatus according to the present disclosure, the motion vector of the current block may be derived based on the affine motion parameters, and the affine motion parameters may be calculated based on the control point vector of the current block.
[0057] In the image encoding / decoding method and apparatus according to the present disclosure, the current block may belong to a predetermined sub-picture, and the sub-picture may be defined as a group of grids having the same index among a plurality of grids constituting one picture.
[0058] Implementation
[0059] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings in this specification so that those skilled in the art can easily implement the present disclosure. However, the present disclosure can be implemented in various different forms and is not limited to the embodiments described herein. In the accompanying drawings, in order to clearly describe the present disclosure, parts that are not related to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.
[0060] Throughout the specification, when a part is described as being "connected" to another part, this includes not only a direct connection but also an electrical connection with another element in between. In addition, throughout the specification, when a part "includes" a certain component, this means that other components may also be included rather than excluded unless otherwise specified.
[0061] The terms "(to) a step of ~" or "a step of ~" used throughout this specification do not mean "a step for ~". In addition, terms such as first and second can be used to describe various elements, but the elements should not be limited by these terms. The above terms are only used for the purpose of distinguishing one component from another.
[0062] In addition, the components shown in the embodiments of the present disclosure are independently shown to represent different characteristic functions, which does not mean that each component is made of separate hardware or single software component units. That is, for ease of description, each component unit is described by being listed as a respective component unit, and at least two of the component units are combined to form a component unit, or a component unit can be divided into multiple component units to perform functions. As long as the integrated implementation and separate implementation of each of these components do not deviate from the essence of the present disclosure, the integrated implementation and separate implementation of each of these components are also included in the scope of the present disclosure.
[0063] In various embodiments of the present disclosure described below, terms such as "~unit", "~group", "~unit", "~module" and "~block" mean a unit that processes at least one function or operation, and these terms can be implemented in hardware or software, or a combination of hardware and software.
[0064] In addition, a coding block refers to a processing unit of a group of target pixels on which encoding and decoding are currently performed, and may be used interchangeably as a coding block and a coding unit. In addition, a coding unit refers to a coding unit (CU), and may generally mean including a coding block (CB).
[0065] In addition, quadtree partitioning refers to partitioning a block into four independent coding units, and binary tree partitioning refers to partitioning a block into two independent coding units. In addition, ternary tree partitioning refers to partitioning a block into three independent coding units at a ratio of 1:2:1.
[0066] Figure 1 is a block diagram illustrating an image encoding apparatus according to the present disclosure.
[0067] Reference Figure 1 The video encoding device 100 may include: a picture division module 110, prediction modules 120 and 125, a transformation module 130, a quantization module 135, a rearrangement module 160, an entropy encoding module 165, an inverse quantization module 140, an inverse transformation module 145, a filter module 150 and a memory 155.
[0068] The picture division module 110 may divide the input picture into one or more processing units. Herein, a processing unit may be a prediction unit (PU), a transform unit (TU), or a coding unit (CU). Hereinafter, in an embodiment of the present disclosure, a coding unit may be used as a unit for performing encoding or a unit for performing decoding.
[0069] The prediction unit may be obtained by dividing one coding unit into at least one square or non-square of the same size, and may be divided so that one prediction unit among a plurality of prediction units divided within one coding unit has a shape and / or size different from another prediction unit. When the prediction unit is not the minimum coding unit when generating a prediction unit for performing intra prediction based on the coding unit, intra prediction may be performed without dividing the coding unit into a plurality of prediction units N×N.
[0070] The prediction modules 120 and 125 may include an inter prediction module 120 that performs inter prediction and an intra prediction module 125 that performs intra prediction. Whether inter prediction or intra prediction is performed for a prediction unit may be determined, and detailed information (e.g., intra prediction mode, motion vector, reference picture, etc.) according to each prediction method may be determined. The residual value (residual block) between the generated prediction block and the original block may be input to the transform module 130. In addition, prediction mode information, motion vector information, etc. used for prediction may be encoded together with the residual value by the entropy encoding module 165 and may be sent to the decoder. However, when the motion information derivation technique from the decoder side according to the present disclosure is applied, since the encoder does not generate prediction mode information and motion vector information, the corresponding information is not sent to the decoder. On the other hand, the encoder may signal and send information indicating that motion information is derived and used from the decoder side and information about the technique for deriving motion information.
[0071] The inter-frame prediction module 120 may predict the prediction unit based on information of at least one of the previous picture or the subsequent picture of the current picture, or in some cases may predict the prediction unit based on information of some coding regions in the current picture. The inter-frame prediction module 120 may include a reference picture interpolation module, a motion prediction module, and a motion compensation module.
[0072] The reference picture interpolation module may receive reference picture information from the memory 155, and may generate pixel information about integer pixels or less than integer pixels according to the reference picture. In the case of luma pixels, an 8-tap DCT-based interpolation filter with different filter coefficients may be used to generate pixel information about integer pixels or less than integer pixels in units of 1 / 4 pixels. In the case of chroma signals, a 4-tap DCT-based interpolation filter with different filter coefficients may be used to generate pixel information about integer pixels or less than integer pixels in units of 1 / 8 pixels.
[0073] The motion prediction module can perform motion prediction based on the reference picture interpolated by the reference picture interpolation module. As a method for obtaining a motion vector, various methods such as a full search based block matching algorithm (FBMA), a three-step search (TSS), and a new three-step search algorithm (NTS) can be used. The motion vector may have a motion vector value in units of 1 / 2 pixel or 1 / 4 pixel based on the interpolated pixel. The motion prediction module can predict the current prediction unit by using various motion prediction methods. As a motion prediction method, various methods such as a skip method, a merge method, an advanced motion vector prediction (AMVP) method, and an intra-block copy method can be used. In addition, when the motion information derivation technology performed from the decoder side according to the contents of the present disclosure is applied, as a method performed by the motion prediction module, a template matching method and a bilateral matching method using a motion trajectory can be applied. In this regard, it will be described later. Figure 3 The template matching method and the bilateral matching method are described in detail.
[0074] The intra prediction module 125 may generate a prediction unit based on reference pixel information adjacent to the current block as pixel information in the current picture. When the adjacent block of the current prediction unit is a block on which inter prediction has been performed and the reference pixel is a pixel on which inter prediction has been performed, the reference pixel included in the block on which inter prediction has been performed may be replaced with reference pixel information of the adjacent block on which intra prediction has been performed. In other words, when a reference pixel is unavailable, information about the unavailable reference pixel may be replaced with at least one reference pixel among the available reference pixels.
[0075] In addition, a residual block including residual information which is a difference between a prediction unit on which prediction has been performed based on the prediction unit generated by the prediction modules 120 and 125 and an original block of the prediction unit may be generated. The generated residual block may be input to the transform module 130.
[0076] The transform module 130 may use a transform method such as discrete cosine transform (DCT), discrete sine transform (DST), and KLT to transform a residual block including residual information between an original block and a prediction unit generated by the prediction modules 120 and 125. Whether to apply DCT, DST, or KLT in order to transform the residual block may be determined based on intra prediction mode information of a prediction unit used to generate the residual block.
[0077] The quantization module 135 may quantize the value transformed into the frequency domain by the transform module 130. The quantization coefficient may vary according to the block or importance of the picture. The value calculated by the quantization module 135 may be provided to the inverse quantization module 140 and the rearrangement module 160.
[0078] The rearrangement module 160 may rearrange the coefficient values on the quantized residual values.
[0079] The rearrangement module 160 can change the coefficient of the two-dimensional block form into the coefficient of the one-dimensional vector form by the coefficient scanning method. For example, the rearrangement module 160 can use the zigzag scanning method to scan from the DC coefficient to the coefficient in the high frequency domain so that the coefficient is changed into the form of a one-dimensional vector. According to the size of the transform unit and the intra-frame prediction mode, the zigzag scanning can be replaced by the vertical scanning of the coefficient of the two-dimensional block form in the column direction or the horizontal scanning of the coefficient of the two-dimensional block form in the row direction. In other words, it can be determined according to the size of the transform unit and the intra-frame prediction mode which scanning method among the zigzag scanning, vertical scanning and horizontal scanning is used.
[0080] The entropy encoding module 165 may perform entropy encoding based on the value calculated by the rearrangement module 160. The entropy encoding may use various encoding methods such as exponential Golomb, context adaptive variable length coding (CAVLC), and context adaptive binary arithmetic coding (CABAC). In this regard, the entropy encoding module 165 may encode residual value coefficient information of the coding units from the rearrangement module 160 and the prediction modules 120 and 125. In addition, according to the present disclosure, information indicating that motion information is derived and used at the decoder side and information about a technique for deriving motion information may be signaled and transmitted.
[0081] The inverse quantization module 140 and the inverse transform module 145 may inversely quantize the value quantized by the quantization module 135, and inversely transform the value transformed by the transform module 130. The residual values generated by the inverse quantization module 140 and the inverse transform module 145 may be combined with the prediction units predicted by the motion prediction module, the motion compensation module, and the intra prediction module included in the prediction modules 120 and 125 to generate a reconstructed block.
[0082] The filter module 150 may include at least one of a deblocking filter, an offset correction module, or an adaptive loop filter (ALF). The deblocking filter may remove block distortions that occur due to boundaries between blocks in a reconstructed picture. The offset correction module may correct the offset relative to the original image in units of pixels in the image after deblocking filtering. In order to perform offset correction on a specific picture, a method of applying an offset in consideration of edge information of each pixel may be used or a method of dividing the pixels included in the image into a predetermined number of regions, determining the region to be offset, and applying the offset to the determined region may be used. Adaptive loop filtering (ALF) may be performed based on a value obtained by comparing the filtered reconstructed image with the original image. After the pixels included in the image are divided into predetermined groups, a filter to be applied to the corresponding group may be determined, and filtering may be performed differently for each group.
[0083] The memory 155 may store the reconstructed block or picture calculated by the filter module 150. When inter prediction is performed, the stored reconstructed block or picture may be provided to the prediction modules 120 and 125.
[0084] Figure 2 is a block diagram illustrating an image decoding apparatus according to the present disclosure.
[0085] Reference Figure 2 , the apparatus 200 for decoding a video may include: an entropy decoding module 210 , a rearrangement module 215 , an inverse quantization module 220 , an inverse transform module 225 , prediction modules 230 and 235 , a filter module 240 and a memory 245 .
[0086] When a video bitstream is input to an apparatus for decoding a video, the input bitstream may be decoded according to an inverse process of the apparatus for encoding a video.
[0087] The entropy decoding module 210 may perform entropy decoding according to the inverse process of entropy encoding performed by the entropy encoding module of the video encoding device. For example, corresponding to the method performed by the video encoding device, various methods such as exponential Golomb, context adaptive variable length coding (CAVLC) and context adaptive binary arithmetic coding (CABAC) may be applied.
[0088] The entropy decoding module 210 may decode information about intra prediction and inter prediction performed by the encoding device.
[0089] The rearrangement module 215 may perform rearrangement on the bitstream entropy decoded by the entropy decoding module 210 based on the rearrangement method used in the encoding apparatus. The rearrangement module may reconstruct and rearrange coefficients in the form of a one-dimensional vector into coefficients in the form of a two-dimensional block.
[0090] The inverse quantization module 220 may perform inverse quantization based on the quantization parameter received from the encoding device and the rearranged coefficient of the block.
[0091] The inverse transform module 225 may perform inverse transform on the quantization result generated by the apparatus for encoding the video, i.e., inverse DCT, inverse DST, and inverse KLT corresponding to the transforms performed by the transform module, i.e., DCT, DST, and KLT. The inverse transform may be performed based on the transmission unit determined by the video encoding apparatus. In the inverse transform module 225 of the video decoding apparatus, a transform scheme (e.g., DCT, DST, and KLT) may be selectively performed according to multiple pieces of information such as a prediction method, a size of a current block, and a prediction direction.
[0092] The prediction modules 230 and 235 may generate a prediction block based on the information on prediction block generation received from the entropy decoding module 210 and the information on a previously decoded block or picture received from the memory 245 .
[0093] As described above, if the size of the prediction unit and the size of the transform unit are the same when intra prediction is performed in the same manner as the operation of the video encoding device, intra prediction can be performed on the prediction unit based on pixels existing in the left, upper left, and top of the prediction unit. However, if the size of the prediction unit and the size of the transform unit are different when intra prediction is performed, intra prediction can be performed using reference pixels based on the transform unit. In addition, intra prediction using N×N partitioning can be used only for the minimum coding unit.
[0094] The prediction modules 230 and 235 may include a prediction unit determination module, an inter prediction module, and an intra prediction module. The prediction unit determination module may receive various information such as prediction unit information, prediction mode information of an intra prediction method, and information about motion prediction of an inter prediction method from the entropy decoding module 210, may divide the current coding unit into prediction units, and may determine whether to perform inter prediction or intra prediction on the prediction unit. On the other hand, if the encoder 100 does not send information related to motion prediction for inter prediction, but sends information indicating that motion information is derived and used from the decoder side and information about a technique for deriving motion information, the prediction unit determination module determines the prediction performance of the inter prediction module 230 based on the information sent from the encoder 100.
[0095] The inter-frame prediction module 230 can use the information required for the inter-frame prediction of the current prediction unit provided by the video encoding device, based on information of at least one of the previous picture or the subsequent picture of the current picture including the current prediction unit, to perform inter-frame prediction on the current prediction unit. In order to perform inter-frame prediction, it can be determined based on the coding block whether the motion prediction method of the prediction unit included in the corresponding coding unit is a skip mode, a merge mode, an AMVP mode, or an intra-block copy mode. Alternatively, the inter-frame prediction module 230 itself can derive motion information from the following information and then perform inter-frame prediction: information indicating that motion information is derived and used from the decoder side and information about a technique for deriving motion information provided by the video encoder.
[0096] The intra prediction module 235 can generate a prediction block based on pixel information in the current picture. When the prediction unit is a prediction unit that performs intra prediction, intra prediction can be performed based on the intra prediction mode information of the prediction unit received from the video encoding device. The intra prediction module 235 may include an adaptive intra smoothing (AIS) filter, a reference pixel interpolation module, and a DC filter. The AIS filter performs filtering on the reference pixels of the current block, and it can be determined whether to apply the filter according to the prediction mode of the current prediction unit. AIS filtering can be performed on the reference pixels of the current block by using the prediction mode of the prediction unit and the AIS filter information received from the device for encoding the video. When the prediction mode of the current block is a mode in which AIS filtering is not performed, the AIS filter may not be applied.
[0097] When the prediction mode of the prediction unit is a prediction unit that performs intra-frame prediction based on a pixel value interpolated by a reference pixel, the reference pixel interpolation module may interpolate the reference pixel to generate a reference pixel in units of pixels equal to or less than an integer pixel. When the prediction mode of the current prediction unit is a prediction mode in which a prediction block is generated without interpolating the reference pixel, the reference pixel may not be interpolated. When the prediction mode of the current block is a DC mode, a DC filter may generate a prediction block by filtering.
[0098] The reconstructed block or picture may be provided to the filter module 240. The filter module 240 may include a deblocking filter, an offset correction module, and an ALF.
[0099] Information about whether to apply a deblocking filter to a corresponding block or picture and information about which of a strong filter and a weak filter to apply when applying the deblocking filter can be received from a video encoding device. The deblocking filter of a video decoding device can receive information about the deblocking filter from a video encoding device and can perform deblocking filtering on the corresponding block.
[0100] The offset correction module may perform offset correction on the reconstructed image based on the type of offset correction and the offset value information applied to the image when performing encoding. Based on information received from the encoding device about whether ALF is applied, ALF coefficient information, etc., ALF may be applied to the coding unit. ALF information may be provided in a manner included in a specific parameter set.
[0101] The memory 245 may store the reconstructed picture or block to be used as a reference picture or block, and may provide the reconstructed picture to the output module.
[0102] Figure 3 A restricted bidirectional prediction process is shown as an embodiment to which the present disclosure is applied.
[0103] Reference Figure 3 , the motion information of the current block may be derived based on the inter-frame mode predefined in the encoding / decoding apparatus (S300).
[0104] The predefined inter-frame mode according to the present disclosure may include at least one of a merge mode, an AMVP mode, an affine mode, or an intra-block copy mode. Here, the motion information may be interpreted differently as a motion vector predictor, a motion vector, a control point vector predictor, a control point vector, a block vector, etc. according to the inter-frame mode.
[0105] 1. Merge Mode
[0106] In the case of merge mode, the motion information of the current block can be set to be the same as the motion information of the merge candidate. In this way, the motion information is derived by merging with the merge candidate, and no separate motion vector difference (mvd) is signaled. Hereinafter, the method of deriving motion information based on merge mode will be described in detail.
[0107] First, a merge candidate list of a current block may be configured (S1). The merge candidate list may include at least one of a spatial merge candidate or a temporal merge candidate of the current block.
[0108] The motion information of the spatial merge candidate can be obtained from the motion information of the spatial neighboring blocks of the current block. Here, the spatial neighboring block is a block belonging to the same picture as the current block, and may mean a block adjacent to the current block. The spatial neighboring block may include a block adjacent to at least one of the left, top, top right, bottom left, or top left of the current block. The top left neighboring block can be used only when at least one of the blocks adjacent to the left, top, top right, and bottom left of the current block is not available.
[0109] The motion information of the temporal merge candidate can be obtained from the motion information of the temporal neighboring blocks of the current block. The temporal neighboring block may be a block belonging to a different picture from the current block, and may be defined as a block at the same position as the current block. Here, the block at the same position may mean at least one of the following blocks: a block (BR) adjacent to the lower right corner of the current block, a block (CTR) including the position of the center sample of the current block, or a block (TL) including the position of the upper left sample of the current block. Alternatively, the block at the same position may mean a block including a position where a predetermined disparity vector is shifted from the position of the upper left sample of the current block. Here, the disparity vector may be determined based on any one of the motion vectors of the above-mentioned spatial neighboring blocks. Alternatively, the disparity vector may be determined based on a combination of at least two of the motion vectors of the above-mentioned spatial neighboring blocks. The combination may mean operations such as maximum, minimum, median, and weighted average. For example, the disparity vector may be set to the motion vector of the left neighboring block or the upper neighboring block. Alternatively, the disparity vector may be set to the median or average between the motion vector of the left neighboring block and the motion vector of the lower left neighboring block.
[0110] The motion vector and reference picture index of the temporal merge candidate can be obtained from the motion vector and reference picture index of the temporal neighboring block, respectively. Alternatively, the motion vector of the temporal merge candidate can be derived as the motion vector of the temporal neighboring block, and the reference picture index of the temporal merge candidate can be set to a default value (e.g., 0) predetermined to the decoding device regardless of the temporal neighboring block.
[0111] The merge candidate list may also include a combined merge candidate. The combined merge candidate may be derived by combining n merge candidates belonging to the pre-generated merge candidate list.
[0112] Here, n may be an integer of 2, 3, 4 or more. The number n of merge candidates to be combined may be a fixed value predetermined to the encoding / decoding device, or may be encoded and signaled by the encoding device. Signaling may be performed in at least one unit of a sequence, a picture, a slice, a tile, a sub-tile (brick), or a predetermined block. The number n of merge candidates to be combined may be variably determined based on the number of remaining merge candidates. Here, the number of remaining merge candidates may mean the difference between the maximum number of merge candidates that can be included in the merge candidate list and the current number of merge candidates in the merge candidate list. The maximum number may be a number predetermined to the encoding / decoding device, or may be encoded and signaled by the encoding device. The current number may mean the number of merge candidates configured before adding the combined merge candidate. For example, when the number of remaining merge candidates is 1, two merge candidates may be used, and when the number of remaining merge candidates is greater than 1, three or more merge candidates may be used.
[0113] The positions of the n merge candidates may be predetermined positions in the merge candidate list. For example, an index (0 to (k-1)) may be assigned to each merge candidate belonging to the merge candidate list. Here, k may mean the total number of merge candidates included in the merge candidate list. In this case, the positions of the n merge candidates may correspond to index 0 to index (n-1) in the merge candidate list. Alternatively, the n merge candidates may be determined by considering the prediction direction of each merge candidate included in the merge candidate list. For example, among the merge candidates belonging to the merge candidate list, only merge candidates whose prediction direction is bidirectional prediction may be selectively used, or only merge candidates whose prediction direction is unidirectional prediction may be selectively used.
[0114] The combined merge candidate may be derived using both the spatial merge candidate and the temporal merge candidate, or may be derived using only one of the spatial merge candidate or the temporal merge candidate. For example, the combined merge candidate may be limited to be derived using only the spatial merge candidate. In this case, the number of merge candidates to be combined may be limited to the number of spatial merge candidates belonging to the pre-generated merge candidate list.
[0115] The combined merge candidate may be added after the spatial / temporal merge candidate in the merge candidate list. That is, the index of the combined merge candidate may be greater than the index of the spatial / temporal merge candidate. Alternatively, the combined merge candidate may be added between the spatial merge candidate and the temporal merge candidate in the merge candidate list. That is, the index of the combined merge candidate may be greater than the index of the spatial merge candidate and less than the index of the temporal merge candidate. Alternatively, the position of the combined merge candidate may be variably determined in consideration of the prediction direction of the combined merge candidate. Depending on whether the prediction direction of the combined merge candidate is bidirectional prediction, the position of the combined merge candidate in the merge candidate list may be rearranged. For example, when the prediction direction of the combined merge candidate is bidirectional prediction, an index less than the spatial merge candidate or the temporal merge candidate may be assigned, otherwise, an index greater than the spatial merge candidate or the temporal merge candidate may be assigned.
[0116] Hereinafter, for the convenience of description, a method of obtaining a combined merge candidate based on two merge candidates will be described.
[0117] The motion information of the combined merge candidate can be obtained by weighted averaging the motion information of the first merge candidate and the second merge candidate. Here, the weights of the weighted average are [1:1], [1:2], [1:3], [2:3], etc., but are not limited thereto. The weights can be predefined in the encoding / decoding device or can be obtained from the decoding device. In this case, the weights can be obtained by considering at least one of the distance between the reference picture of the merge candidate and the current picture or the prediction direction of the merge candidate. Alternatively, the motion information of the combined merge candidate can be derived by obtaining the motion information in the L0 direction from the first merge candidate and the motion information in the L1 direction from the second merge candidate and combining them. The motion information of the combined merge candidate can be derived based on at least one of the above-mentioned derivation methods, and this can be performed by considering the prediction direction of the merge candidates to be combined as described later.
[0118] In the present specification, the motion information may include at least one of a prediction direction flag, a reference picture index, or a motion vector. The motion information may be defined for L0 prediction and L1 prediction, respectively. Here, L0 prediction may mean prediction with reference to reference picture list L0, and L1 prediction may mean prediction with reference to reference picture list L1.
[0119] (1) When the prediction directions of the first merge candidate and the second merge candidate are both unidirectional predictions
[0120] [Case 1] When the first merge candidate is L0 prediction and the second merge candidate is L1 prediction, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the first merge candidate. The prediction direction flag of the combined merge candidate in the L0 direction may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the motion vector of the first merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be derived as the reference picture index of the second merge candidate. The prediction direction flag of the combined merge candidate in the L1 direction may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the motion vector of the second merge candidate.
[0121] [Case 2] When the first merge candidate is L1 predicted and the second merge candidate is L0 predicted, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the second merge candidate. The prediction direction flag of the combined merge candidate in the L0 direction may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the motion vector of the second merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be derived as the reference picture index of the first merge candidate. The prediction direction flag of the combined merge candidate in the L1 direction may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be taken as the motion vector of the first merge candidate.
[0122] [Case 3] When the first merge candidate and the second merge candidate are L0 predictions, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index of the combined merge candidate in the L0 direction. The prediction direction flag of the combined merge candidate in the L0 direction may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the weighted average of the motion vector of the first merge candidate and the motion vector of the second merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be derived as -1, the prediction direction flag in the L1 direction may be derived as 0, and the motion information in the L1 direction may be derived as 0.
[0123] [Case 4] When the first merge candidate and the second merge candidate are L1 predictions, the reference picture index of the combined merge candidate in the L0 direction may be derived as -1, the prediction direction flag in the L0 direction may be derived as 0, and the motion information in the L0 direction may be derived as 0. The reference picture index of the combined merge candidate in the L1 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index of the combined merge candidate in the L1 direction. The prediction direction flag of the combined merge candidate in the L1 direction may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the weighted average of the motion vector of the first merge candidate and the motion vector of the second merge candidate.
[0124] (2) When the prediction directions of the first merge candidate and the second merge candidate are both bidirectional predictions,
[0125] [Case 5] The reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index of the combined merge candidate in the L0 direction. The prediction direction flag of the combined merge candidate in the L0 direction may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the weighted average of the motion vector of the first merge candidate and the motion vector of the second merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index of the combined merge candidate in the L1 direction. The prediction direction flag of the combined merge candidate in the L1 direction may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the weighted average of the motion vector of the first merge candidate and the motion vector of the second merge candidate.
[0126] (3) When the prediction direction of the first merge candidate is bidirectional prediction and the prediction direction of the second merge candidate is unidirectional prediction,
[0127] [Case 6] When the second merge candidate is an L0 prediction, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index of the combined merge candidate in the L0 direction. The prediction direction mark of the combined merge candidate in the L0 direction may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the weighted average of the motion vector of the first merge candidate and the motion vector of the second merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be derived as the reference picture index of the first merge candidate. The prediction direction mark of the combined merge candidate in the L1 direction may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the motion vector of the first merge candidate.
[0128] [Case 7] When the second merge candidate is an L1 prediction, the reference picture index of the combined merge candidate in the L0 direction may be derived as the reference picture index of the first merge candidate. The prediction direction flag of the combined merge candidate in the L0 direction may be derived as 1. The motion vector of the combined merge candidate in the L0 direction may be derived as the motion vector of the first merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be derived as the reference picture index of the first merge candidate or the second merge candidate. For example, the reference picture index of the merge candidate with the smallest index among the first merge candidate and the second merge candidate may be set as the reference picture index of the combined merge candidate in the L1 direction. The prediction direction flag of the combined merge candidate in the L1 direction may be derived as 1. The motion vector of the combined merge candidate in the L1 direction may be derived as the weighted average of the motion vector of the first merge candidate and the motion vector of the second merge candidate.
[0129] The motion information of the current block may be derived from the merge candidate list (S2).
[0130] Specifically, the merge index of the current block may be signaled. The merge index may specify any one of a plurality of merge candidates belonging to the merge candidate list. A merge candidate having the same index as the merge index may be extracted, and the motion information of the current block may be derived using the motion information of the extracted merge candidate. For example, the motion vector, reference picture index, and prediction direction information of the current block may be set to be the same as the motion vector, reference picture index, and prediction direction information of the extracted merge candidate.
[0131] 2. AMVP Model
[0132] In the case of AMVP mode, the motion vector of the neighboring block can be set as the motion vector prediction value of the current block. To this end, a candidate list consisting of motion vectors of spatial / temporal neighboring blocks can be constructed, and the index of any one of the multiple motion vectors of the specified candidate list can be notified by signaling. At the same time, unlike the merge mode, the motion vector can be reconstructed by adding the motion vector prediction value to the motion vector difference notified by signaling.
[0133] 3. Affine mode
[0134] In addition to the translation motion model that only considers parallel movement, another motion model can also be used. For example, a motion model that considers motions such as rotation, perspective, zooming in / out, and parallel movement can be used. This will be called an affine mode. In the affine mode, motion information can be derived in units of predetermined sub-blocks based on the control point vector of the current block. Therefore, the affine mode can be called an inter-frame mode in units of sub-blocks or a merge mode in units of sub-blocks. Figures 4 to 7 This is described in detail.
[0135] 4. In-Block Copy (IBC) Mode
[0136] In the case of the IBC mode, similar to the AMVP mode, the motion vector of the adjacent block is set as the block vector prediction value (bvp) of the current block, and the block vector is reconstructed using the block vector difference (bvd) notified by signaling. However, the difference is that the IBC mode performs motion compensation based on a pre-recovered area in the same picture as the current block, while the AMVP mode performs motion compensation based on a pre-recovered area in a picture different from the current block.
[0137] Reference Figure 3 , according to predetermined conditions, restricted bidirectional prediction can be performed on the current block (S310).
[0138] The predetermined condition may include at least one of a condition for a block size or a condition as to whether encoding is performed in an inter-frame mode in units of subblocks. According to the predetermined condition, the motion information for bidirectional prediction may be adjusted to motion information for unidirectional prediction. This assumes that the motion information obtained in S300 is obtained for L0 prediction and L1 prediction, respectively, and reference is made to Figures 9 to 13 Such restricted bidirectional prediction is described in detail.
[0139] Figure 4 A concept of performing inter-frame prediction using affine transformation prediction in a video encoding method and apparatus according to an embodiment of the present disclosure is shown.
[0140] Affine transformation prediction means a method of performing prediction in an arbitrary shape by generating a motion vector corresponding to a pixel of a reference picture using an affine transformation of a pixel in a current block. However, affine transformation prediction is not limited to prediction in units of pixels, but also refers to a method of performing prediction by dividing a current block into a plurality of sub-blocks and obtaining a motion vector in units of the included sub-blocks.
[0141] In the motion prediction method based on affine transformation prediction, when prediction is performed on the current block 201 in the current picture 200 using the reference block in the reference picture 210, two or more positions representing the block are selected as control points, and rotation and motion prediction of an arbitrary shape are performed using two or more control motion vectors 230 and 231 for the control points. In this case, the motion prediction method includes a method of performing pixel-by-pixel motion prediction by calculating a motion vector for each pixel in the current block 201 using a plurality of control motion vectors, or a method of performing sub-block unit motion prediction by dividing the current block 201 into two or more sub-blocks and calculating a motion vector in units of sub-blocks.
[0142] Figure 5The concept of affine transformation prediction in sub-block units according to an embodiment of the present disclosure is shown.
[0143] Figure 3 300 is a diagram showing an embodiment in which one block is divided into two or more sub-blocks and inter-frame prediction is performed using each motion vector in units of sub-blocks. According to an embodiment of the present disclosure, a method of selecting two or more control points representing the current block 300 and using motion vectors corresponding to the control points to transform and obtain a motion vector for each of the two or more sub-blocks that divide the current block 300. In this case, the meaning of the transformed motion vector can also be used as the meaning of the calculated motion vector.
[0144] According to an embodiment of the present disclosure, it includes a method of selecting two or more control points representing the current block 300, and transforming and obtaining each motion vector of two or more sub-blocks that divide the current block 300 by using the motion vector corresponding to the control point. The control point representing the current block 300 may be two points of the upper left position 311 and the upper right position 321, or may be three points of the upper left position 311, the upper right position 321, and the lower left position 331. In addition, the center position of the block and the lower right position of the block may be used as the control point, and two or more points may also be used as the control point.
[0145] According to an embodiment of the present disclosure, motion prediction of each sub-block is performed by calculating a motion vector of each sub-block that divides the current block 300 according to an equation having a motion vector corresponding to a control point as a parameter.
[0146] In addition, as a method of dividing the current block 300 into two or more sub-blocks, there is a method of dividing into square blocks in which N and M have predefined fixed sizes or non-square blocks in which N and M are different from each other. In addition, a method of transmitting the size of the sub-block using a high-level syntax can also be used. In addition, a method of calculating the size of the sub-block using a motion vector corresponding to a control point representing the current block 300 is also included.
[0147] Figure 6 An inter-frame prediction method based on an affine mode is shown as an embodiment to which the present disclosure is applied.
[0148] Reference Figure 6 , a candidate list for predicting motion information of the current block may be generated ( S600 ).
[0149] The candidate list may include one or more candidates based on an affine pattern (hereinafter referred to as affine candidates). An affine candidate may mean a candidate having a control point vector. A control point vector may mean a motion vector corresponding to a control point of an affine pattern, and may be defined for a corner position of a block (e.g., at least one of an upper left corner, an upper right corner, a lower left corner, or a lower right corner).
[0150] The affine candidate may include at least one of a spatial candidate, a temporal candidate, or a configuration candidate. Here, the spatial candidate may be obtained from a vector of a neighboring block that is spatially adjacent to the current block, and the temporal candidate may be obtained from a vector of a neighboring block that is temporally adjacent to the current block. Here, the neighboring block may mean a block encoded in an affine mode. The vector may mean a motion vector or a control point vector.
[0151] 1. Spatial / temporal candidate generation method
[0152] The width and height of the current block 500 are cbW and cbH, respectively, and the position of the current block is (xCb, yCb). The width and height of the spatial neighboring blocks 510 to 550 are nbW and nbH, respectively, and the position of the spatial neighboring blocks is (xNb, yNb). The spatial neighboring blocks may include at least one of the left block 510, the lower left block 540, the upper right block 430, the upper block 420, or the upper left block 450 of the current block. Alternatively, the spatial neighboring blocks may also include at least one of the blocks adjacent to the right side of the upper left block 450 or the blocks adjacent to the lower side of the upper left block 450.
[0153] The spatial candidate may have n control point vectors (cpMVs). Here, the n value may be an integer of 1, 2, 3, or more. The n value may be determined based on at least one of the following information: information on whether decoding is performed in units of subblocks; information on whether the block is encoded in an affine mode; or information on the type of affine mode (4-parameter or 6-parameter).
[0154] For example, according to the information, when the corresponding block is decoded in units of subblocks or the corresponding block is a block encoded in an affine mode, the corresponding block may have two control point vectors. On the other hand, if the corresponding block is not decoded in units of subblocks or the corresponding block is not a block encoded in an affine mode, the corresponding block may not perform prediction based on the affine mode.
[0155] Alternatively, according to the information, when the corresponding block is a block encoded in an affine mode and the type of the affine mode is 6-parameter, the corresponding block may have 3 control point vectors. On the other hand, if the corresponding block is not a block encoded in an affine mode and the type of the affine mode is not 6-parameter, the corresponding block may not perform prediction based on the affine mode.
[0156] The above information can be encoded by the encoding device and notified by signaling. Alternatively, all or part of the information can be derived from the decoding device based on the characteristics of the block. Here, the block may mean the current block or the spatial / temporal neighboring block of the current block. The characteristics may mean size, shape, position, partition type, intermediate mode, parameters related to residual coefficients, etc. The inter-frame mode is a mode predefined in the decoding device, and may mean merge mode, skip mode, AMVP mode, affine mode, intra / inter combination mode, IBC mode, etc. Alternatively, the value of n can be derived from the decoding device based on the above-mentioned block characteristics.
[0157] In this embodiment, the n control point vectors can be represented as the first control point vector (cpMV[0]), the second control point vector (cpMV[1]), the third control point vector (cpMV[2]), ..., the nth control point vector (cpMV[n-1]).
[0158] As an example, the first control point vector (cpMV[0]), the second control point vector (cpMV[1]), the third control point vector (cpMV[2]), and the fourth control point vector (cpMV[3]) may be vectors corresponding to the positions of the upper left sample, the upper right sample, the lower left sample, and the lower right sample of the block, respectively. Here, it is assumed that the spatial candidate has three control point vectors, and the three control point vectors may be any control point vectors selected from the first control point vector to the nth control point vector. However, the present disclosure is not limited thereto, and the spatial candidate may have two control point vectors, and the two control point vectors may be any control point vectors selected from the first control point vector to the nth control point vector.
[0159] The control point vectors of the spatial candidates may be derived taking into account whether the boundary of the current block touches the CTU boundary.
[0160] (1) When the boundary of the current block does not touch the CTU boundary,
[0161] The first control point vector may be derived based on at least one of the first control point vector of the spatial neighboring block, a predetermined difference, position information (xCb, yCb) of the current block, or position information (xNb, yNb) of the spatial neighboring block.
[0162] The number of difference values may be 1, 2, 3 or more. The number of difference values may be variably determined in consideration of the characteristics of the above-mentioned block, or may be a fixed value predetermined for the decoding device. The difference value may be defined as a difference between one of the plurality of control point vectors and another of the plurality of control point vectors. For example, the difference value may include a first difference value between the second control point vector and the first control point vector, a second difference value between the third control point vector and the first control point vector, a third difference value between the fourth control point vector and the third control point vector, or a fourth difference value between the fourth control point vector and the second control point vector.
[0163] For example, the first control point vector may be derived as in Equation 1 below.
[0164] [Equation 1]
[0165] cpMvLX[0][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb-yNb))
[0166] cpMvLX[0][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb-yNb))
[0167] In Equation 1, the variables mvScaleHor and mvScaleVer may refer to the first control point vector of the spatially adjacent block, or may be a value obtained by applying a shift operation of k to the first control point vector. Here, k may be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9 or more. The variables dHorX and dVerX correspond to the x component and y component of the first difference between the second control point vector and the first control point vector, respectively. The variables dHorY and dVerY correspond to the x component and y component of the second difference between the third control point vector and the first control point vector, respectively. The above variables may be derived as in Equation 2 below.
[0168] [Equation 2]
[0169] mvScaleHor=CpMvLX[xNb][yNb][0][0]<<7
[0170] mvScaleVer=CpMvLX[xNb][yNb][0][1]<<7
[0171] dHorX=(CpMvLX[xNb+nNbW-1][yNb][1][0]-CpMvLX[xNb][yNb][0][0])<<(7-log2NbW)
[0172] dVerX=(CpMvLX[xNb+nNbW-1][yNb][1][1]-CpMvLX[xNb][yNb][0][1])<<(7-log2NbW)
[0173] dHorY=(CpMvLX[xNb][yNb+nNbH-1][2][0]-CpMvLX[xNb][yNb][2][0])<<(7-log2NbH)
[0174] dVerY=(CpMvLX[xNb][yNb+nNbH-1][2][1]-CpMvLX[xNb][yNb][2][1])<<(7-log2NbH)
[0175] The second control point vector may be obtained based on at least one of the first control point vector of the spatially adjacent block, a predetermined difference, the position information (xCb, yCb) of the current block, the block size (width or height), or the position information (xNb, yNb) of the spatially adjacent block. Here, the block size may mean the size of the current block and / or the size of the spatially adjacent block. The difference is as described in the first control point vector, and its detailed description will be omitted. However, the range and / or number of the difference used in the process of obtaining the second control point vector may be different from that of the first control point vector.
[0176] For example, the second control point vector may be derived as in Equation 3 below.
[0177] [Equation 3]
[0178] cpMvLX[1][0]=(mvScaleHor+dHorX*(xCb+cbWidth-xNb)+dHorY*(yCb-yNb))
[0179] cpMvLX[1][1]=(mvScaleVer+dVerX*(xCb+cbWidth-xNb)+dVerY*(yCb-yNb))
[0180] In Equation 3, variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 1, and a detailed description thereof will be omitted.
[0181] The third control point vector may be obtained based on at least one of the first control point vector of the spatially adjacent block, a predetermined difference, position information (xCb, yCb) of the current block, a block size (width or height), or position information (xNb, yNb) of the spatially adjacent block. Here, the block size may mean the size of the current block and / or the size of the spatially adjacent block. The difference is as described in the first control point vector, and a detailed description thereof will be omitted herein. However, the range and / or number of the difference used in deriving the third control point vector may be different from that of the first control point vector or the second control point vector.
[0182] For example, the third control point vector can be obtained according to the following equation 4.
[0183] [Equation 4]
[0184] cpMvLX[2][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb+cbHeight-yNb))
[0185] cpMvLX[2][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb+cbHeight-yNb))
[0186] In Equation 4, variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 1, and a detailed description thereof will be omitted. Meanwhile, through the above process, the nth control point vector of the spatial candidate can be derived.
[0187] (2) When the boundary of the current block touches the CTU boundary,
[0188] The first control point vector may be derived based on at least one of a motion vector (MV) of a spatially neighboring block, a predetermined difference, position information (xCb, yCb) of a current block, or position information (xNb, yNb) of a spatially neighboring block.
[0189] The motion vector may be a motion vector of a subblock located at the lower portion of a spatially adjacent block. The subblock may be positioned at the leftmost, center, or rightmost of a plurality of subblocks located at the lower portion of the spatially adjacent block. Alternatively, the motion vector may mean an average value, a maximum value, or a minimum value of the motion vectors of the subblocks.
[0190] The number of difference values may be 1, 2, 3 or more. The number of difference values may be variably determined in consideration of the characteristics of the above-mentioned blocks, or may be a fixed value predetermined for the decoding device. The difference value may be defined as a difference between one of a plurality of motion vectors stored in units of subblocks in a spatially adjacent block and another of a plurality of motion vectors stored in units of subblocks in a spatially adjacent block. For example, the difference value may mean a difference between a motion vector of a lower right subblock and a motion vector of a lower left subblock of a spatially adjacent block.
[0191] For example, the first control point vector can be obtained according to the following equation 5.
[0192] [Equation 5]
[0193] cpMvLX[0][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb-yNb))
[0194] cpMvLX[0][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb-yNb))
[0195] In Equation 5, variables mvScaleHor and mvScaleVer may refer to the motion vector (MV) of the above-mentioned spatial neighboring block or a value obtained by applying a shift operation of k to the motion vector. Here, k may be an integer of 1, 2, 3, 4, 5, 6, 7, 8, 9 or more.
[0196] The variables dHorX and dVerX correspond to the x component and y component of the predetermined difference, respectively. Herein, the difference means the difference between the motion vector of the lower right sub-block and the motion vector of the lower left sub-block in the spatially adjacent block. The variables dHorY and dVerY can be derived based on the variables dHorX and dVerX. The above variables can be derived according to the following equation 6.
[0197] [Equation 6]
[0198] mvScaleHor=MvLX[xNb][yNb+nNbH-1][0]<<7
[0199] mvScaleVer=MvLX[xNb][yNb+nNbH-1][1]<<7
[0200] dHorX=(MvLX[xNb+nNbW-1][yNb+nNbH-1][0]-MvLX[xNb][yNb+nNbH-1][0])<<(7-log2NbW)
[0201] dVerX=(MvLX[xNb+nNbW-1][yNb+nNbH-1][1]-MvLX[xNb][yNb+nNbH-1][1])<<(7-log2NbW)
[0202] dHorY=-dVerX
[0203] dVerY=dHorX
[0204] The second control point vector may be derived based on at least one of a motion vector (MV) of a spatially adjacent block, a predetermined difference, position information (xCb, yCb) of a current block, a block size (width or height), or position information (xNb, yNb) of a spatially adjacent block. Herein, the block size may mean the size of the current block and / or the size of the spatially adjacent block. The motion vector and the difference are as described in the first control point vector, and a detailed description thereof will be omitted. However, the position of the motion vector used in the process of deriving the second control point vector, the range of the difference, and / or the number may be different from those of the first control point vector.
[0205] For example, the second control point vector can be obtained according to the following equation 7.
[0206] [Equation 7]
[0207] cpMvLX[1][0]=(mvScaleHor+dHorX*(xCb+cbWidth-xNb)+dHorY*(yCb-yNb))
[0208] cpMvLX[1][1]=(mvScaleVer+dVerX*(xCb+cbWidth-xNb)+dVerY*(yCb-yNb))
[0209] In Equation 7, variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 5, and a detailed description thereof will be omitted.
[0210] The third control point vector may be derived based on at least one of a motion vector (MV) of a spatially adjacent block, a predetermined difference, position information (xCb, yCb) of a current block, a block size (width or height), or position information (xNb, yNb) of a spatially adjacent block. Herein, the block size may mean the size of the current block and / or the size of the spatially adjacent block. The motion vector and the difference are as described in the first control point vector, and a detailed description thereof will be omitted. However, the position of the motion vector used in the process of deriving the third control point vector, the range of the difference, and / or the number may be different from the first control point vector or the second control point vector.
[0211] For example, the third control point vector can be obtained according to the following equation 8.
[0212] [Equation 8]
[0213] cpMvLX[2][0]=(mvScaleHor+dHorX*(xCb-xNb)+dHorY*(yCb+cbHeight-yNb))
[0214] cpMvLX[2][1]=(mvScaleVer+dVerX*(xCb-xNb)+dVerY*(yCb+cbHeight-yNb))
[0215] In Equation 8, variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 5, and a detailed description thereof will be omitted. Meanwhile, through the above process, the nth control point vector of the spatial candidate can be derived.
[0216] The above process of deriving affine candidates may be performed for each predefined spatial neighboring block. The predefined spatial neighboring block may include at least one of a left block, a lower left block, an upper right block, an upper block, or an upper left block of the current block.
[0217] Alternatively, the process of deriving affine candidates may be performed for each group of spatial neighboring blocks.Herein, the spatial neighboring blocks may be classified into a first group including a left block and a lower left block and a second group including an upper right block, an upper block, and an upper left block.
[0218] For example, an affine candidate may be derived from the spatial neighboring blocks belonging to the first group. The derivation may be performed until an available affine candidate is found based on a predetermined priority. The priority may be in the order of left block -> lower left block, or in the reverse order. According to the priority, it is determined whether the corresponding spatial neighboring block in the first group is a block decoded by prediction based on an affine mode, and the block decoded by prediction based on the first affine mode may be selected as the affine candidate.
[0219] Similarly, an affine candidate may be derived from the spatial neighboring blocks belonging to the second group. The derivation may be performed until an available affine candidate is found based on a predetermined priority. The priority may be in the order of upper right block->upper block->upper left block, or in the reverse order. According to the priority, it is determined whether the corresponding spatial neighboring block in the second group is a block decoded by prediction based on the affine mode, and a block decoded by prediction based on the first affine mode may be selected as an affine candidate.
[0220] The above-mentioned embodiments may be applied to temporal neighboring blocks in the same / similar manner. Herein, the temporal neighboring block may belong to a different picture from the current block and may be a block at the same position as the current block. The block at the same position may be a block including the position of the upper left sample of the current block, the center position, or the position of a sample adjacent to the lower right sample.
[0221] Alternatively, the temporal neighboring block may mean a block whose position is offset from a block located at the same position by a predetermined disparity vector.Herein, the disparity vector may be determined based on a motion vector of any one of the above-mentioned spatial neighboring blocks of the current block.
[0222] 2. A method for obtaining structural candidates
[0223] The construction candidate may be derived based on a combination of motion vectors of spatial / temporal neighboring blocks in the current block. The construction candidate may be derived based on a combination of at least two control point vectors in the control point vectors corresponding to each corner of the current block (hereinafter referred to as control point vectors (cpMVCorner[n])). Herein, n may be 0, 1, 2, or 3.
[0224] The control point vector (cpMVCorner[n]) may be derived based on the motion vector of the spatial neighboring block and / or the motion vector of the temporal neighboring block. Herein, the spatial neighboring block may include at least one of the following items: a first neighboring block (A, B, or C) adjacent to the upper left sample of the current block; a second neighboring block (D or E) adjacent to the upper right sample of the current block; or a third neighboring block (F or G) adjacent to the lower left sample of the current block. The temporal neighboring block may be a block belonging to a different picture from the current block, and may mean a block located at the same position as the current block (hereinafter referred to as the fourth neighboring block Col). Herein, the fourth neighboring block may mean a block (H, I, or J) including the position of the upper left sample, the upper right sample, or the lower left sample of the current block, or a block adjacent to the position of the lower right sample of the current block.
[0225] The first neighboring block may mean a neighboring block at the upper left (A), above (B), or left (C) of the current block. Whether the motion vectors of neighboring blocks A, B, and C are available is determined according to a predetermined priority, and the motion vectors of the available neighboring blocks may be used to determine the control point vector. This availability determination may be performed until a neighboring block with an available motion vector is found. Herein, the priority may be in the order of A->B->C. However, the present disclosure is not limited thereto, and may be in the order of A->C->B, C->A->B, or B->A->C.
[0226] The second neighboring block may mean an upper neighboring block (D) or an upper right neighboring block (E) of the current block. Likewise, whether the motion vectors of the neighboring blocks D and E are available may be determined according to a predetermined priority, and the control point vector may be determined by using the motion vectors of the available neighboring blocks. This availability determination may be performed until a neighboring block having an available motion vector is found. Herein, the priority may be in the order of D->E, or may be in the order of E->D.
[0227] The third neighboring block may mean a left neighboring block (F) or a lower left neighboring block (G) of the current block. Likewise, whether the motion vector of the neighboring block is available may be determined according to a predetermined priority, and the control point vector may be determined by using the motion vector of the available neighboring block. This availability determination may be performed until a neighboring block with an available motion vector is found. Herein, the priority may be in the order of G->F, or may be in the order of F->G.
[0228] For example, the first control point vector (cpMVCorner[0]) may be set to the motion vector of the first neighboring block, the second control point vector (cpMVCorner[1]) may be set to the motion vector of the second neighboring block, and the third control point vector cpMVCorner[2] may be set to the motion vector of the third neighboring block. The fourth control point vector cpMVCorner[3] may be set to the motion vector of the fourth neighboring block.
[0229] Alternatively, the first control point vector may be derived using at least one motion vector of the first neighboring block or the fourth neighboring block, wherein the fourth neighboring block may be a block including the position of the upper left sample (H). The second control point vector may be derived using at least one motion vector of the second neighboring block or the fourth neighboring block, wherein the fourth neighboring block may be a block including the position of the upper right sample (I). The third control point vector may be derived using at least one motion vector of the third neighboring block or the fourth neighboring block, wherein the fourth neighboring block may be a block including the position of the lower left sample (J).
[0230] Alternatively, any one of the first to fourth control point vectors may be derived based on another one of the first to fourth control point vectors. For example, the second control point vector may be derived by applying a predetermined offset vector to the first control point vector. The offset vector may be a difference vector between the third control point vector and the first control point vector, or may be derived by applying a predetermined scaling factor to the difference vector. The scaling factor may be determined based on at least one of the width or height of the current block and / or the adjacent block.
[0231] The K construction candidates ConstK according to the present disclosure may be determined by a combination of at least two of the first to fourth control point vectors described above. The K value may be an integer of 1, 2, 3, 4, 5, 6, 7 or more. The K value may be derived based on information signaled by the encoding device, or may be a value predetermined to the decoding device. The information may include information indicating the maximum number of construction candidates included in the candidate list.
[0232] Specifically, the first construction candidate Const1 may be obtained by combining the first control point vector to the third control point vector. For example, the first construction candidate Const1 may have a control point vector as shown in Table 1 below. Meanwhile, the control point vector may be configured as shown in Table 1 only when the reference picture information of the first neighboring block is the same as the reference picture information of the second neighboring block and the third neighboring block. Herein, the reference picture information may mean a reference picture index indicating a position of a corresponding reference picture in a reference picture list, or a picture order count (POC) value indicating an output order.
[0233] [Table 1]
[0234] Idx Control point vector 0 cpMvCorner[0] 1 cpMvCorner[1] 2 cpMvCorner[2]
[0235] The second construction candidate Const2 may be derived by combining the first control point vector, the second control point vector, and the fourth control point vector. For example, the second construction candidate Const2 may have a control point vector as shown in Table 2 below. Meanwhile, the control point vector may be configured as shown in Table 2 only when the reference picture information of the first neighboring block is the same as the reference picture information of the second neighboring block and the fourth neighboring block. Herein, the reference picture information is as described above.
[0236] [Table 2]
[0237]
[0238] The third construction candidate Const3 may be obtained by combining the first control point vector, the third control point vector, and the fourth control point vector. For example, the third construction candidate Const3 may have a control point vector as shown in Table 3 below. Meanwhile, the control point vector may be configured as shown in Table 3 only when the reference picture information of the first neighboring block is the same as the reference picture information of the third neighboring block and the fourth neighboring block. Herein, the reference picture information is as described above.
[0239] [Table 3]
[0240]
[0241] The fourth construction candidate Const4 may be derived by combining the second control point vector, the third control point vector, and the fourth control point vector. For example, the fourth construction candidate Const4 may have a control point vector as shown in Table 4 below. Meanwhile, the control point vector may be configured as shown in Table 4 only when the reference picture information of the second neighboring block is the same as the reference picture information of the third neighboring block and the fourth neighboring block. Herein, the reference picture information is as described above.
[0242] [Table 4]
[0243]
[0244] The fifth construction candidate Const5 may be obtained by combining the first control point vector and the second control point vector. For example, the fifth construction candidate Const5 may have a control point vector as shown in Table 5 below. Meanwhile, the control point vector may be configured as shown in Table 5 only when the reference picture information of the first neighboring block is the same as the reference picture information of the second neighboring block. Herein, the reference picture information is as described above.
[0245] [Table 5]
[0246] Idx Control point vector 1 cpMvCorner[0] 2 cpMvCorner[1]
[0247] The sixth construction candidate Const6 may be obtained by combining the first control point vector and the third control point vector. For example, the sixth construction candidate Const6 may have a control point vector as shown in Table 6 below. Meanwhile, the control point vector may be configured as shown in Table 6 only when the reference picture information of the first neighboring block is the same as the reference picture information of the third neighboring block. Herein, the reference picture information is as described above.
[0248] [Table 6]
[0249] Idx Control point vector Control point vector 1 cpMvCorner[0] cpMvCorner[0] 2 cpMvCorner[2] cpMvCorner[1]
[0250] In Table 6, cpMvCorner[1] may be a second control point vector derived based on the first control point vector and the third control point vector. The second control point vector may be derived based on at least one of the first control point vector, a predetermined difference, or a size of a current / neighboring block. For example, the second control point vector may be derived as in Equation 9 below.
[0251] [Equation 9]
[0252] cpMvCorner[1][0]=(cpMvCorner[0][0]<<7)+((cpMvCorner[2][1]-cpMvCorner[0][1])<<(7+Log2(cbHeight / cbWidth)))
[0253] cpMvCorner[1][1]=(cpMvCorner[0][1]<<7)+((cpMvCorner[2][0]-cpMvCorner[0][0])<<(7+Log2(cbHeight / cbWidth)))
[0254] Among the above-mentioned K construction candidates (ConstK), the K value does not limit the position or priority of the construction candidates arranged in the candidate list.
[0255] In addition, all of the first to sixth configuration candidates may be included in the candidate list, or only some of the first to sixth configuration candidates may be included in the candidate list.
[0256] For example, when it is determined that the current block uses three control point vectors, only the construction candidates generated by the combination of three control point vectors from the first control point vector to the fourth control point vector may be used. When it is determined that the current block uses two control point vectors, the construction candidates generated by the combination of at least two control point vectors from the first control point vector to the fourth control point vector may be used, or only the construction candidates generated by the combination of two control point vectors from the first control point vector to the fourth control point vector may be used.
[0257] Alternatively, considering the maximum number of affine candidates included in the candidate list, only some of the construction candidates may be included in the candidate list. Herein, the maximum number may be determined based on information about the maximum number signaled by the encoding device, or may be variably determined considering the above-mentioned characteristics of the current block. In this case, the K value of the construction candidate (ConstK) may mean a priority for insertion into the candidate list.
[0258] Meanwhile, when the difference between two control point vectors belonging to the construction candidate is less than a predetermined threshold, the construction candidate may not be inserted into the candidate list. The difference between the two control point vectors may be divided into a difference in the horizontal direction and a difference in the vertical direction. Herein, the difference in the horizontal direction may mean a difference between the first control point vector 610 and the second control point vector 620, and the difference in the vertical direction may mean a difference between the first control point vector 610 and the third control point vector 630. The threshold may mean 0 or a vector having a size predetermined for the encoding / decoding apparatus.
[0259] The above-mentioned multiple affine candidates can be arranged in the candidate list based on a predetermined priority. For example, multiple affine candidates can be arranged in the candidate list in the order of spatial candidates, temporal candidates, and construction candidates. Alternatively, multiple affine candidates can be arranged in the candidate list in the order of temporal candidates, spatial candidates, and construction candidates. However, the present disclosure is not limited to this, and the temporal candidates can be arranged after the construction candidates. Alternatively, some of the construction candidates can be arranged before the spatial candidates, and the remaining construction candidates can be arranged after the spatial candidates.
[0260] Reference Figure 6 , a control point vector of the current block can be obtained based on the candidate list and the candidate index (S610).
[0261] The candidate index may mean an index encoded to derive a control point vector of the current block. The candidate index may specify any one of a plurality of affine candidates included in the candidate list. The control point vector of the current block may be derived by using the control point vector of the affine candidate specified by the candidate index.
[0262] For example, it is assumed that the type of the affine mode of the current block is 4-parameter (i.e., the current block is determined to use two control point vectors). In this case, when the affine candidate specified by the candidate index has three control point vectors, only two control point vectors (e.g., control point vectors of Idx=0, 1) may be selected from the three control point vectors and set as the control point vectors of the current block. Alternatively, the three control point vectors of the specified affine candidate may be set as the control point vectors of the current block. In this case, the type of the affine mode of the current block may be updated to 6-parameter.
[0263] On the contrary, it is assumed that the type of the affine mode of the current block is 6 parameters (i.e., it is determined that the current block uses 3 control point vectors). In this case, when the affine candidate specified by the candidate index has two control point vectors, an additional control point vector may be generated, and the two control point vectors of the affine candidate and the additional control point vector may be used as the control point vector of the current block. The additional control point vector may be derived based on at least one of the two control point vectors of the affine candidate, the size or position information of the current / neighboring block.
[0264] Alternatively, the two control point vectors of the designated affine candidate may be set as the control point vectors of the current block. In this case, the type of the affine mode of the current block may be updated to 4-parameter.
[0265] Reference Figure 6 , a motion vector of the current block may be derived based on the control point vector of the current block (S620).
[0266] The motion vector may be derived in units of sub-blocks of the current block. To this end, the current block may be divided into a plurality of N×M sub-blocks. Herein, the N×M sub-blocks may be in a non-square (N>M or N<M) shape or a square (N=M) shape. The values of N and M may be 4, 8, 16, 32 or more.
[0267] Taking into account the consumption of storage bandwidth, a motion vector may be derived for each sub-block in the current block, or one motion vector may be derived for the current block. The selective derivation may be performed taking into account whether the current block performs bidirectional prediction (PRED_BI) or at least one of the size of a predetermined reference area (refSize). Herein, a reference area may refer to an area that is referenced for motion compensation of one or more sub-blocks. The reference area may include a reference block specified by a motion vector of one or more sub-blocks and / or pixels for interpolating a minority of pixels of a reference block. The size of the reference area may be represented by any one of the width or height of the reference area, the maximum / minimum value of the width and height, the product of the width and height, the sum of the width and height, and the like.
[0268] For example, when the current block performs bidirectional prediction and the size of the reference area of the n subblocks is less than or equal to a predetermined first threshold (condition 1), a motion vector may be derived for each subblock in the current block.
[0269] Alternatively, when the current block performs unidirectional prediction and the size of the reference area of the m subblocks is less than or equal to a predetermined second threshold (condition 2), a motion vector may be derived for each subblock in the current block.
[0270] The values of n and m can be 1, 2, 3, 4, 5 or more. The values of n and m can be limited to even numbers. n can be set to a value greater than m. For example, n can be set to 4 and m can be set to 2.
[0271] The first threshold may be determined as (15+i)x(15+j), and the second threshold may be determined as (15+i)x(11+j) or (11+i)x(15+j). Herein, i and j may be natural numbers greater than or equal to 0. For example, when i and j are 0, the first threshold and the second threshold may be determined as 225 and 165, respectively.
[0272] The size of the reference area can be determined using at least one of the scaling value (mvScaleHor, mvScaleVer), the first difference value (dHorX, dVerX), or the second difference value (dHorY, dVerY). The method of deriving the scaling value, the first difference value, and the second difference value is as described above, and its detailed description will be omitted.
[0273] The size of the reference area is determined for n subblocks or m subblocks, respectively, and may be determined as in the following Equation 10 in consideration of arrangement of the n subblocks or m subblocks.
[0274] [Equation 10]
[0275] maxW4=Max(0, Max(4*(2048+dHorX), Max(4*dHorY, 4*(2048+dHorX)+4*dHorY)))
[0276] minW4=Min(0,Min(4*(2048+dHorX),Min(4*dHorY,4*(2048+dHorX)+4*dHorY)))
[0277] maxH4=Max(0,Max(4*dVerX,Max(4*(2048+dVerY),4*dVerX+4*(2048+dVerY))))
[0278] minH4=Min(0,Min(4*dVerX,Min(4*(2048+dVerY),4*dVerX+4*(2048+dVerY))))
[0279] bxWX4=((maxW4-minW4)>>11)+9
[0280] bxHX4=(maxH4-minH4)>>11)+9
[0281] bxWXh=((Max(0,4*(2048+dHorX))-Min(0,4*(2048+dHorX)))>>11)+9
[0282] bxHXh=((Max(0,4*dVerX)-Min(0,4*dVerX))>>11)+9
[0283] bxWXv=((Max(0,4*dHorY)-Min(0,4*dHorY))>>11)+9
[0284] bxHXv=((Max(0,4*(2048+dVerY))-Min(0,4*(2048+dVerY)))>>11)+9
[0285] In Equation 10, bxWX4 and bxHX4 may respectively mean the width and height of the reference area of the four sub-blocks. The four sub-blocks may be arranged in a 2×2 form. However, the present disclosure is not limited thereto, and the four sub-blocks may be arranged in a 1×4 or 4×1 form. bxWX4 and bxHX4 may be derived using the first difference (dHorX and dVerX) and the second difference (dHorY and dVerY).
[0286] Meanwhile, bxWXh and bxHXh may respectively mean the width and height of the reference region of the two sub-blocks. The two sub-blocks may be arranged in a 2×1 form (arranged in the horizontal direction). bxWXh and bxHXh may be derived using only the first difference (dHorX and dVerX). bxWXv and bxHXv may respectively mean the width and height of the reference region of the two sub-blocks. The two sub-blocks may be arranged in a 1×2 form (arranged in the vertical direction). bxWXv and bxHXv may be derived using only the second difference (dHorY and dVerY).
[0287] However, in the case of bidirectional prediction, the size of the reference area is determined based on n sub-blocks, and in this case, the n value is not limited to 4, and the sub-block can be a square in the form of 3x3 or 4x4 or a non-square in the form of 2x3 or 3x2. In addition, in the case of unidirectional prediction, the size of the reference area is determined based on m sub-blocks, and in this case, the m value is not limited to 2, and the sub-block can be a square in the form of 2x2 or 3x3 or a non-square in the form of 1x4 or 2x4. In addition, in the case of unidirectional prediction, the size of the reference area is determined based on m sub-blocks, and in this case, the m value is not limited to 2, and the sub-block can be a square in the form of 2x2 or 3x3 or a non-square in the form of 1x4 or 2x4.
[0288] If the above condition 1 or condition 2 is not met, a motion vector can be derived for the current block. That is, all sub-blocks belonging to the current block can share one and the same motion vector, thereby reducing storage bandwidth consumption.
[0289] Based on the control point vector of the current block, an affine motion parameter can be calculated, and a motion vector of the current block can be derived based on the calculated affine motion parameter. Herein, the affine motion parameter may include at least one of the above-mentioned scaling value, the above-mentioned first difference value, or the above-mentioned second difference value.
[0290] Figure 7 A concept of limiting motion vectors in consideration of memory bandwidth is shown as an embodiment to which the present disclosure is applied.
[0291] When a current block has multiple motion vectors, based on one or more reference motion vectors, the characteristics of the remaining motion vector can be limited. The reference motion vector means the motion vector corresponding to the upper left control point. However, the present disclosure is not limited to this, and the reference motion vector can be defined as the motion vector corresponding to the lower left control point or the upper right control point. Through this method, the storage bandwidth used to perform motion prediction and compensation can be reduced. Herein, the characteristics of the motion vector may include at least one of the direction, size, number or position of the motion vector. Through the above restrictions, the remaining motion vector can be set to a vector with the same or similar direction and size as the reference motion vector. That is, when it is determined that too much storage bandwidth is consumed due to the use of motion vectors in units of sub-blocks, multiple sub-blocks belonging to the current block can be set to share / use one motion vector.
[0292] exist Figure 5 , an embodiment is shown in which a motion vector of each sub-block is calculated by dividing a block into two or more sub-blocks, selecting two or more control points representing the block, and using motion vectors corresponding to the corresponding control points.
[0293] in addition, Figure 7 A concept of limiting the remaining motion vectors based on a first motion vector among motion vectors corresponding to two or more control points 311 , 321 , 331 representing the current block 300 is shown.
[0294] like Figure 7 As shown in , according to an embodiment of the present disclosure, when there are motion vectors 310, 320, 330 corresponding to three control points 311, 321, 331 of the current block 300, information such as the direction and magnitude of the second motion vector 320 and / or the third motion vector 330 may be changed to be the same / similar to the first motion vector by using information such as the direction and magnitude of the first motion vector 310. The change of the motion vector refers to clipping or calculating the motion vector to a specific value to change the value of the motion vector.
[0295] In this case, in the case of using the method of limiting the remaining motion vectors by using the first motion vector, a step of recalculating the magnitudes and directions of all or part of the remaining motion vectors using the magnitudes and directions of the first motion vector may be additionally included.
[0296] Limiting the remaining motion vectors by using the first motion vector may mean changing the size and direction of the second motion vector and / or the third motion vector to refer to a position within a predetermined area specified by the first motion vector.
[0297] Figure 7An example of changing the second motion vector 320 and the third motion vector 330 according to the size and direction of the first motion vector 310 is shown.
[0298] Figure 8 An inter-frame prediction method using mask-based prediction in a video encoding method and apparatus according to an embodiment of the present disclosure is shown.
[0299] The method of using the first motion vector to limit the second motion vector and other motion vectors can also be applied to Figure 8 Inter-frame prediction using mask-based prediction is shown in .
[0300] Figure 8 The concept of mask-based motion prediction and compensation proposed in the present disclosure is shown. In the mask-based motion prediction and compensation proposed in the present disclosure, motion prediction and compensation are performed by applying a mask to reference blocks located at two or more different positions of one block.
[0301] exist Figure 8 In the present invention, the following method is provided: for some pixels of a current block 801 in a picture 800 currently performing encoding and / or decoding, pixels are obtained from a reference block 820 referred to by using a motion vector 830, and for other pixels of the current block, pixels are obtained from a reference block 821 referred to by using a motion vector 831. In this case, in the process of obtaining some pixels of the current block, the following method may be used: referencing a target pixel in the form of applying a weight to a pixel value at the same position of a corresponding reference block.
[0302] Fig. 9 The concept of bidirectional prediction in a video encoding and / or decoding method and apparatus according to an embodiment of the present disclosure is shown.
[0303] In the video encoding and decoding method and / or apparatus, bidirectional prediction is shown Fig. 9 In the present invention, a method of performing prediction using two or more reference blocks when encoding and / or decoding a current block 910 existing in a current picture 900 is called bidirectional prediction.
[0304] In this case, the two or more reference blocks may be blocks included in reference pictures 901 and 902 that are temporally encoded and / or decoded prior to the current picture, or blocks included in an area where encoding and / or decoding is performed before the current block in the current picture.
[0305] In addition, one reference block may be a block included in a temporally adjacent picture, and another reference block may be a block included in a region where encoding and / or decoding is performed prior to the current block in the current picture.
[0306] like Fig. 9 As shown in , when bidirectional prediction is performed for a current block 910, a prediction block of the current block 910 may be generated using different motion vectors mv0 (930), mv1 (931) or reference blocks 920 and 921 at a position obtained using two or more motion vectors.
[0307] In this case, the motion vector generally refers to information indicating a relative position of a reference block with respect to the current block 910 in a 2D coordinate plane.
[0308] Additionally, two or more motion vectors may be used.
[0309] Fig.10 A concept of restricted bidirectional prediction is shown as an embodiment to which the present disclosure is applied.
[0310] In the concept of limiting the prediction to the L1 direction in bidirectional prediction, Fig.10 In (a) of FIG. 1 , a method of forcibly performing unidirectional prediction in one direction according to a specific condition when encoding and / or decoding a current block 910 existing in a current picture 900 is called restricted bidirectional prediction.
[0311] In this case, according to an embodiment of the present disclosure, when enforcing unidirectional prediction for one direction, an additional step of determining a predefined specific condition may be included.
[0312] The predefined specific condition may mean a size of a block and a condition of whether to use motion prediction in units of subblocks of the corresponding block.
[0313] In addition, the predefined specific conditions may indicate not only the size of a block and the condition of whether to use motion prediction in units of sub-blocks of the corresponding block, but also the condition of whether to use block partitioning motion prediction such as affine motion prediction and diagonal block partitioning that divides one block into a plurality of blocks. The use of motion prediction in units of sub-blocks may be interpreted in the same meaning as affine motion prediction.
[0314] In addition, the size of the block in the predefined specific condition may mean the size of the block expressed as the sum of the width of the block and the height of the block or the product of the width of the block and the height of the block, and the size of the block in the predefined specific condition may be predefined or transmitted to the decoder through a higher-level syntax.
[0315] In this case, regarding the sum of the width of the block and the height of the block, the size of the block may be 12 derived from (8+4) or (4+8), or may be 16 derived from (8+8). In addition, regarding the product of the width of the block and the height of the block, the size of the block may be 32 derived from (4×8) or (8×4), or may be 64 derived from (8×8).
[0316] When the restricted bidirectional prediction according to an embodiment of the present disclosure is used, even when the motion vector predictor (MVP) of the current block 910 is bidirectionally predicted, a step of forcibly adjusting the motion vector information to perform unidirectional prediction may be additionally included.
[0317] exist Fig.10 In the embodiment of the present disclosure shown in (a) of FIG. 1 , with respect to the current block 910, the following example is shown: limiting the reference to the reference picture 902 in the L1 direction and limiting the reference motion vector mv11002 and the reference block 1012 in the L1 direction. In this case, the example further includes the following steps: forcing the reference motion vector mv1 (1002) in the L1 direction to be initialized as a (0, 0) vector; initializing the reference picture index in the L1 direction; and changing the prediction direction of the current block 910 to an average value L0 during unidirectional prediction.
[0318] Fig.10 (b) in FIG. 1 shows a concept of limiting prediction for the L0 direction in bidirectional prediction according to an embodiment of the present disclosure.
[0319] In the concept of limiting the prediction to the L0 direction in the bidirectional prediction, Fig.10 In (b) of FIG. 1 , a method of forcibly performing unidirectional prediction in one direction under certain conditions when encoding and / or decoding a current block 910 existing in a current picture 900 is called restricted bidirectional prediction.
[0320] In this case, according to an embodiment of the present disclosure, when unidirectional prediction is enforced for one direction, an additional step of determining a predetermined specific condition may be included.
[0321] The predefined specific condition may mean a size of a block and a condition of whether to use motion prediction in units of subblocks of the corresponding block.
[0322] In addition, the predefined specific conditions may indicate not only the size of a block and the condition of whether to use motion prediction in units of sub-blocks of the corresponding block, but also the condition of whether to use block partitioning motion prediction such as affine motion prediction and diagonal block partitioning that divides one block into a plurality of blocks. The use of motion prediction in units of sub-blocks may be interpreted in the same meaning as affine motion prediction.
[0323] When the restricted bidirectional prediction according to an embodiment of the present disclosure is used, even when the motion vector predictor (MVP) of the current block 910 is bidirectionally predicted, a step of forcibly adjusting the motion vector information to perform unidirectional prediction may be additionally included.
[0324] exist Fig.10In the embodiment of the present disclosure shown in (b) of FIG. 1 , the following example is shown: by limiting the reference to the reference picture 901 in the L0 direction with respect to the current block 910, the reference motion vector mv01003 and the reference block 1013 in the L0 direction are limited. In this case, the example further includes the following steps: forcibly initializing the reference motion vector mv0 (1003) in the L0 direction to a (0, 0) vector; initializing the reference picture index in the L0 direction; and changing the prediction direction of the current block 910 to an average value L1 during unidirectional prediction.
[0325] Fig.11 is a flowchart of a process of decoding prediction directionality according to an embodiment of the present disclosure.
[0326] Fig.11 A flow chart of a process of decoding the prediction directionality of a current block is shown, and the process can be used in a motion prediction mode for decoding the prediction directionality of a current block.
[0327] according to Fig.11 In the embodiment shown in , the process of decoding the prediction directionality may include at least one of the following steps: determining whether the current slice type is a slice that allows bidirectional prediction (S1110); or determining whether the current block is a block that allows bidirectional prediction (S1120). In addition, the process of decoding the prediction directionality may also include at least one of the following steps: when the current block is a block that allows bidirectional prediction, parsing the first packet (bin) of the prediction directionality (S1130); or when the size of the current block is not the size of the block that allows bidirectional prediction or the first packet is 0, parsing the second packet of the prediction directionality (S1150).
[0328] In addition, at least one of the following steps may be included: setting the current prediction directionality to a bidirectional prediction direction (S1170); setting the current prediction directionality to a unidirectional prediction direction in the L1 direction (S1180); or setting the current prediction directionality to a unidirectional prediction direction in the L0 direction (S1190).
[0329] When determining whether the current slice type is a slice type that allows bidirectional prediction ( S1110 ), it is determined whether bidirectional prediction is allowed by checking the current slice type.
[0330] In this case, the slice type that allows bidirectional prediction may be a B slice. In addition, in a prediction method using a pre-decoded area of a current picture as a reference picture, the slice type may be diverse.
[0331] The step (S1120) of determining whether the current block is a block that allows bidirectional prediction may mean an additional step of determining a specific condition for the current block, wherein the specific condition may mean the size of the block. In addition, one or more of the following conditions may be additionally included: whether sub-block unit motion prediction of the corresponding block is used; and whether block partitioning motion prediction that divides a block into a plurality of blocks, such as affine motion prediction and diagonal block partitioning, is used.
[0332] exist Fig.11 In the implementation shown in , in the step (S1120) of determining whether the current block is a block allowing bidirectional prediction, the case of using the size of the current block to determine the bidirectional prediction is used as an example, and it is determined whether the size of the current block is the size of the block allowing bidirectional prediction that is predefined and transmitted in a higher-level syntax.
[0333] In this case, the size of a block predefined or transmitted in a higher level syntax that allows bidirectional prediction may be expressed as the sum of a width of the block and a height of the block, or as a product of a width of the block and a height of the block.
[0334] In this case, regarding the sum of the width of the block and the height of the block, the size of the block may be 12 derived from (8+4) or (4+8), or may be 16 derived from (8+8). In addition, regarding the product of the width of the block and the height of the block, the size of the block may be 32 derived from (4×8) or (8×4), or may be 64 derived from (8×8).
[0335] When the size of the current block is the size of a block that allows bidirectional prediction, the step of parsing the first packet for predicting directionality and subsequent steps are performed, and when the size of the current block is not the size of a block that allows bidirectional prediction, the step of parsing the second packet for predicting directionality (S1150) is performed, instead of performing the step of parsing the first packet for predicting directionality (S1130) and the step of determining the value of the first packet (S1140).
[0336] In the step of determining the value of the first packet for predicting the directionality (S1140), when the value of the first packet for predicting the directionality means 1, in the step of setting the prediction directionality of the current block to bidirectional prediction (S1170), the current prediction directionality is set to the bidirectional direction.
[0337] When the size of the current block is not the size of a block that allows bidirectional prediction, or when the value of the first packet for predicting the directionality is 0, a step (S1150) of parsing the second packet for predicting the directionality is performed. In addition, in the step (S1160) of determining the value of the second packet for the corresponding prediction directionality, when the value of the second packet for predicting the directionality means 1, in the step (S1180) of setting the prediction directionality of the current block to a unidirectional prediction in the L1 direction, the current directionality direction is set to a unidirectional prediction in the L1 direction, and when the value of the second packet for predicting the directionality means 0, in the step (S1190) of setting the prediction directionality of the current block to a unidirectional prediction in the L0 direction, the current prediction directionality is set to a unidirectional prediction in the L0 direction.
[0338] Table 7 may be an example of the size of a block for which the size of the current block allows bidirectional prediction. As shown in Table 7, when the blocks are divided using various block structures, when the sum of the width and height of the block is less than or equal to a specific value, bidirectional prediction is not used, and only when the sum of the width and height of the block is greater than a specific value, a restriction may be set so that bidirectional prediction may be used. Table 7 shows an example in which the specific value is limited to 12. The specific value may use a predefined value, or may be signaled in one or more of a sequence parameter set (SPS), a picture parameter set (PPS), and a slice header as a high-level syntax.
[0339] [Table 7]
[0340]
[0341] Fig.12 is a flowchart for determining whether prediction directionality is restricted according to an embodiment of the present disclosure.
[0342] Fig.12 It is a flowchart showing the steps of determining whether to limit the prediction directionality of the current block, and whether to limit the prediction directionality of the current block can be used not only in the steps of encoding and / or decoding the prediction directionality of the current block, but also in the step of setting the prediction directionality of the current block.
[0343] The step of setting the prediction directionality of the current block may mean the following situation: when the current block is encoded and / or decoded in skip mode and / or merge mode, the prediction directionality of the predefined motion candidate is set as the prediction directionality of the current block without going through the step of encoding and / or decoding the prediction directionality of the current block.
[0344] according to Figure 2In the embodiment shown in , the step of determining whether to limit the prediction directionality of the current block may include at least one of the following steps: determining whether the current block is a block that performs motion prediction in units of sub-blocks (S1200); or determining whether the size of the current block is a block size that allows bidirectional prediction (S1210). In addition, when the current block is a restricted bidirectional prediction block, the step of adjusting the prediction directionality, motion vector, and reference picture index of the current block may be included.
[0345] When determining whether the current block is a block that performs motion prediction in units of sub-blocks (S1200), the process is a step of determining whether the current block is a mode in which motion prediction is performed in units of sub-blocks. Motion prediction in units of sub-blocks means a prediction mode in which the current block is divided into a plurality of sub-blocks, and motion prediction is performed with reference to different motion vectors of each sub-block. For example, if the current block performs motion prediction in units of sub-blocks (e.g., affine mode), the bidirectional prediction of the current block may not be restricted, and if the current block does not perform motion prediction in units of sub-blocks (e.g., merge mode), the bidirectional prediction of the current block may be restricted. As described later, even when the current block does not perform motion prediction in units of sub-blocks, it may be determined whether the bidirectional prediction of the current block is restricted by further considering the size of the current block.
[0346] When determining whether the size of the current block is the size of the block allowing bidirectional prediction (S1210), it is determined whether the size of the current block is the size of the block allowing bidirectional prediction predefined or transmitted in a higher level syntax.
[0347] In this case, the size of the block allowing bidirectional prediction, which is predefined or transmitted in a higher level syntax, may be expressed as the sum of the width of the block and the height of the block, or as the product of the width of the block and the height of the block.
[0348] In this case, the block size may be 12 derived from (8+4) or (4+8) or 16 derived from (8+8) regarding the sum of the block width and the block height. In addition, the block size may be 32 derived from (4×8) or (8×4) or 64 derived from (8×8) regarding the product of the block width and the block height.
[0349] For example, if the size of the current block is less than or equal to a predetermined threshold, the bidirectional prediction of the current block may be limited, and if the size of the current block is greater than the predetermined threshold, the bidirectional prediction of the current block may not be limited. Herein, the threshold may be different according to a method of representing the size of the current block, and may be defined as 12, 16, 32, or 64 as described above.
[0350] When the current block is determined to be a restricted bidirectional prediction block, the process may further include the step of adjusting the prediction direction, motion vector and reference picture index of the current block. In an embodiment of the present disclosure, when the current block is determined to be a restricted bidirectional prediction block, the prediction directionality of the current block is set to a unidirectional prediction block in the L0 direction, the motion vector in the L1 direction is initialized using a (0, 0) vector, and the reference picture index in the L1 direction is also initialized.
[0351] Fig.13 is a flowchart for determining whether prediction directionality is restricted according to another embodiment of the present disclosure.
[0352] Fig.13 The invention is a flowchart showing the steps of determining whether to limit the prediction directionality of the current block, and whether to limit the prediction directionality of the current block can be used not only in the step of encoding and / or decoding the prediction directionality of the current block, but also in the step of setting the prediction directionality of the current block. In addition, whether to limit the prediction directionality of the current block can be used in the step of setting the prediction directionality of the sub-block in the current block.
[0353] The step of setting the prediction directionality of the current block may mean the following: when the current block is encoded and / or decoded in skip mode and / or merge mode, the prediction directionality of the predefined motion candidate is set as the prediction directionality of the current block without going through the step of encoding and / or decoding the prediction directionality of the current block. In addition, the process may include the step of setting the prediction directionality for all sub-blocks in the current block.
[0354] according to Fig.13 In the embodiment shown in , the step of determining whether to limit the prediction directionality of the current block may include at least one of the following steps: determining whether the size of the current block is the size of a block that allows bidirectional prediction (S1310); or determining whether the current block is a block that performs motion prediction in units of sub-blocks (S1320). In addition, at least one of the steps of adjusting the prediction directionality, motion vector, and reference picture index of the current block (S1330, S1340) or adjusting the prediction directionality, motion vector, and reference picture index of all sub-blocks in the current block (S1350, S1360) may be further included.
[0355] When determining whether the size of the current block is the size of the block allowing bidirectional prediction (S1310), it is determined whether the size of the current block is the size of the block allowing bidirectional prediction predefined or transmitted in a higher-level syntax.
[0356] In this case, the size of the block allowing bidirectional prediction, which is predefined or transmitted in a higher-level syntax, may be expressed as the sum of the width of the block and the height of the block, or as the product of the width of the block and the height of the block.
[0357] In this case, the block size may be 12 derived from (8+4) or (4+8) or 16 derived from (8+8) regarding the sum of the block width and the block height. In addition, the block size may be 32 derived from (4×8) or (8×4) or 64 derived from (8×8) regarding the product of the block width and the block height.
[0358] When determining whether the current block is a block for performing motion prediction in units of subblocks (S1320), the process is a step of determining whether the current block is a mode for performing motion prediction in units of subblocks. Motion prediction in units of subblocks refers to a prediction mode in which the current block is divided into a plurality of subblocks and motion prediction is performed with reference to different motion vectors for each subblock.
[0359] The steps of adjusting the prediction directionality, motion vector and reference picture index of the current block may be included (S1330 and S1340). In an embodiment of the present disclosure, when it is determined that the current block is a restricted bidirectional prediction block, the prediction directionality of the current block is set to a unidirectional prediction block in the L0 direction, the motion vector in the L1 direction is initialized using a (0, 0) vector, and the reference picture index in the L1 direction is also initialized.
[0360] The steps of adjusting the prediction directionality, motion vector and reference picture index of all sub-blocks in the current block may be included (S1350 and S1360). In an embodiment of the present disclosure, when it is determined that the current block is a restricted bidirectional prediction block, the prediction directionality of all sub-blocks in the current block is set to a unidirectional prediction block in the L0 direction, the motion vector in the L1 direction is initialized using a (0, 0) vector, and the reference picture index in the L1 direction is also initialized.
[0361] Fig.14 A concept of dividing one picture using units defined as a plurality of layers according to an embodiment of the present disclosure is shown.
[0362] When decoding images and videos, an image may be grouped into multiple image blocks and may be decoded independently. In this case, a predetermined fragment region may be defined by grouping multiple image blocks. Herein, a fragment region may represent at least one of a sub-picture, a slice, a tile, a sub-tile (brick), or a CTU.
[0363] For example, a picture may be divided into one or more sub-pictures. A sub-picture may be composed of one or more slices, or may be composed of one or more tiles. A sub-picture may be restricted so that a slice does not span multiple sub-pictures. Alternatively, a sub-picture may be restricted so that a tile does not span multiple sub-pictures.
[0364] In order to divide into sub-pictures, a picture may be divided vertically and horizontally at k-grid intervals. An index may be assigned to each grid. The index may be signaled for each grid. Grouping may be performed for each grid having the same index. A group of grids having the same index may be defined as a sub-picture. A k-grid refers to k sample units, and k may be 4, 8, 16, 32, 64 or more. The k-grid may be determined to be the same as the width and / or height of a CTU predetermined for an encoding / decoding device.
[0365] A slice may consist of one or more tiles, or may consist of one or more bricks. A slice may be replaced by the term tile group. A slice may be specified based on the position of the upper left tile (or brick) and the lower right tile (or brick). In this case, a slice may be implemented as a rectangular shape. A tile may consist of one or more bricks. It will be described later using the term tile or tile group. However, a tile or tile group may be understood to be replaced by a sub-picture, slice, tile, or brick.
[0366] like Fig.14 As shown in , a single image can be reconstructed by dividing it into rectangular shapes such as Tile1_1 to Tile1_5. In this case, a tile can be divided into one or more tiles. This concept can be called a sub-tile. On the contrary, one or more tiles can be grouped together and called a tile group. Looking at this concept hierarchically, the concept can be expressed as a multi-level tile partitioning. A tile refers to a group of one or more coding tree units (CTUs), where the lower right boundary of the tile may not include a complete CTU. Tile groups and sub-tiles may or may not have a rectangular shape. In the case of a tile group, the entire reconstructed image is determined according to the arrangement of the tiles and the form of the bundle. In the case of a sub-tile, the entire reconstructed image can be represented as a bundle of multiple CTUs inside the tile, and may be rectangular or may have the shape of a slice according to a raster scan order.
[0367] First, information for dividing an image into a plurality of tiles may be delivered as part of header information of a compressed bitstream. First, a video or an image is reconstructed based on the division information of tiles for a group of images or an image. When expressing the shape of the tiles, it may be expressed by the number of tiles and the width and height of each tile. In this case, the order indicating the tile shape information may be assumed to be a raster scan order. Fig.14 In the example, when there are 5 tiles, the structure of the tiles can be sequentially reconstructed at the receiving end by using information indicating that the picture is first composed of five tiles and width and height information starting from the first tile.
[0368] In addition, each tile can be divided into sub-tiles, and at the receiving end, how each tile is composed of sub-tiles can be reconstructed by using information about how many sub-tiles each tile is composed of and position and size information such as the width and height of each sub-tile.
[0369] In addition, each tile can be grouped into several groups, and an image can be reconstructed in one bundle. The tile group can be represented by a method of expressing the number of tile groups and the number or order of tiles in each tile group, and the structure of the tile group can be reconstructed at the receiving end.
[0370] The tile represents a unit that can reconstruct an image independently. As a result, when the bitstream information exists, the image reconstruction can be performed independently at the same time, regardless of the order of multiple tiles. According to the group of pictures or the picture, the mutually reconstructed images between the tiles can be referenced or not according to the higher-level header information. In the case of a still image, each tile can be reconstructed completely independently, or can be reconstructed by reference between tiles. As an example, when a reconstructed image that is the same in time and different in space such as an IBC mode or intra-frame prediction is referenced, the reference can be limited to the tile, and the reference between the tiles can be allowed according to the higher-level information. Even in the case of motion compensation, the reference between the tiles cannot be made according to the higher-level information, or the reference between the tiles may be allowed. Whether such spatial reference or temporal reference is allowed can be applied to the sub-tile as the same concept. Entropy decoding can be performed independently of each other, and in some cases, decoding can be performed using some information of the previous sub-tile. Spatial prediction and temporal prediction can be enabled or prohibited between these sub-tiles. It can also be applied to the tile group mentioned above. Each tile can perform entropy decoding independently, but spatial-temporal prediction decoding can be performed only within a tile group. Whether to apply filtering to the boundaries of the reconstructed image between these tiles, between tile groups, or between sub-tiles is determined according to higher-level header information. Herein, filtering may refer to filtering such as deblocking filtering, sample adaptive offset (SAO), adaptive loop filtering (ALF), etc. included in loop filtering in image encoding and decoding devices.
[0371] Herein, sub-tiles may be entropy decoded individually similar to tiles. Alternatively, the first sub-tile may start decoding independently, and a portion of the decoded information of the previous tile may be used to perform entropy decoding on subsequent consecutive sub-tiles. Alternatively, this means that entropy decoding may be performed using reconstructed image information of the previous tile. The multi-level image partitioning may be extended to multiple layers.
[0372] Fig.15 A concept of dividing an image, an image group, or a video into a plurality of tiles and processing the plurality of tiles into three tile groups according to an embodiment of the present disclosure is shown.
[0373] exist Fig.15 In the invention, an image, an image group and a video are divided into a plurality of tiles, and when the plurality of tiles are grouped into three tile groups, each tile group is collected in a separate buffer, and only the reconstructed image corresponding to the tile is shared, and the image corresponding to the tile group can be restored.
[0374] In this case, each tile may be entropy decoded individually or may be reconstructed in an interdependent manner. As previously described, spatiotemporal prediction may be turned on / off by sharing reconstructed images and images between tiles, between tile groups, or between sub-tiles and applying filtering to the boundary areas between them, and spatiotemporal prediction may be signaled using a higher level syntax.
[0375] In this embodiment, the tile group may be replaced by a sub-picture or a slice, and the tile and the sub-tile may be replaced by a slice and a brick, and they may also be applied to the embodiments described later.
[0376] Fig.16 A concept of dividing an image, an image group, or an image into a plurality of tiles, and sharing and restoring only a reconstructed image corresponding to each tile according to an embodiment of the present disclosure is shown.
[0377] exist Fig.16 In the method, images, groups of images, and videos are divided into multiple tiles, each tile is collected in a separate buffer, and only the reconstructed images corresponding to the tiles are shared and can be reconstructed.
[0378] In this case, each tile can be entropy decoded separately or can be reconstructed in an interdependent manner. As mentioned earlier, spatiotemporal prediction can be turned on / off by sharing reconstructed images and images between tiles or between sub-tiles and applying filtering to the boundary areas between them, and higher-level syntax can be used to signal spatiotemporal prediction.
[0379] Fig.17 An example is shown in which an image, an image group, or an image is divided into a plurality of tiles and each tile is divided into sub-tiles according to an embodiment of the present disclosure.
[0380] Fig.17 An example is shown in which an image, a group of images, and a video are divided into multiple tiles and each tile is further divided into sub-tiles. In this case, the sub-tiles are collected in a separate buffer and only the reconstructed image corresponding to the tile is shared, and the image corresponding to the tile can be reconstructed.
[0381] In this case, each sub-tile may be entropy decoded individually, or may be reconstructed in an interdependent manner. The interdependent manner refers to the update information of the reference context table for entropy decoding. As mentioned earlier, spatiotemporal prediction can be turned on / off by sharing the reconstructed image and the image between tiles or between sub-tiles and applying filtering to the boundary area between them, and a higher level syntax can be used to signal the spatiotemporal prediction.
[0382] HRD (Hypothetical Reference Decoder) interactively determines the size and delay information of the buffer on the decoder side in order to smoothly restore and display the video in real time by receiving the compressed bit stream between the decoder and the encoder at a limited rate. Therefore, in order to receive and restore data at a limited rate and output data in real time on the decoder side according to information such as the size, delay time and transmission rate of the buffer defined in the bit stream, it is also necessary to set the decoder to perform the recovery. The buffer size, delay time, transmission rate or the amount of filling the initial buffer can be defined, and using some of this information, other information can be obtained in a proportional manner.
[0383] It has been described that the above-mentioned multi-level tiles or multi-layer tiles can be encoded or decoded independently or dependently. Information about such multi-level tiles or multi-layer tiles can be used to define HRD in multiple levels. HRD for an image, a group of images, or an entire video can be defined. In addition, HRD information can be defined for each tile, and therefore, each tile can independently change the amount of buffering and delay. Alternatively, the size of the buffer, the delay, the amount of initial buffering, and the transmission rate can be considered. By extending this concept, such HRD can be adjusted for each tile group. In addition, according to this concept, HRD can be set separately for sub-tiles, and therefore, the decoder can reconstruct an image or video based on information such as the size of the buffer and the delay according to the individual HRD. In addition, these multi-level HRD rules between levels may be inferred. That is, when HRD is defined for each sub-tile, the HRD for the tile can be inferred as the average, maximum, minimum, and sum of the HDR information of the sub-tile. For example, the size obtained by summing the buffer sizes of the sub-tiles can be used to set the size of the buffer for the corresponding tile. Alternatively, in the case of an initial delay of a tile, a maximum value of the initial delay of a sub-tile can be used. The hierarchical HRD can be sent for an image, a group of images, or an entire video. In particular, in the case of a group of images, the HRD can be defined in a random access point image. In the case of two-way image communication, the HRD can be defined according to the terminal performance by pre-exchanging the terminal performance between devices or between a terminal and a server. In order to facilitate the sending of hierarchical HRD information, the HRD can be defined according to the number of tiles, the number of tile groups, and the number of sub-tiles according to the tile structure information. Alternatively, it can be defined whether the HRD is defined according to the tile, the size of the buffer, and the delay of each HRD, or shared information can be used.
[0384] In the case of special videos such as 360-degree videos, for one image or video, tiles and tile groups of only a portion of the area can be reconstructed. In this case, the bitstream buffer of the decoder can be used alone and the image can be reconstructed according to the tile information and the cross-reference information. In addition, in the video, the tile or tile group to be selectively reconstructed can be changed according to time, and therefore, the HRD must be selectively configured to receive and reconstruct the image.
[0385] Video compression and reconstruction are divided into intra-frame coded / decoded images and inter-frame coded / decoded images. In this case, inter-frame prediction and reconstruction techniques are included in inter-frame coded / decoded images, but inter-frame prediction and reconstruction techniques are not applied to intra-frame coded / decoded images. In addition, intra-frame coded / decoded slices and inter-frame coded / decoded slices are defined, and intra-frame coded / decoded slices use only intra-block coding / decoding techniques, while inter-frame coded / decoded slices use both intra-block coding / decoding techniques and inter-block coding / decoding techniques. In addition, inter-frame tiles can use only intra-block coding / decoding techniques, while inter-frame tiles can use both intra-block coding / decoding techniques and inter-block coding / decoding techniques. In addition, in inter-frame tiles, intra-frame prediction reconstruction can be performed without using information reconstructed by adjacent inter-block coding / decoding techniques such as CIP (constrained intra-frame prediction). The shape of the intra-frame tile is defined by several patterns, so that after some image frames, the subsequent image can be reconstructed regardless of the previous errors. In addition, when the image is completely reconstructed according to the pattern of intra tiles or CIP, the decoder side can remove the image reconstructed before the pattern period from the DPB (Decoded Picture Buffer).
[0386] The various embodiments of the present disclosure are not listed as listing all possible combinations, but are intended to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or may be applied in combination of two or more.
[0387] In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, various embodiments of the present disclosure may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, etc.
[0388] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that enable operations according to the methods of various embodiments to be performed on a device or computer, and includes non-transitory computer-readable media that store and execute software or instructions on a device or computer.
[0389] Industrial availability
[0390] The present disclosure may be used to encode / decode images.
[0391] The solution of the present invention also includes:
[0392] (1) A method for decoding an image, the method comprising:
[0393] Based on an inter-frame mode predefined in the image decoding device, deriving motion information for bidirectional prediction of the current block; and
[0394] performing inter-frame prediction on the current block based on the motion information;
[0395] Wherein, according to the predefined inter-frame mode, the motion information used for bidirectional prediction is adjusted to motion information used for unidirectional prediction.
[0396] (2) The method according to (1),
[0397] Wherein, when the predefined inter-frame mode is a merge mode, the motion information for bidirectional prediction is adjusted to motion information for unidirectional prediction; and
[0398] Wherein, when the predefined inter-frame mode is an affine mode, the motion information used for bidirectional prediction is not adjusted to motion information used for unidirectional prediction.
[0399] (3) The method according to (2),
[0400] Wherein, when the predefined inter-frame mode is a merge mode, the motion information for bidirectional prediction is adjusted to motion information for unidirectional prediction in consideration of the size of the current block.
[0401] (4) The method according to (1),
[0402] Wherein, when the predefined inter-frame mode is a merge mode, obtaining the motion information includes:
[0403] Constructing a merge candidate list of the current block; and
[0404] derive motion information of the current block from the merge candidate list, and
[0405] The merge candidate list includes at least one of spatial merge candidates, temporal merge candidates or combined merge candidates.
[0406] (5) The method according to (1),
[0407] Wherein, when the predefined inter-frame mode is an affine mode, obtaining the motion information includes:
[0408] generating a candidate list consisting of affine candidates of the current block;
[0409] Deriving a control point vector of the current block based on the candidate list and the candidate index; and
[0410] A motion vector of the current block is derived based on the control point vector of the current block.
[0411] (6) The method according to (5),
[0412] The affine candidate includes at least one of a spatial candidate, a temporal candidate or a structural candidate.
[0413] (7) The method according to (5),
[0414] wherein the motion vector of the current block is derived taking into account the size of the predetermined reference area, and
[0415] The reference area includes a reference block specified by motion vectors of n sub-blocks and pixels used to interpolate fractional pixels of the reference block.
[0416] (8) The method according to (7),
[0417] When the size of the reference area is less than or equal to a threshold, a motion vector of the current block is obtained for each sub-block.
[0418] When the size of the reference area is greater than the threshold, a motion vector is obtained for the current block.
[0419] (9) The method according to (8),
[0420] The motion vector of the current block is derived based on affine motion parameters, and
[0421] The affine motion parameter is calculated based on the control point vector of the current block.
[0422] (10) The method according to (1),
[0423] wherein the current block belongs to a predetermined sub-picture, and
[0424] The sub-image is defined as a group of grids with the same index among multiple grids constituting a picture.
Claims
1. A method for decoding an image, the method comprising: Based on an inter-frame mode predefined in the image decoding device, deriving motion information of the current block; as well as performing inter-frame prediction on the current block based on the motion information; Wherein, when the predefined inter-frame mode is a merge mode, the motion information used only for unidirectional prediction is obtained according to the motion information used for bidirectional prediction of the merge candidate specified by the merge index among the multiple merge candidates in the merge candidate list, Wherein, when the predefined inter mode is an affine mode, the motion information for bidirectional prediction is derived according to the motion information for bidirectional prediction of an affine candidate specified by an affine candidate index among a plurality of affine candidates in the affine candidate list, and Wherein, when the predefined inter-frame mode is the merge mode, the multiple merge candidates in the merge candidate list include at least one of a spatial merge candidate, a temporal merge candidate or a combined merge candidate.
2. The method according to claim 1, in, When the predefined inter mode is the merge mode, motion information only for unidirectional prediction is derived from motion information for bidirectional prediction of the merge candidate based on that a product of the width and the height of the current block is 64.
3. A method for encoding an image, the method comprising: Based on the inter-frame mode predefined in the image encoding device, the motion information of the current block is obtained, performing inter-frame prediction on the current block based on the motion information, Wherein, when the predefined inter-frame mode is a merge mode, motion information only used for unidirectional prediction is obtained according to motion information used for bidirectional prediction of a merge candidate among multiple merge candidates in the merge candidate list, Wherein, when the predefined inter-frame mode is an affine mode, the motion information for bidirectional prediction is derived according to the motion information for bidirectional prediction of an affine candidate among the plurality of affine candidates in the affine candidate list, and Wherein, when the predefined inter-frame mode is the merge mode, the multiple merge candidates in the merge candidate list include at least one of a spatial merge candidate, a temporal merge candidate or a combined merge candidate.
4. A non-transitory computer-readable medium for storing a bitstream generated by an image encoding method, the method comprising: Based on the inter-frame mode predefined in the image encoding device, the motion information of the current block is obtained, performing inter-frame prediction on the current block based on the motion information, Wherein, when the predefined inter-frame mode is a merge mode, motion information only used for unidirectional prediction is obtained according to motion information used for bidirectional prediction of a merge candidate among multiple merge candidates in the merge candidate list, Wherein, when the predefined inter-frame mode is an affine mode, the motion information for bidirectional prediction is derived according to the motion information for bidirectional prediction of an affine candidate among the plurality of affine candidates in the affine candidate list, and Wherein, when the predefined inter-frame mode is the merge mode, the multiple merge candidates in the merge candidate list include at least one of a spatial merge candidate, a temporal merge candidate or a combined merge candidate.
