Image encoding / decoding method and device for transmitting compressed video data

Abstract

An image decoding method according to the present disclosure may comprise the steps of: determining motion information of a current block; and acquiring a prediction block of the current block on the basis of the motion information of the current block. In this case, the prediction block may be derived from an extended reference picture, and the extended reference picture may be composed of a reference picture reconstructed before the current picture and an extended region around the reference picture.

Description

Video encoding / decoding method and device for transmitting compressed video data

[0001] The present disclosure relates to a video signal processing method and device.

[0002] Recently, the demand for high-resolution, high-quality images, such as HD (High Definition) and UHD (Ultra High Definition) images, is increasing across various application fields. As image data becomes higher in resolution and quality, the relative amount of data increases compared to conventional image data. Therefore, transmitting image data using existing media such as wired and wireless broadband lines or storing it using existing storage media leads to increased transmission and storage costs. To address these issues arising from the increasing resolution and quality of image data, high-efficiency image compression technologies can be utilized.

[0003] There are various technologies for image compression, such as inter-picture prediction technology that predicts pixel values ​​included in the current picture from pictures before or after the current picture, intra-picture prediction technology that predicts pixel values ​​included in the current picture using pixel information in the current picture, and entropy encoding technology that assigns short codes to values ​​with high frequency of appearance and long codes to values ​​with low frequency of appearance. Using these image compression technologies, image data can be effectively compressed and transmitted or stored.

[0004] Meanwhile, as demand for high-resolution video grows, so does the demand for stereoscopic video content as a new video service. Discussions are underway on video compression technologies to effectively deliver high-resolution and ultra-high-resolution stereoscopic video content.

[0005] The present disclosure aims to provide a method for performing motion compensation based on an extended reference picture and a device therefor.

[0006] The present disclosure aims to provide a method and a device for generating an extended reference picture by padding an area that exceeds the boundary of a reference picture.

[0007] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0008] A video decoding method according to the present disclosure may include a step of determining motion information of a current block; and a step of obtaining a prediction block of the current block based on the motion information of the current block. In this case, the prediction block is derived from an extended reference picture, and the extended reference picture may be composed of a reference picture reconstructed before the current picture and an extended area surrounding the reference picture.

[0009] A video encoding method according to the present disclosure may include a step of determining motion information of a current block; and a step of obtaining a prediction block of the current block based on the motion information of the current block. In this case, the prediction block is derived from an extended reference picture, and the extended reference picture may be composed of a reference picture reconstructed before the current picture and an extended area surrounding the reference picture.

[0010] In the video encoding / decoding method according to the present disclosure, samples within the extended area can be derived by padding samples located at the reference picture boundary.

[0011] In the video encoding / decoding method according to the present disclosure, the extended area includes a horizontal direction restoration area adjacent to the horizontal direction of the reference picture and a vertical direction restoration area adjacent to the vertical direction of the reference picture, and the padding may be performed primarily for one of the horizontal direction and the vertical direction, and then secondarily for the other.

[0012] In the video encoding / decoding method according to the present disclosure, a reference area within a source picture is set based on motion information of a boundary position of the reference picture, and padding can be performed on a padding target area adjacent to the boundary position based on the reference area.

[0013] In the video encoding / decoding method according to the present disclosure, when the motion information does not exist at the boundary position, the reference area can be set based on previously searched motion information.

[0014] In the video encoding / decoding method according to the present disclosure, motion information can be searched in predefined sample units at the boundary of the reference picture.

[0015] In the video encoding / decoding method according to the present disclosure, the source picture may be a picture with an extended boundary.

[0016] In the video encoding / decoding method according to the present disclosure, when there is no restored picture before the reference picture, samples located at the boundary of the reference picture can be padded to derive samples within the extended area.

[0017] In the video encoding / decoding method according to the present disclosure, when the motion information of the boundary position has bidirectional motion information, the reference area can be set using only unidirectional motion information among the bidirectional motion information.

[0018] In the video encoding / decoding method according to the present disclosure, when the reference picture belongs to the L0 reference picture list, the unidirectional motion information may be in the L0 direction, and when the reference picture belongs to the L1 reference picture list, the unidirectional motion information may be in the L1 direction.

[0019] In the video encoding / decoding method according to the present disclosure, when the motion information of the boundary position has bidirectional motion information, an L0 direction reference area and an L1 direction reference area are derived based on the bidirectional information, and padding can be performed on the padding target area through a weighted sum of the L0 direction reference area and the L1 direction reference area.

[0020] In the video encoding / decoding method according to the present disclosure, the extension area includes a first extension area and a second extension area, and the first extension area may be padded based on motion information present at the boundary of the reference picture, and the second extension area may be padded based on samples located at the boundary of the first extension area.

[0021] In the video encoding / decoding method according to the present disclosure, information indicating whether to use the extended reference picture instead of the reference picture can be signaled through a bitstream.

[0022] According to the present disclosure, a computer-readable recording medium for storing a bitstream generated by an image encoding method can be provided.

[0023] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0024] According to the present disclosure, prediction accuracy can be improved by providing a method for performing motion compensation based on an extended reference picture and a device therefor.

[0025] The present disclosure provides a method and a device for generating an extended reference picture by padding an area that exceeds the boundary of a reference picture, thereby improving prediction accuracy.

[0026] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0027] FIG. 1 is a block diagram illustrating an image encoding device according to an embodiment of the present disclosure.

[0028] FIG. 2 is a block diagram showing an image decoding device according to an embodiment of the present disclosure.

[0029] Figure 3 is a diagram schematically illustrating the process of performing inter prediction in an encoder and decoder.

[0030] Figure 4 shows an example in which motion estimation is performed.

[0031] Figures 5 and 6 illustrate examples in which a prediction block of a current block is generated based on motion information generated through motion estimation.

[0032] Figure 7 shows the locations referenced to derive motion vector prediction values.

[0033] Figure 8 is a diagram for explaining a template-based motion estimation method.

[0034] Figure 9 shows examples of template configurations.

[0035] Figure 10 is a diagram for explaining a motion estimation method based on a bilateral matching method.

[0036] Figure 11 is a diagram for explaining a motion estimation method based on a one-way matching method.

[0037] Figures 12 and 13 illustrate examples in which prediction blocks are generated according to the precision of a motion vector.

[0038] Figure 14 shows an example in which motion compensation based on a translational model and a zooming model is performed for the current block.

[0039] Figure 15 shows an example in which motion compensation based on a translational model and a rotational model is performed for the current block.

[0040] Figures 16 and 17 illustrate examples of generating a prediction block for a current block using control point motion vectors.

[0041] Figure 18 shows an example of generating a prediction block for the current block using three control point motion vectors.

[0042] Figure 19 shows an example in which a motion vector is derived in sub-block units.

[0043] Figures 20 and 21 illustrate examples in which motion vectors are derived in units of sub-blocks within the current block when SbTMVP is applied.

[0044] Figures 22 and 23 are diagrams showing examples in which prediction blocks are derived according to motion vector precision.

[0045] Figures 24 and 25 are diagrams for explaining the process of encoding and decoding a motion vector difference value, respectively, when the AMVR method is applied.

[0046] Figure 26 illustrates an example in which there are unavailable samples within a reference block indicated by the motion vector of the current block.

[0047] Figure 27 is a diagram illustrating an extended reference picture.

[0048] Figures 28 and 29 illustrate padding orders.

[0049] Figures 30 and 31 illustrate examples of obtaining an extended area of ​​a reference picture based on a source picture.

[0050] Figure 32 shows an example where motion information of a block located at the boundary of a reference picture is not available.

[0051] Figure 33 shows an example in which padding is performed for an extended area through template matching.

[0052] Figure 34 shows an example of deriving an extended reference picture by combining multiple padding methods.

[0053] Figure 35 shows an example of generating samples that fall outside the boundaries of the current picture through padding.

[0054] The present disclosure may be modified in various ways and encompasses numerous embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present disclosure. Throughout the description of each drawing, similar reference numerals have been used to designate similar components.

[0055] While terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present disclosure, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0056] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0057] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present disclosure. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0058] Hereinafter, preferred embodiments of the present disclosure will be described in more detail with reference to the attached drawings. Hereinafter, identical components in the drawings will be designated by the same reference numerals, and redundant descriptions of identical components will be omitted.

[0059] FIG. 1 is a block diagram illustrating an image encoding device according to an embodiment of the present disclosure.

[0060] Referring to FIG. 1, a video encoding device (100) may include a picture segmentation unit (110), a prediction unit (120, 125), a transformation unit (130), a quantization unit (135), a reordering unit (160), an entropy encoding unit (165), an inverse quantization unit (140), an inverse transformation unit (145), a filter unit (150), and a memory (155).

[0061] Each component shown in Fig. 1 is independently depicted to represent different characteristic functions in the video encoding device, and does not mean that each component is composed of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two components among each component may be combined to form a single component, or one component may be divided into multiple components to perform a function, and such integrated and separate embodiments of each component are also included in the scope of the present disclosure as long as they do not deviate from the essence of the present disclosure.

[0062] Additionally, some components may not be essential components that perform the essential functions of the present disclosure, but may be optional components merely used to enhance performance. The present disclosure may be implemented by including only components essential to implementing the essence of the present disclosure, excluding components used solely for performance enhancement. A structure that includes only essential components, excluding optional components used solely for performance enhancement, is also within the scope of the present disclosure.

[0063] The picture splitting unit (110) can split the input picture into at least one processing unit. At this time, the processing unit may be a prediction unit (PU), a transform unit (TU), or a coding unit (CU). The picture splitting unit (110) can split one picture into a combination of multiple coding units, prediction units, and transform units, and select one combination of coding units, prediction units, and transform units based on a predetermined criterion (e.g., a cost function) to encode the picture.

[0064] For example, a picture can be split into multiple coding units. A recursive tree structure such as a quad tree, a ternary tree, or a binary tree can be used to split a coding unit in a picture. A coding unit that is split into other coding units starting from an image or the largest coding unit as the root can be split into as many child nodes as the number of split coding units. A coding unit that cannot be split any further according to a certain restriction becomes a leaf node. For example, assuming that a quad tree split is applied to a coding unit, a coding unit can be split into at most four different coding units.

[0065] Hereinafter, in the embodiments of the present disclosure, the encoding unit may be used to mean a unit that performs encoding or may be used to mean a unit that performs decoding.

[0066] A prediction unit may be divided into at least one square or rectangular shape of the same size within a single coding unit, or may be divided such that one prediction unit among the divided prediction units within a single coding unit has a different shape and / or size from another prediction unit.

[0067] When predicting within a screen, the transformation unit and the prediction unit can be set to be the same. In this case, the encoding unit can be divided into multiple transformation units, and then intra-screen prediction can be performed for each transformation unit. The encoding unit can be divided in the horizontal direction or the vertical direction. The number of transformation units generated by dividing the encoding unit can be 2 or 4, depending on the size of the encoding unit. Alternatively, when the size of the transformation unit is small, multiple transformation units can be set as a single prediction unit.

[0068] The prediction unit (120, 125) may include an inter-prediction unit (120) that performs inter-prediction and an intra-prediction unit (125) that performs intra-prediction. It may be determined whether to use inter-prediction or intra-prediction for an encoding unit, and specific information (e.g., reference sample line, intra-prediction mode, motion vector, reference picture, etc.) according to each prediction method may be determined. At this time, the processing unit where prediction is performed and the processing unit where the prediction method and specific contents are determined may be different. For example, the prediction method and prediction mode, etc. are determined in the encoding unit, and the prediction may be performed in the prediction unit or the transformation unit. The residual value (residual block) between the generated prediction block and the original block may be input to the transformation unit (130). In addition, the prediction mode information, motion vector information, etc. used for prediction may be encoded together with the residual value in the entropy encoding unit (165) and transmitted to the decoding device. When using a specific encoding mode, it is also possible to encode the original block as is and transmit it to the decoding unit without generating a prediction block through the prediction unit (120, 125).

[0069] The inter-screen prediction unit (120) may predict a prediction unit based on information of at least one picture among the previous or subsequent pictures of the current picture, and in some cases, may predict a prediction unit based on information of a portion of an encoded region within the current picture. The inter-screen prediction unit (120) may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.

[0070] The reference picture interpolation unit can receive reference picture information from the memory (155) and generate pixel information less than an integer pixel from the reference picture. In the case of luminance pixels, a DCT-based 8-tap interpolation filter with different filter coefficients can be used to generate pixel information less than an integer pixel in units of 1 / 4 pixels. In the case of a chrominance signal, a DCT-based 4-tap interpolation filter with different filter coefficients can be used to generate pixel information less than an integer pixel in units of 1 / 8 pixels.

[0071] The motion prediction unit can perform motion prediction based on a reference picture interpolated by the reference picture interpolation unit. Various methods can be used to derive a motion vector, such as FBMA (Full search-based Block Matching Algorithm), TSS (Three Step Search), and NTS (New Three-Step Search Algorithm). The motion vector can have a motion vector value of 1 / 2 or 1 / 4 pixel unit based on the interpolated pixel. The motion prediction unit can predict the current prediction unit by using different motion prediction methods. Various methods can be used as motion prediction methods, such as the Skip method, the Merge method, the AMVP (Advanced Motion Vector Prediction) method, and the Intra Block Copy method.

[0072] The on-screen prediction unit (125) can generate a prediction block based on reference pixel information, which is pixel information within the current picture. The reference pixel information can be derived from one selected from among a plurality of reference pixel lines. The Nth reference pixel line among the plurality of reference pixel lines can include left pixels having an x-axis difference of N from the upper left pixel within the current block and upper pixels having a y-axis difference of N from the upper left pixel. The number of reference pixel lines that the current block can select can be 1, 2, 3, or 4.

[0073] If the neighboring blocks of the current prediction unit are blocks that have performed inter-screen prediction and the reference pixel is a pixel that has performed inter-screen prediction, the reference pixel included in the block that has performed inter-screen prediction can be replaced with the reference pixel information of the neighboring block that has performed intra-screen prediction. That is, if the reference pixel is unavailable, the unavailable reference pixel information can be replaced with information from at least one of the available reference pixels.

[0074] In intra-screen prediction, the prediction mode can have a directional prediction mode that uses reference pixel information according to the prediction direction, and a non-directional mode that does not use directional information when performing prediction. The mode for predicting luminance information and the mode for predicting chrominance information can be different, and the intra-screen prediction mode information used to predict luminance information or the predicted luminance signal information can be utilized to predict chrominance information.

[0075] When performing intra-screen prediction, if the size of the prediction unit and the size of the transformation unit are the same, intra-screen prediction for the prediction unit can be performed based on the pixels on the left side of the prediction unit, the pixels on the upper left side, and the pixels on the upper side.

[0076] The on-screen prediction method can generate prediction blocks by applying a smoothing filter to reference pixels according to the prediction mode. Depending on the selected reference pixel line, whether or not the smoothing filter is applied can be determined.

[0077] In order to perform an intra-screen prediction method, the intra-screen prediction mode of the current prediction unit can be predicted from the intra-screen prediction modes of prediction units existing around the current prediction unit. When the prediction mode of the current prediction unit is predicted using mode information predicted from the surrounding prediction units, if the intra-screen prediction modes of the current prediction unit and the surrounding prediction units are the same, information indicating that the prediction modes of the current prediction unit and the surrounding prediction units are the same can be transmitted using predetermined flag information, and if the prediction modes of the current prediction unit and the surrounding prediction units are different, entropy encoding can be performed to encode the prediction mode information of the current block.

[0078] Additionally, a residual block containing residual value information, which is the difference between the prediction unit that performed the prediction based on the prediction unit generated in the prediction unit (120, 125) and the original block of the prediction unit, can be generated. The generated residual block can be input to the transformation unit (130).

[0079] In the transformation unit (130), the residual block including the residual value information of the prediction unit generated through the original block and the prediction unit (120, 125) can be transformed using a transformation method such as DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), or KLT. Whether to apply DCT, DST, or KLT to transform the residual block can be determined based on at least one of the size of the transformation unit, the shape of the transformation unit, the prediction mode of the prediction unit, or the prediction mode information within the screen of the prediction unit. Meanwhile, the transformation can be performed by separating the horizontal direction and the vertical direction.

[0080] After performing transformations in the horizontal and vertical directions, a secondary transformation can be performed. The secondary transformation may be in a form in which the horizontal and vertical directions are not separated. The secondary transformation can be performed on the transformation coefficients obtained by the primary transformation to generate final transformation coefficients. Meanwhile, the number of final transformation coefficients output by the secondary transformation may be smaller than the number of transformation coefficients input for the secondary transformation. Specifically, the secondary transformation can be performed using a reduced transformation matrix having different numbers of columns and rows.

[0081] The quantization unit (135) can quantize values ​​converted to the frequency domain by the transformation unit (130). The quantization coefficients can vary depending on the block or the importance of the image. The values ​​produced by the quantization unit (135) can be provided to the dequantization unit (140) and the reordering unit (160).

[0082] The rearrangement unit (160) can perform rearrangement of coefficient values ​​for quantized residual values.

[0083] The reordering unit (160) can change a two-dimensional block-shaped coefficient into a one-dimensional vector form through a coefficient scanning method. For example, the reordering unit (160) can change the two-dimensional block-shaped coefficient into a one-dimensional vector form by scanning from the DC coefficient to the coefficient of the high-frequency region using a zig-zag scan method. Depending on the size of the conversion unit and the intra-screen prediction mode, a vertical scan that scans the two-dimensional block-shaped coefficient in the column direction, a horizontal scan that scans the two-dimensional block-shaped coefficient in the row direction, or a diagonal scan that scans the two-dimensional block-shaped coefficient in the diagonal direction may be used instead of the zig-zag scan. That is, depending on the size of the conversion unit and the intra-screen prediction mode, it is possible to determine which scan method among the zig-zag scan, the vertical scan, the horizontal scan, or the diagonal scan is to be used.

[0084] The entropy encoding unit (165) can perform entropy encoding based on the values ​​produced by the rearrangement unit (160). Entropy encoding can use various encoding methods such as, for example, Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC).

[0085] The entropy encoding unit (165) can encode various information such as residual value coefficient information of the encoding unit, block type information, prediction mode information, division unit information, prediction unit information, transmission unit information, motion vector information, reference frame information, block interpolation information, and filtering information from the rearrangement unit (160) and the prediction unit (120, 125).

[0086] The entropy encoding unit (165) can entropy encode the coefficient values ​​of the encoding unit input from the rearrangement unit (160).

[0087] The inverse quantization unit (140) and the inverse transformation unit (145) inversely quantize the values ​​quantized in the quantization unit (135) and inversely transform the values ​​transformed in the transformation unit (130). The residual values ​​generated in the inverse quantization unit (140) and the inverse transformation unit (145) can be combined with the predicted prediction units predicted through the motion estimation unit, motion compensation unit, and intra-screen prediction unit included in the prediction unit (120, 125) to generate a reconstructed block.

[0088] The filter unit (150) may include at least one of a deblocking filter, an offset correction unit, and an ALF (Adaptive Loop Filter).

[0089] A deblocking filter can remove block distortion caused by boundaries between blocks in a reconstructed picture. To determine whether to perform deblocking, a deblocking filter can be applied to the current block based on the pixels contained in several columns or rows within the block. When applying a deblocking filter to a block, a strong filter or a weak filter can be applied depending on the required deblocking filtering strength. Furthermore, when applying a deblocking filter, horizontal and vertical filtering can be processed in parallel when performing vertical and horizontal filtering.

[0090] The offset correction unit can correct the offset from the original image on a pixel-by-pixel basis for an image that has undergone deblocking. To perform offset correction for a specific picture, the pixels contained in the image can be divided into a certain number of regions, the regions to be offset can be determined, and the offset can be applied to those regions. Alternatively, the offset can be applied by considering the edge information of each pixel.

[0091] Adaptive Loop Filtering (ALF) can be performed based on the comparison of the filtered restored image with the original image. After dividing the pixels included in the image into predetermined groups, a filter to be applied to each group can be determined, and filtering can be performed differentially for each group. Information regarding whether to apply ALF can be transmitted by luminance signal for each coding unit (CU), and the shape and filter coefficients of the ALF filter to be applied can vary depending on each block. Furthermore, an ALF filter of the same form (fixed form) can be applied regardless of the characteristics of the target block.

[0092] The memory (155) can store a restoration block or picture produced through the filter unit (150), and the stored restoration block or picture can be provided to the prediction unit (120, 125) when performing inter-screen prediction.

[0093] FIG. 2 is a block diagram showing an image decoding device according to an embodiment of the present disclosure.

[0094] Referring to FIG. 2, the image decoding device (200) may include an entropy decoding unit (210), a rearrangement unit (215), an inverse quantization unit (220), an inverse transformation unit (225), a prediction unit (230, 235), a filter unit (240), and a memory (245).

[0095] When a video bitstream is input to a video encoding device, the input bitstream can be decoded in the opposite procedure to that of the video encoding device.

[0096] The entropy decoding unit (210) can perform entropy decoding in a procedure opposite to that of the entropy encoding unit of the video encoding device. For example, various methods such as Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC) can be applied in response to the method performed in the video encoding device.

[0097] The entropy decoding unit (210) can decode information related to intra-screen prediction and inter-screen prediction performed in the encoding device.

[0098] The reordering unit (215) can perform reordering based on the method in which the bitstream entropy-decoded by the entropy decoding unit (210) is reordered by the encoding unit. The coefficients expressed in the form of a one-dimensional vector can be reordered by restoring them back to coefficients in the form of a two-dimensional block. The reordering unit (215) can perform reordering by receiving information related to the coefficient scanning performed by the encoding unit and performing reverse scanning based on the scanning order performed by the corresponding encoding unit.

[0099] The dequantization unit (220) can perform dequantization based on the quantization parameters provided from the encoding device and the coefficient values ​​of the rearranged block.

[0100] The inverse transform unit (225) can perform an inverse transform of the transform performed by the transform unit on the quantization result performed by the image encoding device. That is, at least one of an inverse transform of a secondary transform (secondary inverse transform) or an inverse transform for DCT, DST, and KLT (i.e., first inverse transform) can be performed. The inverse transform can be performed based on a transmission unit determined by the image encoding device. The inverse transform unit (225) of the image decoding device can determine a transform matrix for the second inverse transform or a transform technique (e.g., DCT, DST, KLT) for the first inverse transform according to a plurality of pieces of information such as a prediction method, the size and shape of the current block, the prediction mode, and the prediction direction within the screen. Alternatively, information for determining the transform matrix or the transform technique may be explicitly encoded and signaled.

[0101] The prediction unit (230, 235) can generate a prediction block based on the prediction block generation related information provided by the entropy decoding unit (210) and the previously decoded block or picture information provided by the memory (245).

[0102] As described above, when performing intra-screen prediction in the same manner as the operation in the video encoding device, if the size of the prediction unit and the size of the transformation unit are the same, intra-screen prediction for the prediction unit is performed based on the pixels on the left side of the prediction unit, the pixels on the upper left side, and the pixels on the upper side. However, when performing intra-screen prediction, if the size of the prediction unit and the size of the transformation unit are different, intra-screen prediction can be performed using reference pixels based on the transformation unit. In addition, intra-screen prediction using NxN division only for the minimum coding unit can be used.

[0103] The prediction unit (230, 235) may include a prediction unit determination unit, an inter-screen prediction unit, and an intra-screen prediction unit. The prediction unit determination unit may receive various information such as prediction unit information input from the entropy decoding unit (210), prediction mode information of an intra-screen prediction method, and motion prediction-related information of an inter-screen prediction method, and may distinguish a prediction unit from a current encoding unit and determine whether the prediction unit performs inter-screen prediction or intra-screen prediction. The inter-screen prediction unit (230) may perform inter-screen prediction on the current prediction unit based on information included in at least one of a previous picture or a subsequent picture of the current picture including the current prediction unit, using information necessary for inter-screen prediction of the current prediction unit provided from the video encoding device. Alternatively, inter-screen prediction may be performed based on information on a pre-restored portion of the current picture including the current prediction unit.

[0104] In order to perform inter-screen prediction, it is possible to determine whether the motion prediction method of the prediction unit included in the encoding unit is Skip Mode, Merge Mode, AMVP Mode, or Intra-screen Block Copy Mode based on the encoding unit.

[0105] The intra-screen prediction unit (235) can generate a prediction block based on pixel information within the current picture. If the prediction unit is a prediction unit that has performed intra-screen prediction, intra-screen prediction can be performed based on intra-screen prediction mode information of the prediction unit provided by the video encoding device. The intra-screen prediction unit (235) can include an AIS (Adaptive Intra Smoothing) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter is a part that performs filtering on the reference pixels of the current block, and can determine and apply whether to apply the filter according to the prediction mode of the current prediction unit. AIS filtering can be performed on the reference pixels of the current block using the prediction mode and AIS filter information of the prediction unit provided by the video encoding device. If the prediction mode of the current block is a mode that does not perform AIS filtering, the AIS filter may not be applied.

[0106] The reference pixel interpolation unit can generate a reference pixel of a pixel unit less than an integer value by interpolating the reference pixel when the prediction mode of the prediction unit is a prediction unit that performs intra-screen prediction based on the pixel value interpolated from the reference pixel. If the prediction mode of the current prediction unit is a prediction mode that generates a prediction block without interpolating the reference pixel, the reference pixel may not be interpolated. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is the DC mode.

[0107] The restored block or picture may be provided to a filter unit (240). The filter unit (240) may include a deblocking filter, an offset correction unit, and an ALF.

[0108] Information regarding whether a deblocking filter has been applied to a corresponding block or picture may be received from a video encoding device, and if a deblocking filter has been applied, information regarding whether a strong or weak filter has been applied. The deblocking filter of the video decoding device may receive information related to the deblocking filter provided by the video encoding device, and the video decoding device may perform deblocking filtering on the corresponding block.

[0109] The offset correction unit can perform offset correction on the restored image based on the type of offset correction applied to the image during encoding and offset value information.

[0110] ALF can be applied to an encoding unit based on information such as whether ALF is applied and ALF coefficient information provided from an encoding device. This ALF information can be provided by being included in a specific parameter set.

[0111] The memory (245) can store a restored picture or block so that it can be used as a reference picture or reference block, and can also provide the restored picture to an output unit.

[0112] As described above, in the following embodiments of the present disclosure, for convenience of explanation, the term coding unit is used as an encoding unit, but it may also be a unit that performs not only encoding but also decoding.

[0113] In addition, the current block represents a block to be encoded / decoded, and may represent a coding tree block (or coding tree unit), an encoding block (or encoding unit), a transform block (or transform unit), a prediction block (or prediction unit), or a block to which an in-loop filter is applied, depending on the encoding / decoding step. In this specification, a 'unit' represents a basic unit for performing a specific encoding / decoding process, and a 'block' may represent a pixel array of a predetermined size. Unless otherwise distinguished, 'block' and 'unit' may be used with the same meaning. For example, in the embodiment described below, an encoding block (coding block) and an encoding unit (coding unit) may be understood to have the same meaning.

[0114] Furthermore, we will refer to the picture that contains the current block as the current picture.

[0115] When encoding the current picture, redundant data between pictures can be removed through inter-prediction. Inter-prediction can be performed on a block-by-block basis. Specifically, a prediction block of the current block can be generated from a reference picture using motion information of the current block. Here, the motion information can include at least one of a motion vector, a reference picture index, and a prediction direction.

[0116] Figure 3 is a diagram schematically illustrating the process of performing inter prediction in an encoder and decoder.

[0117] As in the example illustrated in FIG. 3, to perform inter prediction, motion information for the current block can be acquired (S310). Here, the motion information can include at least one of a motion vector, a reference picture index, or a weight applied to the prediction block. For the current block, motion information for at least one of the L0 direction or the L1 direction can be acquired.

[0118] In the encoder, motion information of the current block can be derived through motion estimation, and the derived motion information can be encoded and signaled to the decoder. Meanwhile, the encoding / decoding of motion information can be based on a motion information merging mode, a motion vector prediction mode, a template-based motion estimation method, or a bilateral matching method, which will be described later.

[0119] In the decoder, motion information of the current block can be derived based on the information transmitted from the encoder.

[0120] Alternatively, the motion information of the current block can be derived from the decoder in the same manner as in the encoder. This method can be referred to as decoder-side motion estimation.

[0121] Once motion information for the current block is derived, a prediction block for the current block can be obtained based on the derived motion information (S320). For example, a reference block spaced apart by a motion vector from the current block's position within the reference picture can be set as the prediction block for the current block.

[0122] Below, we will explain in more detail the process of calculating inter predictions.

[0123] The motion information of the current block can be generated through motion estimation.

[0124] Figure 4 shows an example in which motion estimation is performed.

[0125] In Fig. 4, it is assumed that the POC (Picture Order Count) of the current picture is T, and the POC of the reference picture is (T-1).

[0126] A search range for motion estimation can be set from the same location as the reference point of the current block within the reference picture. Here, the reference point may be the location of the upper left sample of the current block.

[0127] For example, in FIG. 4, a rectangle of size (w0+w01) and (h0+h1) is set as a search range centered on a reference point. In the above example, w0, w1, h0, and h1 may have the same value. Alternatively, at least one of w0, w1, h0, and h1 may be set to have a different value from the other. Alternatively, the sizes of w0, w1, h0, and h1 may be determined so as not to exceed a Coding Tree Unit (CTU) boundary, a slice boundary, a tile boundary, or a picture boundary.

[0128] Within the search range, reference blocks of the same size as the current block can be set, and the cost of each reference block relative to the current block can be measured. The cost can be calculated using the similarity between the two blocks.

[0129] For example, the cost can be calculated based on the absolute sum of the differences between the original samples in the current block and the original samples (or reconstructed samples) in the reference block. A smaller absolute sum can reduce the cost.

[0130] Afterwards, the cost of each reference block is compared, and the reference block with the optimal cost can be set as the prediction block of the current block.

[0131] Additionally, the distance between the current block and the reference block can be set as a motion vector. Specifically, the x-coordinate difference and the y-coordinate difference between the current block and the reference block can be set as the motion vector.

[0132] Furthermore, the index of the picture containing the reference block identified through motion estimation is set as the reference picture index.

[0133] Additionally, the prediction direction can be set based on whether the reference picture belongs to the L0 reference picture list or the L1 reference picture list.

[0134] Additionally, motion estimation can be performed for each of the L0 direction and the L1 direction. If prediction is performed for both the L0 direction and the L1 direction, motion information in the L0 direction and motion information in the L1 direction can be generated, respectively.

[0135] Figures 5 and 6 illustrate examples in which a prediction block of a current block is generated based on motion information generated through motion estimation.

[0136] Figure 5 shows an example of generating a prediction block by unidirectional (i.e., L0 direction) prediction, and Figure 6 shows an example of generating a prediction block by bidirectional (i.e., L0 and L1 direction) prediction.

[0137] In the case of unidirectional prediction, a prediction block of the current block is generated using a single motion information. For example, the motion information may include an L0 motion vector, an L0 reference picture index, and prediction direction information indicating the L0 direction.

[0138] In the case of bidirectional prediction, a prediction block is generated using two pieces of motion information. For example, a reference block in the L0 direction, determined based on motion information for the L0 direction (L0 motion information), can be set as an L0 prediction block, and an L1 prediction block can be generated based on a reference block in the L1 direction, determined based on motion information for the L1 direction (L1 motion information). Thereafter, the L0 prediction block and the L1 prediction block can be weighted and combined to generate a prediction block of the current block.

[0139] In the examples shown in FIGS. 4 to 6, the L0 reference picture is illustrated as existing in the previous direction of the current picture (i.e., having a POC value smaller than that of the current picture), and the L1 reference picture is illustrated as existing in the subsequent direction of the current picture (i.e., having a POC value larger than that of the current picture).

[0140] However, unlike the illustrated example, the L0 reference picture may exist in the subsequent direction of the current picture, or the L1 reference picture may exist in the previous direction of the current picture. For example, both the L0 reference picture and the L1 reference picture may exist in the previous direction of the current picture, or both may exist in the subsequent direction of the current picture. Alternatively, bidirectional prediction may be performed using the L0 reference picture existing in the subsequent direction of the current picture and the L1 reference picture existing in the previous direction of the current picture.

[0141] Motion information for blocks for which inter prediction has been performed can be stored in memory. At this time, the motion information can be stored on a sample-by-sample basis. Specifically, the motion information for a block to which a specific sample belongs can be stored as motion information for that specific sample. The stored motion information can be used to derive motion information for neighboring blocks to be encoded / decoded in the future.

[0142] In the encoder, information encoding residual samples corresponding to the difference between the sample of the current block (i.e., the original sample) and the predicted sample, and motion information required to generate a predicted block can be signaled to the decoder. The decoder can decode information about the signaled difference value to derive a difference sample, and add a prediction sample within the predicted block generated using the motion information to the difference sample to generate a restored sample.

[0143] At this time, in order to effectively compress the motion information signaled to the decoder, one of a plurality of inter prediction modes may be selected. Here, the plurality of inter prediction modes may include a motion information merging mode and a motion vector prediction mode.

[0144] The motion vector prediction mode is a mode that signals by encoding the difference between a motion vector and a motion vector prediction value. Here, the motion vector prediction value can be derived based on motion information of neighboring blocks or neighboring samples adjacent to the current block.

[0145] Figure 7 shows the locations referenced to derive motion vector prediction values.

[0146] For convenience of explanation, the current block is assumed to have a size of 4x4.

[0147] In the illustrated example, 'LB' represents a sample contained in the leftmost column and bottommost row within the current block. 'RT' represents a sample contained in the rightmost column and topmost row within the current block. A0 to A4 represent samples neighboring to the left of the current block, and B0 to B5 represent samples neighboring to the top of the current block. For example, A1 represents a sample neighboring to the left of LB, and B1 represents a sample neighboring to the top of RT.

[0148] Col indicates the location of a sample neighboring the lower right of the current block within a co-located picture. A co-located picture is a picture different from the current picture, and information for specifying the co-located picture (e.g., a co-located picture index) can be explicitly encoded and signaled in the bitstream. Alternatively, a reference picture having a predefined reference picture index can be set as the co-located picture.

[0149] The motion vector prediction value of the current block can be derived from at least one motion vector prediction candidate included in a motion vector prediction list.

[0150] The number of motion vector prediction candidates that can be inserted into the motion vector prediction list (i.e., the size of the list) may be predefined in the encoder and decoder. For example, the maximum number of motion vector prediction candidates may be 2.

[0151] A motion vector stored at the location of a neighboring sample adjacent to the current block or a scaled motion vector derived by scaling the motion vector can be inserted into the motion vector prediction list as a motion vector prediction candidate. At this time, the motion vector prediction candidates can be derived by scanning the neighboring samples adjacent to the current block in a predefined order.

[0152] For example, it is possible to check whether a motion vector is stored at each location in the order of A0 to A4. Then, according to the above scanning order, the first available motion vector found can be inserted into the motion vector prediction list as a motion vector prediction candidate.

[0153] As another example, in the order of A0 to A4, it is checked whether a motion vector is stored at each position, and the motion vector of the position that is found first and has the same reference picture as the current block can be inserted into the motion vector prediction list as a motion vector prediction candidate. If there is no neighboring sample that has the same reference picture as the current block, a motion vector prediction candidate can be derived based on the first found available vector. Specifically, the first found available motion vector can be scaled, and then the scaled motion vector can be inserted into the motion vector prediction list as a motion vector prediction candidate. At this time, the scaling can be performed based on the output order difference between the current picture and the reference picture (i.e., the POC difference) and the output order difference between the current picture and the reference picture of the neighboring sample (i.e., the POC difference).

[0154] Furthermore, it is possible to check whether a motion vector is stored at each location in the order of B0 to B5. Then, according to the above scanning order, the first available motion vector found can be inserted into the motion vector prediction list as a motion vector prediction candidate.

[0155] As another example, in the order of B0 to B5, it is checked whether a motion vector is stored at each position, and the motion vector of the position that has the same reference picture as the current block that is found first can be inserted into the motion vector prediction list as a motion vector prediction candidate. If there is no neighboring sample that has the same reference picture as the current block, a motion vector prediction candidate can be derived based on the first found available vector. Specifically, the first found available motion vector can be scaled, and then the scaled motion vector can be inserted into the motion vector prediction list as a motion vector prediction candidate. At this time, the scaling can be performed based on the output order difference between the current picture and the reference picture (i.e., the POC difference) and the output order difference between the current picture and the reference picture of the neighboring sample (i.e., the POC difference).

[0156] As in the example described above, a motion vector prediction candidate can be derived from a sample adjacent to the left of the current block, and a motion vector prediction candidate can be derived from a sample adjacent to the top of the current block.

[0157] At this time, the motion vector prediction candidate derived from the left sample may be inserted into the motion vector prediction list before the motion vector prediction candidate derived from the upper sample. In this case, the index assigned to the motion vector prediction candidate derived from the left sample may have a smaller value than the motion vector prediction candidate derived from the upper sample.

[0158] Conversely, the motion vector prediction candidate derived from the top sample may be inserted into the motion vector prediction list before the motion vector prediction candidate derived from the left sample.

[0159] Among the motion vector prediction candidates included in the above motion vector prediction list, the motion vector prediction candidate with the highest encoding efficiency can be set as the motion vector predictor (MVP) of the current block. In addition, index information indicating the motion vector prediction candidate set as the motion vector predictor of the current block among the plurality of motion vector prediction candidates can be encoded and signaled to a decoder. When the number of motion vector prediction candidates is two, the index information can be a 1-bit flag (e.g., an MVP flag). In addition, a motion vector difference (MVD), which is the difference between the motion vector of the current block and the motion vector predictor, can be encoded and signaled to a decoder.

[0160] The decoder can construct a motion vector prediction list, similar to the encoder. Furthermore, it can decode index information from the bitstream and select one of multiple motion vector prediction candidates based on the decoded index information. The selected motion vector prediction candidate can be set as the motion vector prediction value of the current block.

[0161] Additionally, the motion vector differential can be decoded from the bitstream. Afterwards, the motion vector of the current block can be derived by combining the motion vector prediction value and the motion vector differential value.

[0162] When bidirectional prediction is applied to the current block, a motion vector prediction list can be generated for each of the L0 and L1 directions. That is, the motion vector prediction list can be composed of motion vectors in the same direction. Accordingly, the motion vector of the current block and the motion vector prediction candidates included in the motion vector prediction list have the same direction.

[0163] When the motion vector prediction mode is selected, reference picture index and prediction direction information can be explicitly encoded and signaled to the decoder. For example, when there are multiple reference pictures in the reference picture list and motion estimation is performed for each of the multiple reference pictures, a reference picture index for specifying a reference picture from which motion information of the current block is derived among the multiple reference pictures can be explicitly encoded and signaled to the decoder.

[0164] At this time, if the reference picture list contains only one reference picture, encoding / decoding of the reference picture index may be omitted.

[0165] The prediction direction information may be an index pointing to one of L0 unidirectional prediction, L1 unidirectional prediction, or bidirectional prediction. Alternatively, an L0 flag indicating whether prediction is performed in the L0 direction and an L1 flag indicating whether prediction is performed in the L1 direction may be encoded and signaled, respectively.

[0166] Motion Information Merge Mode is a mode in which the motion information of the current block is set to be identical to the motion information of neighboring blocks. In Motion Information Merge Mode, motion information can be encoded / decoded using a motion information merge list.

[0167] Motion information merging candidates can be derived based on motion information from neighboring blocks or neighboring samples adjacent to the current block. For example, after defining reference locations around the current block, it is possible to check whether motion information exists at the defined reference locations. If motion information exists at the defined reference locations, the motion information at those locations can be inserted into the motion information merging list as a motion information merging candidate.

[0168] In the example of Fig. 7, the predefined reference positions may include at least one of A0, A1, B0, B1, B5, and Col. Furthermore, motion information merging candidates may be derived in the order of A1, B1, B0, A0, B5, and Col.

[0169] Among the motion information merge candidates included in the motion information merge list, the motion information of the motion information merge candidate with the optimal cost can be set as the motion information of the current block. Furthermore, index information (e.g., a merge index) indicating the motion information merge candidate selected from among the multiple motion information merge candidates can be encoded and transmitted to the decoder.

[0170] In the decoder, a motion information merge list can be constructed in the same manner as in the encoder. Furthermore, motion information merge candidates can be selected based on the merge index decoded from the bitstream. The motion information of the selected motion information merge candidate can be set as the motion information of the current block.

[0171] Unlike the motion vector prediction list, the motion information merge list is composed of a single list regardless of the prediction direction. That is, the motion information merge candidates included in the motion information merge list may have only L0 motion information or only L1 motion information, or may have bidirectional motion information (i.e., L0 motion information and L1 motion information).

[0172] Motion information about the current block can also be derived using the restoration sample area surrounding the current block. Here, the restoration sample area used to derive motion information about the current block can be referred to as a template.

[0173] Figure 8 is a diagram for explaining a template-based motion estimation method.

[0174] In Fig. 4, it is described that the predicted block of the current block is determined based on the cost between the current block and the reference block within the search range. According to the present embodiment, unlike Fig. 4, motion estimation for the current block can be performed based on the cost between a template neighboring the current block (hereinafter referred to as the "current template") and a reference template having the same size and shape as the current template.

[0175] For example, the cost can be calculated based on the absolute sum of the differences between the restored samples in the current template and the restored samples in the reference block. A smaller absolute sum can reduce the cost.

[0176] Once the reference template with the optimal cost is determined within the search range, the reference block neighboring the reference template can be set as the predicted block of the current block.

[0177] And, motion information of the current block can be set based on the distance between the current block and the reference block, the index of the picture to which the reference block belongs, and whether the reference picture is included in the L0 or L1 reference picture list.

[0178] Since the template defines the restored area around the current block as a template, the decoder itself can perform motion estimation in the same manner as the encoder. Accordingly, when deriving motion information using a template, there is no need to encode and signal the motion information other than information indicating whether a template is used.

[0179] The current template may include at least one of a region adjacent to the top of the current block or a region adjacent to the left of the current block. The region adjacent to the top may include at least one row, and the region adjacent to the left may include at least one column.

[0180] Figure 9 shows examples of template configurations.

[0181] The current template can be configured according to one of the examples illustrated in FIG. 9.

[0182] Alternatively, unlike the example illustrated in FIG. 9, the template may be configured only with the area adjacent to the left of the current block, or only with the area adjacent to the top of the current block.

[0183] The size and / or shape of the current template may be predefined in the encoder and decoder.

[0184] Alternatively, after defining a plurality of template candidates having different sizes and / or shapes, index information specifying one of the plurality of template candidates can be encoded and signaled to a decoder.

[0185] Alternatively, one of multiple template candidates can be adaptively selected based on at least one of the size, shape, or position of the current block. For example, if the current block borders the upper boundary of the CTU, the current template can be constructed using only the region adjacent to the left of the current block.

[0186] Template-based motion estimation can be performed for each of the reference pictures stored in the reference picture list. Alternatively, motion estimation can be performed only for some of the reference pictures. For example, motion estimation can be performed only for reference pictures with a reference picture index of 0, or only for reference pictures with a reference picture index less than a threshold, or only for reference pictures with a POC difference from the current picture less than a threshold.

[0187] Alternatively, a reference picture index can be explicitly encoded and signaled, and then motion estimation can be performed only for the reference picture pointed to by the reference picture index.

[0188] Alternatively, motion estimation can be performed on the reference picture of the neighboring block corresponding to the current template. For example, if the template consists of a left adjacent region and an upper adjacent region, at least one reference picture can be selected using at least one of the reference picture index of the left adjacent block or the reference picture index of the upper adjacent block. Thereafter, motion estimation can be performed on the selected at least one reference picture.

[0189] Information indicating whether template-based motion estimation is applied can be encoded and signaled to a decoder. The information can be a 1-bit flag. For example, a true (1) flag indicates that template-based motion estimation is applied in the L0 direction and L1 direction of the current block. On the other hand, a false (0) flag indicates that template-based motion estimation is not applied. In this case, motion information of the current block can be derived based on a motion information merging mode or a motion vector prediction mode.

[0190] Conversely, template-based motion estimation may be applied only when it is determined that neither the motion information merging mode nor the motion vector prediction mode is applied to the current block. For example, if the first flag indicating whether the motion information merging mode is applied and the second flag indicating whether the motion vector prediction mode is applied are both 0, template-based motion estimation may be performed.

[0191] For each of the L0 and L1 directions, information indicating whether template-based motion estimation is applied can be signaled. That is, whether template-based motion estimation is applied in the L0 direction and whether template-based motion estimation is applied in the L1 direction can be determined independently. Accordingly, template-based motion estimation can be applied to one of the L0 and L1 directions, while another mode (e.g., motion information merging mode or motion vector prediction mode) can be applied to the other.

[0192] When template-based motion estimation is applied to both the L0 direction and the L1 direction, the prediction block of the current block can be generated based on a weighted sum operation of the L0 prediction block and the L1 prediction block. Alternatively, even when template-based motion estimation is applied to one of the L0 direction and the L1 direction, but another mode is applied to the other direction, the prediction block of the current block can be generated based on a weighted sum operation of the L0 prediction block and the L1 prediction block.

[0193] Alternatively, a template-based motion estimation method may be inserted as a motion information merging candidate in a motion information merging mode or a motion vector prediction candidate in a motion vector prediction mode. In this case, whether or not to apply a template-based motion estimation method may be determined based on whether the selected motion information merging candidate or the selected motion vector prediction candidate indicates a template-based motion estimation method.

[0194] Based on the two-way matching method, it is also possible to generate movement information of the current block.

[0195] Figure 10 is a diagram for explaining a motion estimation method based on a bilateral matching method.

[0196] The bilateral matching method can be performed only when the temporal order (i.e., POC) of the current picture exists between the temporal order of the L0 reference picture and the temporal order of the L1 reference picture.

[0197] When a bilateral matching method is applied, a search range can be set for each of the L0 reference picture and the L1 reference picture. At this time, an L0 reference picture index for identifying the L0 reference picture and an L1 reference picture index for identifying the L1 reference picture can be encoded and signaled, respectively.

[0198] As another example, only the L0 reference picture index may be encoded and signaled, and an L1 reference picture may be selected based on the distance between the current picture and the L0 reference picture (hereinafter referred to as the L0 POC difference). For example, among the L1 reference pictures included in the L1 reference picture list, an L1 reference picture having an absolute value of the distance from the current picture (hereinafter referred to as the L1 POC difference) equal to the absolute value of the distance between the current picture and the L0 reference picture may be selected. If there is no L1 reference picture having an L1 POC difference equal to the L0 POC difference, an L1 reference picture having an L1 POC difference most similar to the L0 POC difference may be selected among the L1 reference pictures.

[0199] At this time, among the L1 reference pictures, only L1 reference pictures that are temporally different from the L0 reference picture can be used for bilateral matching. For example, if the POC of the L0 reference picture is smaller than that of the current picture, one of the L1 reference pictures that has a POC larger than that of the current picture can be selected.

[0200] Conversely, one could also encode and signal only the L1 reference picture index, and select the L0 reference picture based on the distance between the current picture and the L1 reference picture.

[0201] Alternatively, a bilateral matching method may be performed using the L0 reference picture having the closest distance to the current picture among the L0 reference pictures and the L1 reference picture having the closest distance to the current picture among the L1 reference pictures.

[0202] Alternatively, a bilateral matching method may be performed using an L0 reference picture (e.g., index 0) assigned with a predefined index in the L0 reference picture list and an L1 reference picture (e.g., index 0) assigned with a predefined index in the L1 reference picture list.

[0203] Alternatively, the LX (X is 0 or 1) reference picture may be selected based on an explicitly signaled reference picture index, and the L|X-1| reference picture may be selected as a reference picture having the closest distance to the current picture among the L|X-1| reference pictures, or as a reference picture having a predefined index in the L|X-1| reference picture list.

[0204] As another example, L0 and / or L1 reference pictures can be selected based on motion information of neighboring blocks of the current block. For example, L0 and / or L1 reference pictures to be used for bilateral matching can be selected using the reference picture index of the neighboring block to the left or above the current block.

[0205] The search range can be set within a predetermined range from a collocated block within a reference picture.

[0206] As another example, the search range can be set based on initial motion information. This initial motion information can be derived from neighboring blocks of the current block. For example, the motion information of the left or top neighboring block of the current block can be set as the initial motion information of the current block.

[0207] When the bilateral matching method is applied, the L0 motion vector and the L1 motion vector are set to have opposite directions. This indicates that the signs of the L0 motion vector and the L1 motion vector have opposite signs. In addition, the size of the LX motion vector can be proportional to the distance between the current picture and the LX reference picture (i.e., the POC difference).

[0208] Thereafter, motion estimation can be performed using the cost between a reference block belonging to the search range of the L0 reference picture (hereinafter referred to as an L0 reference block) and a reference block belonging to the search range of the L1 reference picture (hereinafter referred to as an L1 reference block).

[0209] If an L0 reference block whose vector to the current block is (x, y) is selected, an L1 reference block located at a distance of (-Dx, -Dy) from the current block can be selected. Here, D can be determined by the ratio of the distance between the current picture and the L0 reference picture and the distance between the L1 reference picture and the current picture.

[0210] For example, in the example illustrated in FIG. 10, the absolute value of the distance between the current picture (T) and the L0 reference picture (T-1) and the absolute value of the distance between the current picture (T) and the L1 reference picture (T+1) are equal to each other. Accordingly, in the illustrated example, the L0 motion vector (x0, y0) and the L1 motion vector (x1, y1) have equal magnitudes but opposite distances. If an L1 reference picture with a POC of (T+2) were used, the L1 motion vector (x1, y1) would be set to (-2*x0, -2*y0).

[0211] Once the L0 reference block and L1 reference block with the optimal cost are selected, the L0 reference block and the L1 reference block can be set as the L0 prediction block and the L1 prediction block of the current block, respectively. Thereafter, the final prediction block of the current block can be generated through a weighted sum operation of the L0 reference block and the L1 reference block.

[0212] When the bilateral motion matching method is applied, the decoder can perform motion estimation in the same manner as the encoder. Accordingly, information indicating whether the bilateral motion matching method is applied can be explicitly encoded / decoded, while encoding / decoding of motion information such as motion vectors can be omitted. As previously explained, at least one of the L0 reference picture index or L1 reference picture index may also be explicitly encoded / decoded.

[0213] As another example, information indicating whether a bilateral matching method is applied may be explicitly encoded / decoded, and if a bilateral matching method is applied, the L0 motion vector or the L1 motion vector may be explicitly encoded and signaled. If the L0 motion vector is signaled, the L1 motion vector may be derived based on the POC difference between the current picture and the L0 reference picture and the POC difference between the current picture and the L1 reference picture. If the L1 motion vector is signaled, the L0 motion vector may be derived based on the POC difference between the current picture and the L0 reference picture and the POC difference between the current picture and the L1 reference picture. In this case, the encoder may explicitly encode the smaller one of the L0 motion vector and the L1 motion vector.

[0214] Information indicating whether a bilateral matching method has been applied may be a 1-bit flag. For example, a true flag (e.g., 1) may indicate that a bilateral matching method has been applied to the current block. A false flag (e.g., 0) may indicate that a bilateral matching method has not been applied to the current block. In this case, the current block may be subject to motion information merging mode or motion vector prediction mode.

[0215] Conversely, the bilateral matching method may be applied only when it is determined that neither the motion information merging mode nor the motion vector prediction mode is applied to the current block. For example, the bilateral matching method may be applied when both the first flag indicating whether the motion information merging mode is applied and the second flag indicating whether the motion vector prediction mode is applied are 0.

[0216] Alternatively, the bilateral matching method may be inserted as a motion information merging candidate in the motion information merging mode or as a motion vector prediction candidate in the motion vector prediction mode. In this case, whether the bilateral matching method is applied may be determined based on whether the selected motion information merging candidate or the selected motion vector prediction candidate indicates the bilateral matching method.

[0217] In the bidirectional matching method, it is exemplified that the temporal order of the current picture must exist between the temporal order of the L0 reference picture and the temporal order of the L1 reference picture. A unidirectional matching method, which is not subject to the constraints of the above bidirectional matching method, may also be applied to generate a prediction block of the current block. Specifically, in the unidirectional matching method, two reference pictures having a temporal order (i.e., POC) smaller than that of the current block or two reference pictures having a temporal order larger than that of the current block may be used. In this case, both reference pictures may be derived from the L0 reference picture list or the L1 reference picture list. Alternatively, one of the two reference pictures may be derived from the L0 reference picture list and the other may be derived from the L1 reference picture list.

[0218] Figure 11 is a diagram for explaining a motion estimation method based on a one-way matching method.

[0219] The unidirectional matching method can be performed based on two reference pictures having a POC smaller than that of the current picture (i.e., forward reference pictures) or two reference pictures having a POC larger than that of the current picture (i.e., backward reference pictures). In Fig. 11, motion estimation based on the unidirectional matching method is exemplified as being performed based on a first reference picture (T-1) and a second reference picture (T-2) having a POC smaller than that of the current picture (T).

[0220] At this time, a first reference picture index for identifying a first reference picture and a second reference picture index for identifying a second reference picture may be encoded and signaled, respectively. At this time, among the two reference pictures used in the unidirectional matching method, a reference picture having a smaller POC difference from the current picture may be set as the first reference picture. Accordingly, when the first reference picture is selected, only reference pictures included in the reference picture list having a larger POC difference from the current picture than the first reference picture may be set as the second reference picture. The second reference picture index may be set to point to an index of one of the rearranged reference pictures after rearranging reference pictures having the same temporal direction as the first reference picture and having a larger POC difference from the current picture than the first reference picture.

[0221] Conversely, the reference picture with a larger POC difference from the current picture among the two reference pictures can be set as the first reference picture. In this case, the second reference picture index can be set to point to the index of one of the rearranged reference pictures after rearranging the reference pictures that have the same temporal direction as the first reference picture and have a smaller POC difference from the current picture than the first reference picture.

[0222] Alternatively, a one-way matching method may be performed using a reference picture assigned with a predefined index within a reference picture list and a reference picture having the same temporal direction as the reference picture. For example, a reference picture having an index of 0 within the reference picture list may be set as the first reference picture, and a reference picture having the smallest index among the reference pictures having the same temporal direction as the first reference picture within the reference picture list may be selected as the second reference picture.

[0223] Both the first reference picture and the second reference picture can be selected from the L0 reference picture list or the L1 reference picture list. In Fig. 11, two L0 reference pictures are illustrated as being used in the unidirectional matching method. Alternatively, the first reference picture may be selected from the L0 reference picture list, and the second reference picture may be selected from the L1 reference picture list.

[0224] Information indicating whether the first reference picture and / or the second reference picture belongs to the L0 reference picture list or the L1 reference picture list may be additionally encoded / decoded.

[0225] Alternatively, one-way matching can be performed using one of the L0 reference picture list and the L1 reference picture list, whichever is set as default. Alternatively, two reference pictures can be selected from the L0 reference picture list and the L1 reference picture list, whichever has a larger number of reference pictures.

[0226] Afterwards, a search range within the first reference picture and the second reference picture can be set.

[0227] The search range can be set within a predetermined range from a collocated block within a reference picture.

[0228] As another example, the search range can be set based on initial motion information. This initial motion information can be derived from neighboring blocks of the current block. For example, the motion information of the left or top neighboring block of the current block can be set as the initial motion information of the current block.

[0229] Thereafter, motion estimation can be performed using the cost between the first reference block belonging to the search range of the first reference picture and the second reference block belonging to the search range of the second reference picture.

[0230] At this time, under the unidirectional matching method, the size of the motion vector should be set to increase in proportion to the distance between the current picture and the reference picture. Specifically, if a first reference block whose vector with the current picture is (x, y) is selected, the second reference block should be spaced apart from the current block by (Dx, Dy). Here, D can be determined by the ratio of the distance between the current picture and the first reference picture and the distance between the current picture and the second reference picture.

[0231] For example, in the example of FIG. 11, the distance between the current picture and the first reference picture (i.e., the POC difference) is 1, and the distance between the current picture and the second reference picture (i.e., the POC difference) is 2. Accordingly, when the first motion vector for the first reference block in the first reference picture is (x0, y0), the second motion vector (x1, y1) for the second reference block in the second reference picture can be set to (2x0, 2y0).

[0232] Once the first and second reference blocks with optimal costs are selected, the first and second reference blocks can be set as the first and second prediction blocks of the current block, respectively. Thereafter, a weighted sum operation of the first and second prediction blocks can be performed to generate the final prediction block of the current block.

[0233] When a unidirectional motion matching method is applied, the decoder can perform motion estimation in the same manner as the encoder. Accordingly, information indicating whether a unidirectional motion matching method is applied can be explicitly encoded / decoded, while encoding / decoding of motion information such as motion vectors can be omitted. As previously explained, at least one of the first reference picture index or the second reference picture index may also be explicitly encoded / decoded.

[0234] As another example, information indicating whether a unidirectional matching method is applied may be explicitly encoded / decoded, and if a unidirectional matching method is applied, the first motion vector or the second motion vector may be explicitly encoded and signaled. If the first motion vector is signaled, the second motion vector may be derived based on the POC difference between the current picture and the first reference picture and the POC difference between the current picture and the second reference picture. If the second motion vector is signaled, the first motion vector may be derived based on the POC difference between the current picture and the first reference picture and the POC difference between the current picture and the second reference picture. In this case, the encoder may explicitly encode a smaller one of the first motion vector and the second motion vector.

[0235] Information indicating whether a unidirectional matching method has been applied may be a 1-bit flag. For example, a true flag (e.g., 1) may indicate that a unidirectional matching method has been applied to the current block. A false flag (e.g., 0) may indicate that a unidirectional matching method has not been applied to the current block. In this case, the current block may be subject to motion information merging mode or motion vector prediction mode.

[0236] Conversely, the one-way matching method may be applied only when it is determined that neither the motion information merging mode nor the motion vector prediction mode is applied to the current block. For example, the one-way matching method may be applied when the first flag indicating whether the motion information merging mode is applied and the second flag indicating whether the motion vector prediction mode is applied are both 0.

[0237] Alternatively, the unidirectional matching method may be inserted as a motion information merging candidate in the motion information merging mode or as a motion vector prediction candidate in the motion vector prediction mode. In this case, whether the unidirectional matching method is applied may be determined based on whether the selected motion information merging candidate or the selected motion vector prediction candidate indicates the unidirectional matching method.

[0238] By adjusting the precision of motion vectors, it is also possible to detect the inter-frame movement of an object. Specifically, the position of each pixel within a picture is specified as an integer. However, the inter-frame movement of an object may not be expressed as an integer position.

[0239] Taking this into account, we can search for motion vectors in fractional pixel units by performing interpolation on the reference picture.

[0240] Figures 12 and 13 illustrate examples in which prediction blocks are generated according to the precision of a motion vector.

[0241] Figure 12 shows the location of the current block within the current picture, and Figure 13 shows an example in which a prediction block is obtained according to a motion vector.

[0242] Specifically, (a) of Fig. 13 shows an example in which the motion vector precision is in integer pixel units, and (b) and (c) of Fig. 13 show examples in which the motion vector precision is in 1 / 2 pixel units and 1 / 4 pixel units, respectively.

[0243] The motion vector precision can also be set in smaller units than those shown. For example, the motion vector precision can be set in units of 1 / 8 pixel, 1 / 16 pixel, or 1 / 32 pixel.

[0244] When the motion vector of the current block is expressed in integer units, a reference block composed of integer position samples can be set as a prediction block of the current block, as in the example illustrated in (a) of Fig. 13.

[0245] On the other hand, if the motion vector of the current block is expressed in fractional units, a reference block composed of fractional position samples can be set as the prediction block of the current block, as in the examples shown in (b) and (c) of Fig. 13. In this case, the fractional position samples within the reference block can be generated by interpolating integer position samples. The interpolation filter can have a size of 4 or 8 taps.

[0246] As another example, to reduce complexity, fractional position samples can be generated via linear interpolation using only integer position samples adjacent to the fractional position.

[0247] Information indicating the motion vector precision of the current block may be encoded and signaled. For example, after assigning different indices to each of a plurality of motion vector precision candidates, the index of the motion vector precision candidate corresponding to the motion vector precision of the current block may be encoded and signaled.

[0248] At this time, the number and / or types of available motion vector candidates can be determined based on at least one of the size of the current block, the shape of the current block, the reference picture, or the motion compensation model. Here, the motion compensation model can include at least one of a translation model, a zooming model, or a rotation model. A motion compensation model that combines a translation model with at least one of a zooming model or a rotation model can be referred to as an affine model.

[0249] An index indicating one of the available motion vector candidates for the current block may be encoded. Depending on the number of motion vector candidates available for the current block, the maximum number of bits required to encode the index may be determined.

[0250] By adjusting the precision of the motion vector, the motion vector can be searched more precisely, and thus the prediction accuracy for the current block can be improved.

[0251] Meanwhile, the motion vector expressed in fractional positions can be scaled up to integers and encoded.

[0252] Compensation for the motion of an object may be performed based on at least one of a translational model for compensating for linear motion of the object (e.g., motion in the horizontal and / or vertical direction), a zooming model for compensating for changes in the size of the object, and a rotational model for compensating for rotational motion of the object. Here, zooming may refer to enlargement or reduction in size.

[0253] Figure 14 shows an example in which motion compensation based on a translational model and a zooming model is performed for the current block.

[0254] For convenience of explanation, the current block is assumed to have a size of 4x4, as illustrated in Fig. 12.

[0255] In Figure 14, the variable α represents a size adjustment parameter. The size of the reference block can be derived by multiplying the size of the current block by the variable α.

[0256] A size adjustment parameter α less than 1 indicates that the reference block is smaller than the current block, and a size adjustment parameter α greater than 1 indicates that the reference block is larger than the current block.

[0257] Figures 14 (a) and (b) show examples when the size adjustment parameter α is less than 1, and Figure 14 (c) shows examples when the size adjustment parameter α is greater than 1.

[0258] Based on the motion vector of the current block, the upper left position of the reference block can be determined. Specifically, the upper left position of the reference block can be set as a position spaced apart by the motion vector from the position corresponding to the upper left sample of the current block within the reference picture. Thereafter, based on the resizing parameter, a reference block whose width and height are α times the width and height of the current block, respectively, can be set. Fractional position samples within the reference block can be generated by interpolating integer position samples.

[0259] A reference block derived by a motion vector and a scale parameter can be set as a prediction block of the current block.

[0260] Meanwhile, information about the scaling parameter α may be encoded and signaled. Specifically, a different index may be assigned to each of a plurality of scaling parameter candidates, and an index specifying the scaling parameter candidate applied to the current block may be encoded and signaled.

[0261] Alternatively, the resizing parameters of the current block can be derived based on the resizing parameters of neighboring blocks. For example, the resizing parameters of neighboring blocks at a predefined location can be set as the resizing parameters of the current block.

[0262] Alternatively, when sequentially searching multiple neighboring blocks, the size adjustment parameter of the first searched available neighboring block can be set as the size adjustment parameter of the current block.

[0263] Alternatively, the size adjustment parameter of a neighboring block may be set as a size adjustment parameter candidate. In this case, a plurality of neighboring blocks may be sequentially searched to generate a size adjustment parameter candidate list including a plurality of size adjustment parameter candidates. One of the plurality of size adjustment parameter candidates included in the plurality of size adjustment parameter candidate lists may be set as the size adjustment parameter of the current block. In this case, an index indicating a candidate among the plurality of size adjustment parameter candidates that is identical to the size adjustment parameter of the current block may be encoded and signaled.

[0264] Meanwhile, the neighboring blocks used to derive the size adjustment parameters of the current block may include at least one of an upper neighboring block, a left neighboring block, an upper-left neighboring block, an upper-right neighboring block, or a lower-left neighboring block.

[0265] Figure 15 shows an example in which motion compensation based on a translational model and a rotational model is performed for the current block.

[0266] For convenience of explanation, the current block is assumed to have a size of 4x4, as illustrated in Fig. 12.

[0267] First, as illustrated in (a) of Fig. 15, the location of a temporary block within a reference picture can be determined based on the motion vector of the current block. Specifically, a block location that is spaced apart by the motion vector from the location corresponding to the upper left sample of the current block within the reference picture can be determined as the upper left sample.

[0268] Thereafter, the temporary block can be rotated, as in the example illustrated in (b) of Fig. 15. The block at the rotated position can be set as a reference block, and the reference block can be set as a prediction block of the current block.

[0269] Meanwhile, a rotation matrix can be used to rotate a temporary block specified by a motion vector. That is, the prediction sample for the current block can be set to a sample at a position obtained by applying a rotation matrix to the sample position within the temporary block.

[0270] Mathematical expression 1 represents the rotation matrix.

[0271]

[0272] In the above mathematical expression 1, (pos_x, pos_y) represents the position of a sample within a temporary block. That is, (pos_x, pos_y) can be derived by adding a motion vector to the position of the target sample to be predicted within the current block.

[0273] (pos_x', pos_y') represents the rotated position from the position of the sample within the temporary block, and θ represents the rotation angle.

[0274] The sample value at the position (pos_x', pos_y') within the reference picture can be set as the value of the predicted sample for the position of the target sample. If the position (pos_x', pos_y') is a fractional position, the sample at that position can be generated by interpolating integer position samples.

[0275] Meanwhile, information indicating the rotation angle θ can be encoded and signaled. For example, after assigning a different index to each of a plurality of rotation angle candidates, the index of the rotation angle candidate corresponding to the rotation angle of the current block can be encoded and signaled.

[0276] Alternatively, the rotation angle of the current block can be derived based on the rotation angle of a neighboring block. For example, the rotation angle of a neighboring block at a predefined position can be set to the rotation angle of the current block.

[0277] Alternatively, when sequentially searching multiple neighboring blocks, the rotation angle of the first searched available neighboring block can be set to the rotation angle of the current block.

[0278] Alternatively, the rotation angle of a neighboring block may be set as a rotation angle candidate. In this case, a rotation angle candidate list including a plurality of rotation angle candidates may be generated by sequentially searching a plurality of neighboring blocks. One of the plurality of rotation angle candidates included in the plurality of rotation angle candidate lists may be set as the rotation angle of the current block. In this case, an index indicating a candidate among the plurality of rotation angle candidates having the same rotation angle as the current block may be encoded and signaled.

[0279] Meanwhile, the neighboring block used to derive the rotation angle of the current block may include at least one of an upper neighboring block, a left neighboring block, an upper left neighboring block, an upper right neighboring block, or a lower left neighboring block.

[0280] Although not shown, motion compensation for the current block can also be performed by simultaneously applying the translation model, zoom model, and rotation model.

[0281] Meanwhile, the motion vector precision for the current block or the number and / or types of motion vector precision candidates available for the current block may be determined differently depending on the motion compensation model.

[0282] For example, the number and / or type of motion vector precision candidates available for the current block may differ between cases where only the translational model is applied and cases where at least one of the zooming model or the rotational model is applied.

[0283] For example, if a translational model is applied to the current block, candidates larger than 1 / 4 pixel unit may be available for the current block. Conversely, if at least one zooming model or rotational model is additionally applied to the current block along with the translational model, candidates larger than 1 / 16 pixel unit may be available for the current block.

[0284] Alternatively, if a translational model is applied to the current block, the motion vector precision of the current block may be set to 1 / 4 pixel units. On the other hand, if at least one of a zooming model or a rotational model is additionally applied to the current block along with the translational model, the motion vector precision of the current block may be set to 1 / 16 pixel units.

[0285] Meanwhile, the encoder and decoder may have pre-stored the motion vector precisions or motion vector precision candidates available for each motion compensation model. Alternatively, information indicating the motion vector precisions or motion vector precision candidates available for each motion compensation model may be encoded and signaled through the upper header.

[0286] Motion compensation can be performed on an affine model, in which a zooming model and / or a rotation model are added to a translational model, using the motion vectors of the control points. Here, the control points may correspond to corners of the current block. For example, to perform motion compensation based on the affine model, at least one of the motion vector of the upper left corner, the motion vector of the upper right corner, or the motion vector of the lower left corner may be used.

[0287] Hereinafter, the motion vector of a control point will be referred to as a control point motion vector.

[0288] Figures 16 and 17 illustrate examples of generating a prediction block for a current block using control point motion vectors.

[0289] For convenience of explanation, the current block is assumed to have a size of 4x4, as illustrated in Fig. 12.

[0290] In (a) and (b) of FIG. 16, it is exemplified that a prediction block for the current block is derived by a motion vector of a first control point corresponding to the upper left corner of the current block (first control point motion vector, A) and a motion vector of a second control point corresponding to the upper right corner of the current block (second control point motion vector, B).

[0291] In addition to the example shown, it is also possible to derive a prediction block of the current block by additionally using the motion vector of the lower left corner or by using the motion vector of the lower left corner instead of the upper right corner.

[0292] Figure 18 shows an example of generating a prediction block for the current block using three control point motion vectors.

[0293] In (a) and (b) of FIG. 18, it is exemplified that a prediction block for the current block is derived by a motion vector of a first control point corresponding to the upper left corner of the current block (first control point motion vector, A), a motion vector of a second control point corresponding to the upper right corner of the current block (second control point motion vector, B), and a motion vector of a third control point corresponding to the lower left corner of the current block (third control point motion vector, C).

[0294] As in the examples illustrated in FIGS. 16 to 18, translational, zooming, and rotational motion compensation for the current block can be performed using two control point motion vectors or three control point motion vectors.

[0295] Information indicating the number of control point motion vectors may be encoded and signaled. The information may be signaled on a block-by-block basis. For example, the information may indicate whether two or three control point motion vectors are used in the current block.

[0296] Alternatively, the number of control point motion vectors can be adaptively determined based on at least one of the size or shape of the current block.

[0297] Alternatively, if the control point motion vectors of the current block are derived from neighboring blocks, the number of control point motion vectors for the current block may be set equal to the number of control point motion vectors of the neighboring blocks.

[0298] Using control point motion vectors, a motion vector for each sample within the current block can be derived. Equation 2 represents a formula for deriving a motion vector for each sample using two control point motion vectors.

[0299]

[0300] In the above mathematical expression 2, (mv x , mv y ) represents the motion vector at the (x, y) position within the current block. (mv Ax , mv Ay ) represents the first control point motion vector (A), and (mv Bx , mv By ) represents the second control point motion vector (B). W represents the width of the current block.

[0301] When three control point motion vectors are used, a motion vector per sample can be derived by the following mathematical expression 3.

[0302]

[0303] In the above mathematical expression 3, (mv Cx , mv Cy ) represents the third control point motion vector (C).

[0304] Once a motion vector is derived for each sample, motion compensation can be performed for each sample, as in the example illustrated in Fig. 17. Specifically, a reference sample indicated by the motion vector of the prediction target sample can be set as a prediction sample for the prediction target sample.

[0305] Meanwhile, if the motion vector of the prediction target sample is expressed in fractional units, integer position samples can be interpolated to generate fractional position samples, and the generated fractional position samples can be set as prediction samples for the prediction target sample.

[0306] At this time, the precision of the motion vector for each sample may be different. For example, the motion vector for the first prediction target sample may be derived in units of 1 / 2 pixels, while the motion vector for the second prediction target sample may be derived in units of 1 / 4 pixels.

[0307] In this case, fractional position samples can be generated according to the motion vector precision for each prediction target sample. Alternatively, the motion vector of the prediction target sample can be adjusted according to the reference motion vector precision, and then a prediction sample for the prediction target sample can be derived based on the adjusted motion vector. For example, if the reference motion vector precision is 1 / 2, the motion vector for the second prediction target sample can be adjusted in units of 1 / 4 pixels.

[0308] The reference motion vector precision can be determined on a block-by-block basis. Alternatively, the precision of control point motion vectors can be set as the reference motion vector precision. Alternatively, the reference motion vector precision can be predefined in the encoder and decoder.

[0309] As another example, to reduce complexity, motion vectors can be derived on a sub-block basis.

[0310] Figure 19 shows an example in which a motion vector is derived in sub-block units.

[0311] The size and / or shape of a sub-block may be predefined in the encoder and decoder. For example, a sub-block may be a square block of size 2x2 or 4x4.

[0312] Alternatively, the size and / or shape of the sub-block may be adaptively determined based on the size and / or shape of the current block. For example, if the current block is square, the sub-block may also be square. Conversely, if the current block is non-square, the sub-block may also be non-square.

[0313] Alternatively, information regarding at least one of the division method or division shape of the current block may be explicitly encoded and signaled. For example, information regarding at least one of the size of a sub-block, the shape of a sub-block, the position of a division line dividing the current block, or the number of division lines may be explicitly encoded and signaled. The information may be encoded and signaled on a block-by-block basis, or may be encoded and signaled via an upper header.

[0314] In Fig. 19, it is assumed that the sub-block is a square block of size 2x2.

[0315] A motion vector of a sub-block can be derived using coordinates of a predefined position within a sub-block. Here, the predefined position can be one of the positions of the upper left sample, the upper right sample, the lower left sample, the lower right sample, or the center position within the sub-block.

[0316] By substituting the coordinates of a predefined position within a sub-block into (x, y) in Equation 2, the motion vector of the sub-block can be derived.

[0317] As in the example described above, motion vectors can be derived in sub-block units based on the affine motion model.

[0318] Meanwhile, motion vectors can also be derived for each sub-block using collocated pictures. As described above, deriving motion vectors for each sub-block using collocated pictures can be called SbTMVP (Sub-block Temporal Motion Vector Prediction).

[0319] A collocated picture may be one of the reference pictures included in a reference picture list. For example, a picture with an index of 0 in the reference picture list may be selected as a collocated picture.

[0320] Alternatively, information indicating the index of a reference picture to be set as a collocated picture within the reference picture list may be explicitly encoded and signaled.

[0321] Figures 20 and 21 illustrate examples in which motion vectors are derived in units of sub-blocks within the current block when SbTMVP is applied.

[0322] The size and / or shape of the sub-block may be predefined in the encoder and decoder.

[0323] Alternatively, the size and / or shape of the sub-block may be adaptively determined based on the size and / or shape of the current block. For example, if at least one of the width or height of the current block is greater than a threshold, the size of the sub-block may be set to 8x8. Otherwise, the size of the sub-block may be set to 4x4.

[0324] Alternatively, information indicating the size and / or shape of the sub-block may be explicitly encoded and signaled.

[0325] In the example shown in Fig. 20, it is assumed that the size of the current block is 16x16 and the size of the sub-block is 4x4.

[0326] When SbTMVP is applied, an initial motion vector of the current block can be derived. The initial motion vector can be derived based on at least one of a motion vector prediction list or a motion information merge list. For example, an index indicating one of the motion vector prediction candidates included in the motion vector prediction list can be encoded and signaled. The initial motion vector can be derived by adding a motion vector differential value to the motion vector prediction candidate indicated by the index. Meanwhile, the motion vector differential value can also be explicitly encoded and signaled.

[0327] Alternatively, encoding of the index may be omitted, and a motion vector prediction candidate having a predefined index in the motion vector prediction list may be set as a prediction value for the initial motion vector. Here, the motion vector prediction candidate having a predefined index may be a motion vector prediction candidate having an index of 0 or a motion vector prediction candidate having the largest index.

[0328] Alternatively, an index indicating one of the motion information merging candidates included in the motion information merging list may be encoded and signaled. The initial motion vector may be set to be identical to the motion vector of the motion information merging candidate indicated by the index.

[0329] Alternatively, encoding of the index can be omitted, and the initial motion vector can be derived based on a motion information merge candidate having a predefined index in the motion information merge list. Here, the motion information merge candidate having a predefined index can be a motion information merge candidate having an index of 0 or a motion information merge candidate having the largest index.

[0330] Alternatively, the initial motion vector can be derived using the motion vector of a neighboring block at a predefined position. Here, the neighboring block at the predefined position can be a left neighboring block or an upper neighboring block.

[0331] The motion vector of a neighboring block at a predefined position can be set as a predicted value of the initial motion vector, and the difference value can be added to the predicted value to derive the initial motion vector.

[0332] Alternatively, the motion vector of a neighboring block at a predefined position can be set as the initial motion vector.

[0333] Alternatively, the initial motion vector can be derived using a template-based motion estimation method (i.e., template matching method) or bilateral matching.

[0334] The precision of the initial motion vector may be predefined in the encoder and decoder. For example, the precision of the initial motion vector may be fixed in integer pixel units.

[0335] Alternatively, information indicating the precision of the initial motion vector may be explicitly encoded and signaled. The information may be an index indicating one of a plurality of motion vector precision candidates.

[0336] When deriving an initial motion vector using a motion vector prediction candidate, the motion vector prediction candidates can be derived based on the motion vector precision of the initial motion vector. That is, the motion vector prediction candidate can be adjusted according to the motion vector precision of the initial motion vector, and then the adjusted initial motion vector prediction candidate can be inserted into the motion vector prediction list.

[0337] When deriving an initial motion vector using a motion information merge candidate, the motion information merge candidates can be derived based on the motion vector precision of the initial motion vector. That is, the motion information merge candidate can be adjusted according to the motion vector precision of the initial motion vector, and then the adjusted initial motion information merge candidate can be inserted into the motion information merge list.

[0338] Meanwhile, among the motion information merge candidates included in the motion information merge list, only those candidates whose reference picture is identical to the collocated picture of the current block can be used to derive the initial motion vector. That is, if the reference picture of a motion information merge candidate is different from the collocated picture of the current block, the initial motion vector may not be derived from the motion information merge candidate.

[0339] If there are multiple candidates among motion information merging candidates whose reference pictures are identical to the collocated pictures of the current block, an index indicating one of the multiple candidates may be encoded and signaled. Alternatively, if there are multiple candidates among motion information merging candidates whose reference pictures are identical to the collocated pictures of the current block, the initial motion vector may be derived from the candidate with the smallest index or the candidate with the largest index among the multiple candidates.

[0340] If a motion information merging candidate has both motion information in the L0 direction and motion information in the L1 direction, one of the motion information in the L0 direction and the motion information in the L1 direction can be selected according to a preset priority, and an initial motion vector can be derived from the selected motion information.

[0341] The priority may be determined based on at least one of the magnitude of the motion vector of the motion merging candidate, the index of the reference picture of the motion merging candidate, or whether the reference picture of the motion merging candidate is the same as the collocated picture.

[0342] Alternatively, it can be set to always derive the initial motion vector based on the motion information in the L0 direction.

[0343] When the initial motion vector is derived based on a template matching method, motion estimation can be performed according to the precision of the initial motion vector. For example, if the precision of the initial motion vector is integer pixel units, motion estimation based on template matching can also be performed only at integer positions.

[0344] Similarly, when the initial motion vector is derived based on bilateral matching, motion estimation can be performed according to the precision of the initial motion vector.

[0345] Meanwhile, as a result of the bilateral matching, a motion vector in the L0 direction (L0 motion vector) and a motion vector in the L1 direction (L1 motion vector) are derived. In this case, one of the L0 motion vector and the L1 motion vector can be set as the initial motion vector according to the preset priority.

[0346] Alternatively, it can be set to always derive the initial motion vector based on the motion information in the L0 direction.

[0347] Alternatively, information indicating which of the L0 motion vector and the L1 motion vector is set as the initial motion vector may be encoded and signaled.

[0348] Once the initial motion vector is derived, the position of the collocated block within the collocated block can be determined using the initial motion vector. For example, a block located at a position spaced apart by the initial motion vector from a position corresponding to the current block within the reference picture can be set as the collocated block. At this time, the position of the collocated block can be determined based on a predefined position within the current block. Here, the predefined position can be an upper left position, an upper right position, a lower left position, a lower right position, or a center position.

[0349] Depending on the division method of the current block, the collocated block can be divided into multiple collocated sub-blocks. In addition, the motion vector of each collocated sub-block within the collocated block can be set to the motion vector of each sub-block within the current block.

[0350] As another example, the initial motion vector can be used to determine the positions of the collocated sub-blocks corresponding to each sub-block within the current block within the collocated picture. In this case, the positions of the collocated sub-blocks can be derived based on predefined positions within the sub-blocks. Here, the predefined positions can be the upper left position, the upper right position, the lower left position, the lower right position, or the center position.

[0351] Thereafter, the motion vector of the collocated sub-block corresponding to the sub-block can be set as the motion vector of the sub-block. Specifically, the motion vector stored at a position corresponding to a predefined position within the collocated sub-block can be set as the motion vector of the sub-block.

[0352] Meanwhile, if the motion information of the collocated sub-block is unavailable, a predefined motion vector can be set as the motion vector of the sub-block. Here, the predefined motion vector can be a zero vector (i.e., (0, 0)) or an initial motion vector.

[0353] Alternatively, if the motion information of the collocated sub-block corresponding to the sub-block is not available, the motion vector of the sub-block may be derived from another location within the collocated sub-block.

[0354] Specifically, if a position corresponding to a predefined position within a collocated sub-block is encoded using intra prediction, then there is no motion vector at that position. For example, assuming that the predefined position is a central position (e.g., c10 in FIG. 21), if no motion vector is stored at the central position, the motion vector of the sub-block cannot be derived.

[0355] In this case, the motion vector of the sub-block can be derived based on the motion vector stored at a location different from the central location. Specifically, the motion vector of the sub-block can be derived based on the motion vector stored at a location adjacent to the central location (e.g., the upper adjacent location c6, the left adjacent location c9, or the upper left adjacent location c5).

[0356] Alternatively, if the center position is not available, the samples within the collocated sub-blocks may be searched according to the scan order, and the first available motion vector found may be set as the motion vector of the sub-block. Here, the scan order may be a horizontal scan, a vertical scan, a diagonal scan, or a raster scan.

[0357] Alternatively, if motion information of a collocated sub-block is unavailable, the motion vector of the sub-block may be set as the motion vector of the collocated block. For example, a motion vector stored at a position corresponding to a predefined position within the current block within the collocated block may be set as the motion vector of the sub-block.

[0358] As in the example described above, motion vectors can be derived for each sub-block using the affine motion model or SbTMVP. When motion vectors are derived for each sub-block, motion compensation can be performed for each sub-block based on the motion vector of each sub-block.

[0359] By performing motion compensation for each sub-block, a prediction block for the current block can be obtained. That is, the prediction block may be composed of prediction samples of each of the sub-blocks.

[0360] When detecting motion between frames, the precision of the motion vector can be adjusted. Specifically, the location of each sample within a picture is defined as an integer position. However, the position reflecting the motion may be a decimal position, not an integer position.

[0361] Taking this into account, we can search for motion vectors more precisely through reference picture interpolation.

[0362] Figures 22 and 23 are diagrams showing examples in which prediction blocks are derived according to motion vector precision.

[0363] Figure 22 shows the position of the current block within the current picture, and Figure 23 shows the position of the reference block according to the motion vector precision.

[0364] As in the examples illustrated in FIGS. 22 and 23, the motion vector of the current block can be defined as the distance from the sample corresponding to the upper left position of the current block in the reference picture to the sample corresponding to the upper left position of the reference block in the reference picture.

[0365] Figure 23 (a) illustrates a case where the motion vector precision of the current block is an integer pel, Figure 23 (b) illustrates a case where the motion vector precision of the current block is 1 / 2 pel, and Figure 23 (c) illustrates a case where the motion vector precision of the current block is 1 / 4 pel.

[0366] In Figure 23, motion vectors are expressed up to 1 / 4 vector precision, but motion vectors can also be expressed more precisely, such as 1 / 8, 1 / 16, or 1 / 32.

[0367] Meanwhile, information indicating the motion vector precision of the current block may be encoded and signaled. For example, the information may be an index identifying one of the motion vector precision candidates. Specifically, each of the motion vector precision candidates may be assigned a different index, and the information may indicate the index of the motion vector precision candidate applied to the current block.

[0368] By adjusting the precision of the motion vector used for inter-screen prediction, more precise motion vector search can be achieved. If the reference block indicated by the motion vector exists at a real number location, the samples at the real number location can be generated using samples at integer locations and an interpolation filter. Furthermore, motion vectors expressed as real numbers can be scaled up to integers and then encoded / decoded.

[0369] In this way, motion vectors (MV), motion vector predictors (MVPs), and motion vector differences (MVDs) can be encoded / decoded as integer values ​​through integerization. Specifically, motion vectors, motion vector predictors, and / or motion vector differences can be integerized based on motion vector precision.

[0370] For example, if the motion vector precision is 1 / N, integerization can be performed by multiplying the motion vector difference MVD by N. For example, if the motion vector difference MVD is (4 / 16, 8 / 16), the motion vector difference MVD can be integerized by multiplying by 16. That is, the integerized motion vector difference MVD can be expressed as (4, 8).

[0371] Based on the motion vector precision, the actual MVD can be derived from the integerized MVD. For example, if the motion vector precision is 1 / N, the actual MVD can be derived by dividing the integerized MVD by N. For example, if the integerized MVD is (4, 8) and the motion vector precision is 1 / 8, the actual MVD can be (4 / 8, 8 / 8). Alternatively, if the integerized MVD is (4, 8) and the motion vector precision is 1 / 4, the actual MVD can be (4 / 4, 8 / 4).

[0372] Depending on the motion vector precision, the range of expression of the integerized MVD may vary. For example, assume that the motion vector difference MVD is (4 / 16, 8 / 16) (i.e., (1 / 4, 2 / 4)). If the motion vector precision is 1 / 16, the integerized MVD is derived as (4, 8). On the other hand, if the motion vector precision is 1 / 4, the integerized MVD is derived as (1, 2).

[0373] Comparing the two cases above, if the motion vector precision is adjusted from 1 / 16 to 1 / 4, the integerized MVD value can be reduced from (4, 8) to (1, 2).

[0374] As a result, depending on the motion vector precision, the number of bits required to encode / decode the integerized motion vector difference MVD may vary. Accordingly, when encode / decode the motion vector difference MVD, a motion vector precision that can minimize the number of bins can be selected. Then, based on the selected motion vector precision, the motion vector difference MVD can be integerized, and the integerized motion vector difference MVD can be encode / decoded. In addition, information regarding the motion vector precision can be additionally encode / decode.

[0375] In the decoder, the actual MVD can be reconstructed from the decoded MVD based on the motion vector precision. Then, the motion vector MV can be derived by combining the reconstructed MVD and the motion vector prediction value MVP.

[0376] As above, the method of adjusting the value of the motion vector difference MVD to be encoded / decoded based on the motion vector precision is called AMVR (Adaptive Motion Vector Resolution).

[0377] Figures 24 and 25 are diagrams for explaining the process of encoding and decoding a motion vector difference value, respectively, when the AMVR method is applied.

[0378] For convenience of explanation, it is assumed that the motion vector and motion vector difference are expressed in units of 1 / 16 before integerization is performed, and 1 / 16 is expressed as the original motion vector precision.

[0379] The motion vector difference value MVD can be derived by differentiating the motion vector prediction value MVP from the motion vector MV (S2410).

[0380] The motion vector difference MVD may be composed of a horizontal direction component (i.e., x-axis component) and a vertical direction component (i.e., y-axis component).

[0381] If the motion vector difference value is 0, that is, if both the horizontal direction component and the vertical direction component are 0, the value of the motion vector difference value MVD to be encoded becomes 0 regardless of the motion vector precision. Therefore, if the motion vector difference value MVD is 0, encoding of AMVR-related information can be omitted (S2420).

[0382] On the other hand, if the motion vector difference is not 0, i.e., if at least one of the horizontal direction component and the vertical direction component is not 0, the motion vector precision can be determined (S2430). Meanwhile, the motion vector precision can be encoded as AMVR-related information.

[0383] Information related to AMVR may include at least one of a flag (e.g., amvr_flag) indicating whether the AMVR method is applied to the current block, and an index (e.g., amvr_prec_idx) indicating one of a plurality of motion precision candidates when the AMVR method is applied.

[0384] If the AMVR method is not applied to the current block, the motion vector precision can be set to a default value. In this case, amvr_flag can be encoded as a value of 0. Meanwhile, the default value can be 1, 1 / 2, 1 / 4, 1 / 8, or 1 / 16.

[0385] When the AMVR method is applied to the current block, an index indicating one of multiple motion vector precision candidates, i.e., amvr_prec_idx, may be additionally decoded. In this case, amvr_flag may be encoded with a value of 1, and amvr_prec_idx may be encoded with a value from 0 to (n-1), where n represents the number of motion vector precision candidates. For example, the multiple motion vector precision candidates may include at least one of 4, 2, 1, 1 / 2, 1 / 4, 1 / 8, or 1 / 16. Meanwhile, the default value may not be set to the multiple motion vector precision candidates indicated by the index. That is, when the motion vector precision of the current block is the default value, it is encoded and signaled with the value of amvr_flag, 0, and encoding of amvr_prec_idx may be omitted.

[0386] In the encoder, the optimal motion vector precision can be determined by performing Rate Distortion Optimization (RDO) for each combination of amvr_flag and amvr_prec_idx. That is, the combination with the optimal cost can be selected by performing RDO for the following cases.

[0387] 1) If amvr_flag is 0

[0388] 2) If amvr_flag is 1 and amvr_prec_idx is 0

[0389] 3) If amvr_flag is 1 and amvr_prec_idx is 1

[0390] 4) If amvr_flag is 1 and amvr_prec_idx is 2

[0391] Depending on the motion vector precision of the current block, a variable for scaling the motion vector difference, i.e., a scaling parameter, can be set. As an example, Table 1 illustrates the values ​​of the variable amvrshift according to the motion vector precision.

[0392] amvr_flagamvr_prec_idxamvrshift0 (1 / 4)-210 (1 / 2)311 (1-pel)412 (4-pel)6

[0393] If the finest motion vector precision that can be applied to the current block is 1 / 16, the motion vector precision can be expressed as in the following mathematical expression 4.

[0394]

[0395] As shown in Table 1, when the value of amvr_flag is 0, the variable amvrshift is set to 2. This indicates that the motion vector precision is 1 / 4 according to Equation 4.

[0396] When the value of amvr_flag is 1, the variable amvrshft can be determined according to the value of amvr_prec_idx. For example, when amvr_prec_idx is 1, the variable amvrshift is set to 4. This indicates that the motion vector precision is 1 according to Equation 4.

[0397] In the encoder, the motion vector difference value MVD can be scaled down and encoded using the variable amvrshift according to the motion vector precision. As an example, mathematical expression 5 shows an example of performing a scale down operation on the motion vector difference value MVD.

[0398]

[0399] In the above mathematical expression 5, MVD_x represents the horizontal component of the motion vector difference, and MVD_y represents the vertical component of the motion vector difference. MVD'_x and MVD'_y represent the results of the scale down operation.

[0400] The encoder can encode motion vector difference values ​​and AMVR information with changed precision (S2440).

[0401] In the decoder, the motion vector difference value MVD can be decoded (S2510).

[0402] If the motion vector difference value is 0, decoding of AMVR-related information is omitted, and the motion vector MV of the current block can be set to be the same as the motion vector prediction value (S2520).

[0403] On the other hand, if the motion vector difference is not 0, i.e., if at least one of the horizontal direction component and the vertical direction component is not 0, information related to AMVR can be additionally decoded (S2530).

[0404] Based on the AMVR information, a variable amvrshift can be derived for scaling the motion vector difference. For example, as shown in Table 1, a variable amvrshfit can be derived based on amvr_flag and / or amvr_prec_idx.

[0405] Thereafter, by using the variable amvrshift, the decoded MVD can be scaled up to obtain a motion vector difference MVD restored to its original precision (S2540). Mathematical expression 6 shows an example in which a scale-up operation is applied to the decoded MVD.

[0406]

[0407] In mathematical expression 6, MVD' represents the decoded motion vector difference. MVD represents the motion vector difference restored to its original precision, i.e., 1 / 16, through a scale-up operation.

[0408] Afterwards, the motion vector MV can be obtained by combining the motion vector difference MVD restored to the original precision and the motion vector prediction value MVP.

[0409] As in the example described above, when the motion vector prediction mode is applied, the decoder can derive the motion vector MV by combining the motion vector prediction value MVP and the motion vector difference value MVD.

[0410] If the motion vector of the current block points to a location outside the reference picture or a location adjacent to the boundary of the reference picture, there may be unavailable samples within the reference block (i.e., samples outside the boundary of the reference picture). Here, the reference picture refers to a picture reconstructed before the current picture.

[0411] Figure 26 illustrates an example in which there are unavailable samples within a reference block indicated by the motion vector of the current block.

[0412] For convenience of explanation, the sizes of the current block and reference block are assumed to be 4x4.

[0413] To ensure that there are no unavailable samples in the reference block, the encoder may determine, when performing motion estimation, that motion information indicating a reference block containing unavailable samples (e.g., motion information indicating a location outside the reference picture) is unavailable.

[0414] Alternatively, in the encoder, the search range for performing motion estimation can be set so as not to exceed the boundary of the reference picture.

[0415] By setting the encoder as above, it is possible to prevent unavailable samples from existing in the reference block indicated by the motion information of the current block.

[0416] As another example, padding can be performed on the outer area of ​​a reference picture to expand the reference picture, and motion estimation or motion compensation of the current block can be performed using the expanded reference picture. Here, padding can refer to the process of obtaining values ​​of samples within the expanded area.

[0417] Figure 27 is a diagram illustrating an extended reference picture.

[0418] In the example illustrated in Fig. 27, it is exemplified that an extension area exists around a reference picture, and the sizes of the extension areas are defined by W0, W1, H0, and H1. For example, W0 may indicate the size (i.e., width) of a left extension area of ​​the reference picture, and W1 may indicate the size (i.e., width) of a right extension area of ​​the reference picture. H0 may indicate the size (i.e., height) of an upper extension area of ​​the reference picture, and H1 may indicate the size (i.e., height) of a lower extension area of ​​the reference picture.

[0419] As a result, the extended reference picture can have the size of (W+W0+W1)x(H+H0+H1), where W represents the width of the reference picture and H represents the height of the reference picture.

[0420] Samples within the extended area can be generated through padding.

[0421] Padding can be performed by copying samples located at the boundaries of a reference picture within a reference picture. For example, samples within an upper extension area of ​​a reference picture can be obtained by copying samples located at the upper boundary within the reference picture, and samples within a lower extension area of ​​the reference picture can be obtained by copying samples located at the lower boundary within the reference picture. Padding for the upper extension area and the lower extension area can be defined as vertical padding.

[0422] Additionally, samples within the left extension area of ​​the reference picture may be obtained by copying samples located at the left border within the reference picture, and samples within the right extension area of ​​the reference picture may be obtained by copying samples located at the right border within the reference picture. Padding for the left extension area and the right extension area may be defined as horizontal padding.

[0423] Horizontal padding and vertical padding can be performed sequentially.

[0424] Figures 28 and 29 illustrate padding orders.

[0425] Figure 28 shows an example in which vertical padding is performed after horizontal padding is performed first.

[0426] As in the example illustrated in Fig. 28, first, horizontal padding can be performed to obtain horizontal extension areas, i.e., a left extension area and a right extension area.

[0427] Thereafter, padding can be performed for the upper extension area based on the samples located at the upper boundary within the horizontal extension area and the reference picture, and padding can be performed for the lower extension area based on the samples located at the lower boundary within the horizontal extension area and the reference picture.

[0428] Figure 29 shows an example in which horizontal padding is performed after vertical padding is performed primarily.

[0429] As in the example illustrated in Fig. 29, first, vertical padding can be performed to obtain vertical extension areas, i.e., upper extension areas and lower extension areas.

[0430] Thereafter, padding can be performed for the left extension area based on the samples located at the left border within the vertical extension area and the reference picture, and padding can be performed for the lower right extension area based on the samples located at the right border within the vertical extension area and the reference picture.

[0431] The padding order may be predefined in the encoder and decoder.

[0432] Alternatively, information indicating the padding order may be encoded and signaled. The information may be encoded and signaled for each reference picture.

[0433] Alternatively, information indicating a padding order for a sequence, slice, or picture may be encoded and signaled. In this case, the information indicating the padding order may be commonly applied to all reference pictures referenced by the sequence, slice, or picture.

[0434] Padding of the extended region of a reference picture can also be performed based on other reference pictures. Specifically, the other reference pictures to be referenced during padding can be determined based on the motion information stored in the reference picture. Hereinafter, the other reference pictures referenced during padding of the extended region of a reference picture will be referred to as the "source picture."

[0435] Figures 30 and 31 illustrate examples of obtaining an extended area of ​​a reference picture based on a source picture.

[0436] For convenience of explanation, the reference picture is assumed to have a POC of (T-1), and the source picture used to fill the areas corresponding to A, B, C, and D shown in Fig. 31 is assumed to have a POC of (T-2).

[0437] The source picture can be determined based on motion information stored in the reference picture. For example, the source picture can be identified based on motion information present at the boundary of the reference picture. Here, the motion information present at the boundary of the reference picture can represent motion information stored in a block located at the boundary of the reference picture.

[0438] Based on the motion information present in the reference picture, a reference area (i.e., a restoration area) within the source picture is specified. Thereafter, the specified reference area within the source picture can be filled into a padding target area within the expansion area. Here, the padding target area may refer to the entirety or a portion of the expansion area. In the example illustrated in Fig. 31, areas A, B, C, and D are exemplified as padding target areas.

[0439] At this time, the size and shape of the reference area within the source picture may differ depending on the location where the motion information within the reference picture is stored.

[0440] For example, as in the example illustrated in FIG. 31, when referring to motion information of a block (e.g., block a) located at the left and top boundaries within a reference picture, the top restoration area and the left restoration area around the reference block specified by the motion information can be set as reference areas for filling the padding target area adjacent to the block within the extended area.

[0441] For example, when referring to motion information of a block (e.g., block b) located at the upper boundary within a reference picture, an upper restoration area around the reference block specified by the motion information can be set as a reference area for filling a padding target area adjacent to the block within the extended area.

[0442] For example, when referring to motion information of a block (e.g., block c) located at the left border within a reference picture, a left restoration area around the reference block specified by the motion information can be set as a reference area for filling a padding target area adjacent to the block within the extended area.

[0443] For example, when referring to motion information of a block (e.g., block d) located at the left and bottom boundaries within a reference picture, the left restoration area and the bottom restoration area around the reference block specified by the motion information can be set as reference areas for filling the padding target area adjacent to the block within the extended area.

[0444] That is, when a reference block within a source picture is specified based on motion information of a block adjacent to the boundary of the reference block, a reference area around the reference block can be set in the same direction as the boundary to which the block touches.

[0445] The padding target area within the extended area of ​​the reference picture can be set to be the same as the reference area within the source picture. That is, in the example shown in Fig. 31, areas A', B', C', and D' within the source picture can be filled into areas A, B, C, and D within the extended area, respectively.

[0446] Meanwhile, in the example illustrated in Fig. 31, each of the four padding target regions is illustrated as referencing the same source picture (i.e., a reference picture with a POC of (T-2)). Depending on the motion information of blocks adjacent to the boundary of the reference picture, the source picture may be different.

[0447] Alternatively, a preset picture can be set as the source picture and a reference area for padding can be set only within the source picture. For example, the preset picture can be a collocated picture (collocated picture) of the reference picture.

[0448] In the example of Fig. 31, it is assumed that the reference picture with POC (T-2) is a call picture, and the motion information in the d region points to the reference picture with POC (T-3).

[0449] In this case, a scaling factor can be derived using the distance m between the reference picture (i.e., the reference picture with a POC of (T-1)) and the source picture (i.e., the call picture with a POC of (T-2)) and the distance n between the reference picture (i.e., the reference picture with a POC of (T-1)) and the reference picture indicated by the motion information of the d region (i.e., the reference picture with a POC of (T-3)). Thereafter, the derived scaling factor can be multiplied by the motion vector of the d region to derive a scaled motion vector.

[0450] Meanwhile, in the above example, the scaling factor can be derived as 1 / 2 (i.e., n / m). Accordingly, the scaled motion vector can be derived by multiplying the motion vector in the d region by 1 / 2.

[0451] Based on the scaled motion vector, a reference region can be set within the source picture (i.e., a call picture with a POC of (T-2). Once the reference region within the source picture is set, the set reference region can be set to the D region of the reference picture.

[0452] Alternatively, unlike the example above, if the motion information of region d does not point to the source picture, the motion information of region d may be determined to be unavailable. That is, padding may be performed on the padding target region using only the region in which the motion information points to the source picture. In Fig. 31, it is exemplified that regions A to D are padded based on the stored motion information of each of blocks a to d. Unlike the illustrated example, instead of the motion information stored in each of blocks a to d, motion information may be newly derived for each of blocks a to d. For example, motion estimation may be performed on block a within the source picture to directly derive the motion information of block a.

[0453] Meanwhile, the search for motion information existing within a reference picture can be performed in predefined sample units. Here, the predefined sample unit represents the distance between locations where the search is performed, and can be set to 4, 8, 16, or 32, etc.

[0454] At the upper and lower boundaries within a reference picture, horizontal search can be performed in predefined sample units. For example, if the predefined sample unit is 4, after searching for motion information at position (0, 0), motion information can be searched at position (4, 0).

[0455] At the left and right boundaries within a reference picture, vertical search can be performed in predefined sample units. For example, if the predefined sample unit is 4, after searching for motion information at position (0, 0), motion information can be searched at position (0, 4).

[0456] According to the predefined sample unit, the size of the padding target area within the expansion area can be determined. For example, when motion information is searched in units of 4 samples, an area of ​​size (4 x HX) within the expansion area adjacent to the position where the motion information is searched or an area of ​​size (WX x 4) within the expansion area can be set as the padding target area. Here, X can be an integer greater than or equal to 0. As described above, H0 and H1 can represent the heights of the upper expansion area and the lower expansion area, respectively, and W0 and W1 can represent the widths of the left expansion area and the right expansion area, respectively.

[0457] Meanwhile, if the location where the motion information is searched is included in a block located at the corner of the reference picture, padding may be performed not only for an area of ​​size (4 x HX) adjacent to the vertical direction of the block and an area of ​​size (WX x 4) adjacent to the horizontal direction of the block, but also for an area of ​​size (WX x HX) adjacent to the two areas above.

[0458] Meanwhile, the size of the block containing the location where the motion information is searched can also be determined based on the sample unit. For example, if the sample unit is N, the size of the block containing the location where the motion information is searched can be NxN.

[0459] Alternatively, when encoding / decoding a reference picture, the search unit and block size of motion information can be determined by referring to the tree partitioning structure. For example, padding can be performed using motion information stored in each of the coding blocks adjacent to the boundary of the reference picture, or padding can be performed by considering the block containing the position where the motion information is searched to be the corresponding coding block.

[0460] When bidirectional motion information exists in a reference picture, two reference areas can be set based on the bidirectional motion information. Specifically, based on the L0 motion information, an L0 reference area can be derived from the L0 reference picture, and based on the L1 motion information, an L1 reference area can be derived from the L1 reference picture.

[0461] Afterwards, the weighted sum of the L0 reference area and the L1 reference area can be set as samples of the padding target area.

[0462] Alternatively, one direction of the bidirectional motion information may be selected, and padding may be performed on the padding target area within the expansion area using only the motion information in the selected direction.

[0463] For example, a higher priority can be given to a reference block pointed to by a motion vector that is farther away. In other words, among the motion information in two directions, only the motion information in the direction with the larger motion vector size can be used.

[0464] Alternatively, a higher priority can be given to a reference block pointed to by a motion vector that is closer to the reference block. In other words, among the motion information in two directions, only the motion information in the direction with the smaller motion vector size can be used.

[0465] Alternatively, the priority can be determined based on the direction in which padding is performed (i.e., toward the left, right, top, or bottom of the picture). For example, if padding is performed toward the top of the picture, a motion vector pointing toward the top can be given higher priority. If both motion vectors point toward the top, the priority can be determined based on the magnitude of the motion vector.

[0466] For example, samples within an extended region can be derived using motion information in a predefined direction among L0 motion information and L1 motion information. Here, the predefined direction may be the L0 direction.

[0467] For example, one of the L0 motion information and the L1 motion information can be selected by comparing the distance between the current picture and the L0 reference picture indicated by the L0 motion information (hereinafter referred to as the L0 direction distance) and the distance between the current picture and the L1 reference picture indicated by the L1 motion information (hereinafter referred to as the L1 direction distance). Here, the distance between pictures can represent the POC (Picture Order Count) difference.

[0468] For example, if the L0 direction distance is greater than the L1 direction distance, samples within the extended region can be derived based on the L0 motion information. Conversely, if the L1 direction distance is greater than the L0 direction distance, samples within the extended region can be derived based on the L1 motion information.

[0469] Alternatively, conversely, the samples within the extended region can be derived using the motion information of the smaller of the L0 direction distance and the L1 direction distance.

[0470] Meanwhile, when the L0 direction distance and the L1 direction distance are the same, samples within the extended area can be derived using motion information in a preset direction. For example, the preset direction may be the L0 direction.

[0471] Alternatively, if the reference picture is included in the L0 reference picture list, L0 motion information may be selected from the bidirectional motion information, and if the reference picture is included in the L1 reference picture list, L1 motion information may be selected from the bidirectional motion information.

[0472] Meanwhile, the source picture may be an extended reference picture or an unextended reference picture.

[0473] If there is no previously restored picture before the reference picture, padding for the extended area can be performed using samples adjacent to the boundary of the reference picture.

[0474] When padding is performed on an extended area based on motion information stored in a reference picture, correction can be performed on the extended area or the reference area. Specifically, a correction parameter is derived based on the difference between a block in which motion information is stored within a reference picture (e.g., block a of FIG. 31) and a block indicated by the motion information of the block (e.g., block a' of FIG. 31), and the reference area (e.g., area A' of FIG. 31) corrected based on the correction parameter can be filled into a padding target area within the extended area (e.g., area A of FIG. 31).

[0475] The correction parameters may include offsets or weights. That is, the sample values ​​of the padding target region may be derived based on at least one of the offsets or weights.

[0476] Mathematical expression 7 shows an example of deriving an offset.

[0477]

[0478] In the above mathematical expression 7, average(a) represents the average value of samples included in a block in which motion information within a reference picture is searched (e.g., block a in FIG. 31), and average(a') represents the average value of samples included in a reference block within a source picture (e.g., block a' in FIG. 31).

[0479] Once the offset is derived, a correction can be performed to add (or subtract) the offset to all samples within the reference region (e.g., region A' of FIG. 31). Thereafter, the corrected samples can be set as samples within the padding target region (e.g., region A of FIG. 31).

[0480] Mathematical expression 8 shows an example of deriving weights.

[0481]

[0482] Once the weights are derived, a correction can be performed by multiplying (or dividing) all samples within the reference region by the weights. The corrected samples can then be set as samples in the padding target region.

[0483] Meanwhile, there may be cases where motion information does not exist at the boundary location of a reference picture. For example, if a block located at the boundary of a reference picture is encoded / decoded using intra prediction, the location occupied by that block does not have motion information.

[0484] Figure 32 shows an example where motion information of a block located at the boundary of a reference picture is not available.

[0485] In the example illustrated in Fig. 32, it is illustrated that no motion information exists at a location corresponding to block b. Meanwhile, a block for which motion information cannot be used may be referred to as an unavailable block.

[0486] In this case, padding based on motion information cannot be performed for the padding target region adjacent to block b (i.e. region B).

[0487] Accordingly, for a padding target region (e.g., region B) adjacent to an unavailable block (e.g., block b) in a reference picture, padding can be performed by copying samples at positions adjacent to the boundary of the corresponding block.

[0488] Alternatively, for padding target areas adjacent to unavailable blocks in the reference picture, padding can be performed by copying samples adjacent to the boundary of the adjacent padding target area (e.g., area A or area C).

[0489] For example, padding target area B can be created by copying samples located at the border between padding target area A and padding target area B among the samples included in padding target area A adjacent to the top.

[0490] For example, a padding target area B can be created by copying samples located at the border between padding target area C and padding target area B among the samples included in the lower adjacent padding target area C.

[0491] Alternatively, it may be obtained by weighting (or interpolating) some or all of the samples located at the boundary of the current block, the samples included in the first padding target area adjacent to the first direction of the padding target area, or the samples included in the second padding target area adjacent to the second direction of the padding target area.

[0492] For example, a padding target area B can be obtained by interpolating samples included in a padding target area A adjacent to the top and samples included in a padding target area C adjacent to the bottom. Here, the samples included in the padding target area A can be located at the boundary between the padding target area A and the padding target area B, and the samples included in the padding target area C can be located at the boundary between the padding target area C and the padding target area B.

[0493] Alternatively, motion information for unavailable blocks can be derived through motion information merging. Specifically, the motion information of a neighboring block adjacent to the unavailable block can be set as the motion information for the unavailable block.

[0494] Here, the neighboring block may be adjacent to the top, bottom, right or left of the unavailable block.

[0495] The location of the neighboring block may vary depending on the location of the unavailable block. For example, if the unavailable block is adjacent to the upper or lower boundary of the reference picture, motion information may be derived by referring to at least one of the neighboring blocks adjacent to the left or the right of the unavailable block. On the other hand, if the unavailable block is adjacent to the left or the right boundary of the reference picture, motion information may be derived by referring to at least one of the neighboring blocks adjacent to the top or the bottom of the unavailable block. If the unavailable block is located at a corner of the reference picture, motion information may be derived by referring to at least one of the neighboring blocks adjacent to the horizontal direction or the neighboring blocks adjacent to the vertical direction of the unavailable block.

[0496] When multiple referential neighboring blocks exist, neighboring blocks can be selected based on a predefined priority order. For example, if an unavailable block is located at the left border of the reference picture, motion information can be searched for in the order of the lower neighboring block and the upper neighboring block. If motion information exists in the lower neighboring block, the motion information of the lower neighboring block can be used. If motion information does not exist in the lower neighboring block, the motion information of the upper neighboring block can be used.

[0497] Alternatively, when there are multiple pieces of available motion information, the multiple pieces of motion information can be weighted or combined to derive bidirectional motion information. For example, when motion information exists in both the lower neighboring block and the upper neighboring block of a block without motion information, the motion vector of the lower neighboring block and the motion vector of the upper neighboring block can be weighted and set as the motion vector of the block without motion information. Meanwhile, the reference picture index of the block without motion information can be set to be the same as one of the lower neighboring block and the upper neighboring block.

[0498] Alternatively, if the lower neighboring block of an unavailable block has first-direction motion information and the upper neighboring block has second-direction motion information, the bidirectional motion information combining the two pieces of motion information can be set as the motion information of the block without motion information. Here, one of the first and second directions can represent the L0 direction, and the other can represent the L1 direction.

[0499] As another example, if there are multiple pieces of available motion information, all of the motion information can be used for the unavailable block. For example, the motion information of the lower adjacent block can be set as the motion information of the unavailable block, and a first reference area can be set based on the motion information. Furthermore, the motion information of the upper adjacent block can be set as the motion information of the unavailable block, and a second reference area can be set based on the motion information. Thereafter, the first reference area and the second reference area can be weighted and combined to obtain samples within the padding target area adjacent to the unavailable block.

[0500] Padding for extended areas can also be performed by performing template matching within the source picture.

[0501] Figure 33 illustrates an example of padding performed on an extended area through template matching. Unlike the form illustrated in Figure 8, the current template is a, the initial reference template is a', and the final reference template determined to be optimal is a''.

[0502] An initial position within a source picture can be set based on motion information of a block located at the boundary of a reference block. Specifically, a position spaced apart by a motion vector from the same position as the block within the source picture can be set as the initial position. For example, in the example illustrated in FIG. 33, reference block a' within the source picture is specified based on motion information of block a within the reference picture. The initial position can point to the upper left position or the center position of reference block a'.

[0503] Afterwards, a search area can be set based on the initial position, and motion estimation can be performed within the search area to determine the optimal reference block. The search area can be an area that includes the initial position. For example, a rectangular area centered on the initial position can be set as the search area, or a rectangular area with the initial position as its vertex can be set as the search area.

[0504] Alternatively, for simplicity, the initial position and four or eight directional positions around the initial position may be set as the search area. Here, the four directional positions may represent positions whose x-axis coordinate or y-axis coordinate differs by 1 from the initial position, and the eight directional positions may represent positions whose at least one of the x-axis coordinate or y-axis coordinate differs by 1 from the initial position.

[0505] For each location included in the search area, a template matching cost with a block within a reference picture can be calculated. Specifically, the template matching cost can be obtained based on a cost function between a block within the reference picture and a reference block of the same size as the block within the reference picture that includes the location within the search area within the source picture. Here, the cost function can include at least one of Sum of Absolute Differences (SAD), Sum of Absolute Transformed Differences (SATD), Sum of Squared Differences (SSD), or Mean-Removed Sum of Absolute Differences (MR-SAD).

[0506] Meanwhile, template matching can be performed in steps with decreasing precision.

[0507] For example, for each integer position within the search area, a template matching cost can be calculated. The integer position with the lowest cost can then be set as the optimal position.

[0508] Afterwards, the template matching cost can be calculated for the 1 / 2-pel fraction positions around the optimal position. Here, the 1 / 2-pel fraction positions can exist in 4 or 8 directions around the optimal position.

[0509] For 1 / 2-pel fraction positions, the cost between the current template and the reference template can be calculated. If the cost at the lowest 1 / 2-pel fraction position is less than the cost at the existing optimal position, the optimal position can be updated to that 1 / 2-pel fraction position. On the other hand, if the cost at the lowest 1 / 2-pel fraction position is not less than the cost at the existing optimal integer position, the optimal position can be maintained without being updated.

[0510] As described above, motion estimation based on template matching can be performed stepwise with decreasing precision to determine the optimal position. The final precision at which motion estimation based on template matching is performed can be 1 / 2-pel, 1 / 4-pel, 1 / 8-pel, or 1 / 16-pel.

[0511] Alternatively, to reduce complexity, motion estimation based on template matching can be performed only at integer positions.

[0512] Once the optimal location is determined, the restoration area around the reference block containing that location can be set as the reference area.

[0513] For example, in the example illustrated in FIG. 33, reference block a'' is illustrated as having the smallest template matching cost. Accordingly, restoration area A'' around reference block a'' can be set as the reference area, and reference area A'' can be set as the padding target area A.

[0514] Alternatively, N reference areas can be set in descending order of template matching cost, where N is a natural number greater than or equal to 1, for example, 2, 3, or 4.

[0515] For example, when N is 2, the restoration area around the first reference block with the smallest template matching cost can be set as the first reference area, and the restoration area around the second reference block with the next smallest template matching cost can be set as the second reference area. Thereafter, the weighted sum of the first reference area and the second reference area can be set as the samples within the padding target area.

[0516] Meanwhile, a reference picture can also be expanded by combining a padding method adjacent to the reference picture boundary with a padding method based on motion information. Here, the padding method based on motion information can refer to a method of setting a restoration area around a reference block indicated by motion information as a reference area (e.g., FIG. 31) or a method of setting a reference area through template matching.

[0517] Figure 34 shows an example of deriving an extended reference picture by combining multiple padding methods.

[0518] First, a reference picture can be expanded primarily through a padding method based on motion information. In the example illustrated in Fig. 34, it is illustrated that a primarily expanded reference picture of size (w0 + w1 + W) x (h0 + h1 + H) is generated through the primary expansion. That is, a padding method based on motion information can be applied to a left expansion area of ​​size w0, a right expansion area of ​​size w1, an upper expansion area of ​​size h0, and a lower expansion area of ​​size h1.

[0519] The reference picture can be expanded secondarily by copying samples located at the boundary of the first expanded reference picture. In the example illustrated in Fig. 34, it is exemplified that an expanded reference picture of the size (W0 + W1 + W) x (H0 + H1 + H) is generated by the second expansion. That is, a padding method of copying samples adjacent to the boundary of the first expanded area can be applied to a left expanded area of ​​the size (W0-w0), a right expanded area of ​​the size (W1-w1), an upper expanded area of ​​the size (H0-h0), and a lower expanded area of ​​the size (H1-h1).

[0520] Alternatively, information indicating the padding method may be encoded and signaled. The information may be encoded and signaled for each reference picture.

[0521] Alternatively, information indicating a padding scheme for a sequence, slice, or picture may be encoded and signaled. In this case, the information indicating the padding scheme may be commonly applied to all reference pictures referenced by the sequence, slice, or picture.

[0522] In the above example, it is illustrated that a reference picture is expanded to create an expanded reference picture, and inter prediction (i.e., motion compensation) is performed for the current block using the expanded reference picture.

[0523] At this time, information indicating whether to expand the reference picture may be encoded and signaled. The information may be encoded and signaled for each reference picture.

[0524] Alternatively, information indicating whether a sequence, slice, or picture is to be extended may be encoded and signaled. In this case, the information indicating whether the sequence, slice, or picture is to be extended may be commonly applied to all reference pictures referenced by the sequence, slice, or picture.

[0525] Alternatively, information indicating whether to extend a picture, i.e., a current picture that is currently being encoded / decoded, may be encoded and signaled. For example, if the information indicates that the current picture should be extended, the extension area of ​​the current picture can be acquired during encoding / decoding of the current picture, or after encoding / decoding of the current picture is completed. Thereafter, the extended current picture can be stored in a reference picture list. The extended current picture stored in the reference picture list can be used as a reference picture for a picture to be encoded / decoded in the future.

[0526] In another example, instead of expanding the size of the reference picture, the original reference picture is used, but if the reference block indicated by the motion information of the current block includes a sample included outside the reference picture, the sample included outside the reference picture can be derived by the padding method described above.

[0527] Meanwhile, considering the case where samples outside the boundary of the current picture are used, padding can also be used to derive samples outside the boundary of the current picture.

[0528] For example, if the current block is adjacent to the upper boundary or the left boundary of the current picture, samples outside the boundary of the current picture must be used as reference samples when performing intra prediction for the current block.

[0529] Alternatively, if the block vector of the current block points outside the current picture, the reference block pointed to by the block vector contains samples that are outside the boundary of the current picture.

[0530] In the above cases, the prediction accuracy of the current block can be improved by generating samples that are outside the boundary of the current picture through padding.

[0531] Figure 35 shows an example of generating samples that fall outside the boundaries of the current picture through padding.

[0532] In the example of Fig. 35, the current block is shown as being adjacent to the left border of the current picture, and the area existing on the left side of the current block is set as the padding target area.

[0533] If the current block is adjacent to the boundary of the current picture, template matching can be performed based on a block that has been encoded / decoded before the current block (hereinafter referred to as the previous block).

[0534] Meanwhile, the previous block may be adjacent to the current block and adjacent to the boundary of the current picture.

[0535] A search area within a source picture can be set based on the motion information of the previous block or the location of the previous block, and a reference block with the smallest cost compared to the previous block can be searched within the search area.

[0536] If motion information is stored in the previous block, the initial position within the source picture can be set based on the motion information of the previous block.

[0537] On the other hand, if motion information is not stored in the previous block, the motion vector of the previous block is set to a zero vector, and a predefined reference picture is set as the source picture. For example, the source picture may be a collocated picture.

[0538] Motion estimation based on template matching determines the reference block with the lowest cost relative to the previous block within the search region. Then, a reference region within the source picture can be established based on the positional difference between the previous block and the padding target region surrounding the current block. For example, reference region b may be spaced from reference block a by the distance between the previous block A and the padding target region B.

[0539] Afterwards, the reference area can be copied to the padding target area to obtain samples within the padding target area.

[0540] Alternatively, the motion information of a block adjacent to the current block can be set as the motion information of the current block, and then the reference template most similar to the current template can be searched to set the reference area.

[0541] At this time, the current template may consist of only the upper restoration region or only the left restoration region, depending on the location of the current block. For example, if the current block is adjacent to the left border of the picture, the current template may consist of only the upper restoration region, and if the current block is adjacent to the upper border of the picture, the current template may consist of only the left restoration region.

[0542] Meanwhile, if the current block is the first block in the current picture (i.e., block A in FIG. 35), there is no previous block. In this case, the padding target area around the current block can be filled with a preset value.

[0543] The preset value may be a value predefined in the encoder and decoder.

[0544] Alternatively, the preset value may be a value derived in advance based on the bit depth. For example, the preset value may be a median value according to the bit depth. For example, if the bit depth is N bits, the median value may represent a value of 1<<(N-1). That is, if the bit depth is 10 bits, the median value may be 512, and if the bit depth is 8 bits, the median value may be 128.

[0545] As described above, samples derived through padding can be set as reference samples for intra prediction of the current block.

[0546] Meanwhile, the padding method described above can be performed not only at picture boundaries but also at boundaries of independent processing units. Here, an independent processing unit can represent a subpicture, slice, tile, or CTU.

[0547] For example, if the current block is adjacent to the upper boundary of the current CTU, the upper region of the current block can be set as the padding target region. Here, the current CTU represents the CTU that includes the current block. Thereafter, a reference region within the source picture can be set based on motion information of the neighboring block adjacent to the upper or left of the current block.

[0548] Padding can be performed on the current picture only if the encoding type of the current picture indicates that inter prediction can be performed. That is, if the current picture is of type P or type B, padding can be performed on the current picture, but if the current picture is of type I, padding is not performed on the current picture.

[0549] Based on the block vectors of neighboring blocks, it is also possible to generate samples that fall outside the boundaries of the current picture.

[0550] Specifically, if a block vector is stored in a neighboring block adjacent to the current block, the reference block indicated by the block vector of the neighboring block within the current picture can be determined. Thereafter, a reference area can be set by considering the positional difference between the neighboring block and the padding target area. Specifically, an area located as far from the reference block as the positional difference between the neighboring block and the padding target area can be set as the reference area.

[0551] Once encoding / decoding for the current picture is completed, the current picture can be used as a reference picture for inter prediction of the next picture. To this end, the current picture, for which encoding / decoding has been completed, can be inserted into the picture buffer.

[0552] Meanwhile, if padding has been performed on the current picture, the padded current picture can be inserted into the picture buffer. That is, a picture with a size larger than the original size can be stored in the picture buffer.

[0553] Alternatively, instead of storing the padded current picture in the picture buffer, a method (e.g., FIGS. 27 to 34) for generating an extended reference picture for a current picture whose encoding / decoding has been completed may be applied to generate the extended current picture. The extended current picture may be stored in the picture buffer and used as a reference picture for a picture to be encoded / decoded in the future.

[0554] Alternatively, the padded area using motion information around the current picture may be maintained as is, and an extended reference picture may be generated by applying a method (e.g., FIGS. 27 to 34) for generating an extended reference picture for an unpadded area around the current picture or an area padded without using motion information, thereby generating an extended current picture.

[0555] Alternatively, the padding result for the padding target area adjacent to the boundary position of the current picture where no motion information exists may be maintained as is, and for the padding target area adjacent to the boundary position of the current picture where motion information exists, padding may be newly performed for the padding target area based on the motion information at the boundary position.

[0556] Meanwhile, the size of the extension area for the current picture may be set to be the same as the size of the extension area for the reference picture. Alternatively, information indicating the size of the extension area for the current picture or the reference picture may be encoded and signaled through the upper header.

[0557] Alternatively, the size of the extension area for the current picture may be smaller than or equal to the size of the extension area for the reference picture. Accordingly, the size of the extended current picture becomes smaller than the size of the expanded reference picture. To make the size of the extended current picture the same as that of the extended reference picture, additional padding may be performed on the extended current picture when storing the extended current picture in the picture buffer. That is, additional padding may be performed on the additional extension area by the size difference between the extended reference picture and the extended current picture.

[0558] At this time, additional padding can be performed by copying samples located at the boundaries of the extended current picture to the additional extended area.

[0559]

[0560] Applying the embodiments described above, focusing on the decoding or encoding process, to the encoding or decoding process is within the scope of the present disclosure. Changing the embodiments described above, in a given order, to a different order is also within the scope of the present disclosure.

[0561] Although the above-described disclosure is described based on a series of steps or a flowchart, this does not limit the chronological order of the invention, and may be performed simultaneously or in a different order as needed. In addition, each component (e.g., unit, module, etc.) constituting the block diagram in the above-described disclosure may be implemented as a hardware device or software, or multiple components may be combined to be implemented as a single hardware device or software. For example, the hardware device may include at least one of a processor for performing calculations, a memory for storing data, a transmitter for transmitting data, and a receiver for receiving data.

[0562] The above-described disclosure may be implemented in the form of program commands that can be executed by various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination.

[0563] In addition, according to the present disclosure, a computer-readable recording medium can be provided that stores a bitstream generated by the above-described encoding method. The bitstream can be transmitted by an encoding device, and a decoding device can receive the bitstream and decode an image.

[0564] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. The hardware devices may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa.

[0565] The present disclosure may be applied to a computing or electronic device capable of encoding / decoding a video signal.

Claims

1. A step of determining the movement information of the current block; and A step of obtaining a prediction block of the current block based on the motion information of the current block, The above prediction block is derived from an extended reference picture, A video decoding method, characterized in that the extended reference picture is composed of a reference picture restored before the current picture and an extended area around the reference picture.

2. In paragraph 1, A video decoding method characterized in that samples located at the boundary of the above reference picture are padded to derive samples within the extended area.

3. In paragraph 2, The above extension area includes a horizontal direction restoration area adjacent to the horizontal direction of the reference picture and a vertical direction restoration area adjacent to the vertical direction of the reference picture, An image decoding method, characterized in that the above padding is performed primarily for one of the horizontal and vertical directions and then secondarily for the other.

4. In paragraph 1, Based on the motion information of the boundary position of the above reference picture, a reference area within the source picture is set, An image decoding method, characterized in that padding is performed on a padding target area adjacent to the boundary position based on the above reference area.

5. In paragraph 4, An image decoding method, characterized in that, if the motion information does not exist at the boundary location, the reference area is set based on previously searched motion information.

6. In paragraph 5, A video decoding method, characterized in that motion information is searched for in predefined sample units at the boundary of the above reference picture.

7. In paragraph 4, A video decoding method, characterized in that the above source picture is a picture with an extended boundary.

8. In paragraph 4, If there is no restored picture before the above reference picture, An image decoding method characterized in that samples located at the boundary of the above reference picture are padded to derive samples within the extended area.

9. In paragraph 4, An image decoding method characterized in that, when the motion information of the boundary position has bidirectional motion information, the reference area is set using only unidirectional motion information among the bidirectional motion information.

10. In paragraph 9, If the above reference picture belongs to the L0 reference picture list, the unidirectional motion information is in the L0 direction, A video decoding method, wherein the unidirectional motion information is in the L1 direction when the above reference picture belongs to the L1 reference picture list.

11. In paragraph 4, If the motion information of the above boundary position has bidirectional motion information, an L0 direction reference area and an L1 direction reference area are derived based on the bidirectional information, An image decoding method, characterized in that padding is performed on the padding target area through a weighted sum of the L0 direction reference area and the L1 direction reference area.

12. In paragraph 1, The above expansion area includes a primary expansion area and a secondary expansion area, The above first extension area is padded based on motion information existing at the reference picture boundary, An image decoding method, characterized in that the second expansion area is padded based on samples located at the boundary of the first expansion area.

13. In paragraph 1, A video decoding method, characterized in that information indicating whether to use the extended reference picture is signaled through a bitstream.

14. A step for determining the movement information of the current block; and A step of obtaining a prediction block of the current block based on the motion information of the current block, The above prediction block is derived from an extended reference picture, A video encoding method, characterized in that the extended reference picture is composed of a reference picture restored before the current picture and an extended area around the reference picture.

15. A processor for obtaining compressed video data; and Including a transmission unit for transmitting the compressed video data, The above compressed video data is, A step of determining movement information of the current block; and Based on the motion information of the current block, a prediction block of the current block is obtained through a step of obtaining the prediction block of the current block, The above prediction block is derived from an extended reference picture, A device for transmitting compressed video data, characterized in that the extended reference picture comprises a reference picture restored before the current picture and an extended area around the reference picture.

Images