Image encoding / decoding method and device

Abstract

The present disclosure relates to an image encoding / decoding method and device, and a recording medium, the method comprising the steps of: determining a filter for interpolating a reference picture of a current block in inter prediction of a current video; interpolating the reference picture with the determined filter so as to obtain an interpolated reference picture; and performing motion prediction on the basis of the interpolated reference picture, wherein the filter for interpolating the reference picture can be determined by comparing the size of the current block, the absolute correlation value of the reference block of the current block, or the mean sum of absolute difference (SAD) value of the reference block with a threshold value.

Description

Video encoding / decoding method and device

[0001] The present disclosure proposes an intra-subpartition based intra-screen prediction technique for effectively performing intra-screen prediction.

[0002] The present disclosure proposes a technique of an adaptive reference block interpolation filter for effective inter-screen prediction.

[0003] Recently, the demand for multimedia data, such as video, has been rapidly increasing on the Internet. However, the pace of growth in channel bandwidth is struggling to keep pace with the rapidly increasing volume of multimedia data.

[0004] Accordingly, since a technology with higher encoding / decoding efficiency is required, the present disclosure proposes improved prediction technologies compared to conventional technologies.

[0005] This disclosure aims to overcome the limitations of existing intra-subpartition-based intra-screen prediction techniques. Specifically, existing intra-subpartition-based intra-screen prediction techniques have limitations in that they partition a single CU into K subpartitions and then apply the same intra-screen prediction mode to the subpartitions. Furthermore, the subpartitions cannot apply different transformation kernels, and their partitioning methods are also limited.

[0006] The present disclosure seeks to overcome the limitations of existing interpolation filters by providing an interpolation filter that adaptively improves encoding / decoding efficiency according to the frequency characteristics of a block.

[0007] The video encoding / decoding method, device, and recording medium of the present disclosure include, in inter-screen prediction of a current video, a step of determining a filter for interpolating a reference picture of a current block, a step of interpolating the reference picture with the determined filter to obtain an interpolated reference picture, and a step of performing motion prediction based on the interpolated reference picture, wherein the filter for interpolating the reference picture may be determined by comparing the size of the current block, the absolute value of the correlation of the reference block of the current block, or the Mean SAD (Sum of Absolute Difference) value of the reference block with a threshold value.

[0008] In the video encoding / decoding method, device, and recording medium of the present disclosure, when the filter for interpolating the reference picture is determined by comparing the size of the current block with a threshold value, the filter for interpolating the reference picture in response to the size of the current block being greater than the threshold value may be determined as an 8-tap or 6-tap interpolation filter, and the filter for interpolating the reference picture in response to the size of the current block being less than or equal to the threshold value may be determined as a 12-tap interpolation filter.

[0009] In the video encoding / decoding method, device and recording medium of the present disclosure, the threshold value may vary depending on the size of the current block.

[0010] In the video encoding / decoding method, device and recording medium of the present disclosure, the MeanSAD value of the reference block can be calculated based on the width and height of the reference block and the sample values ​​of the reference block.

[0011] In the video encoding / decoding method, device and recording medium of the present disclosure, in response to the MeanSAD value of the reference block being greater than a threshold value, the filter interpolating the reference picture may be determined as a 12-tap interpolation filter or an interpolation filter longer than 12 taps.

[0012] In the video encoding / decoding method, device, and recording medium of the present disclosure, when the resolution of the current video is less than 4K, the filter for interpolating the reference picture is determined by comparing the absolute value of the correlation of the reference block with a threshold value, and when the resolution of the current video is 4K or more, the filter for interpolating the reference picture may be determined as an 8-tap interpolation filter or a 6-tap interpolation filter.

[0013] In the video encoding / decoding method, device, and recording medium of the present disclosure, when the absolute correlation value of the reference block is greater than or equal to a threshold value, the filter for interpolating the reference picture may be determined as an 8-tap interpolation filter or a 6-tap interpolation filter, and when the absolute correlation value of the reference block is less than the threshold value, the filter for interpolating the reference picture may be determined as a 12-tap interpolation filter.

[0014] The intra-subpartition based intra-screen prediction technique of the present disclosure is more adaptively applied to a situation than the conventional intra-subpartition based intra-screen prediction technique, and thus can improve encoding / decoding efficiency in encoding / decoding of an image.

[0015] FIG. 1 is a block diagram showing an image encoding device according to an embodiment of the present invention.

[0016] FIG. 2 is a block diagram showing an image decoding device (200) according to one embodiment of the present invention.

[0017] Figure 3 shows a flowchart of intra-subpartition (ISP)-based intra-screen prediction.

[0018] Figure 4 is a diagram showing adjacent blocks used to generate an MPM list.

[0019] Figure 5 is a diagram showing an example of an on-screen prediction mode.

[0020] Figure 6 is an example diagram illustrating a CU division method in ISP-based on-screen prediction.

[0021] FIG. 7 is a diagram illustrating an example of using a previous subpartition in configuring an MPM list of the current subpartition.

[0022] FIG. 8 is a diagram illustrating an embodiment of implicitly determining a combination of transformation kernels according to a splitting direction and a prediction mode of a subpartition.

[0023] FIG. 9 is a diagram illustrating an example of performing a transformation by making a non-rectangular subpartition into a rectangular shape.

[0024] Figure 10 shows an example of an equation for filter coefficients.

[0025] Figure 11 shows examples of formulas for DCT-II coefficients and IDCT-II coefficients.

[0026] Fig. 12 is a diagram showing the relationship between the Magnitude and Frequency graphs representing the frequency characteristics at the 1 / 2 Luma sample position of a 6-tap Gaussian-based IF, an 8-tap DCT-IF, and the proposed 12-tap DCT-IF.

[0027] Figure 13 shows an example of an auto-correlation formula.

[0028] Figure 14 shows an example of 12-tap DCT-IF coefficients.

[0029] Figure 15 shows the locations of fractional samples and integer samples.

[0030] Figure 16 shows an example of generating a sample at position b0 as an example of generating a fractional sample.

[0031] Figure 17 shows the experimental results of the performance of the proposed method.

[0032] Figure 18 shows an example of the Mean SAD formula.

[0033] Figure 19 illustrates a formula for a method using correlation to find out the frequency characteristics of a reference block.

[0034] Figure 20 shows a formula for a method using Mean SAD to find out the frequency characteristics of a reference block.

[0035] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

[0036] Terms such as first, second, A, and B may be used to describe various components, but these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0037] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0038] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0039] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted consistent with their meaning in the context of the relevant technology, and shall not be construed in an idealized or overly formal sense unless explicitly defined herein.

[0040] Typically, an image can be composed of a series of still images, and these still images can be divided into GOP (Group of Pictures) units, and each still image can be referred to as a picture or a frame. Higher-level concepts can exist in units such as GOP and sequence, and each picture can also be divided into predetermined areas such as slices, tiles, and blocks. In addition, one GOP can include units such as I-picture, P-picture, and B-picture. An I-picture can mean a picture that is encoded / decoded on its own without using a reference picture, and a P-picture and a B-picture can mean a picture that is encoded / decoded by performing processes such as motion estimation and motion compensation using a reference picture. In general, for P pictures, I pictures and P pictures can be used as reference pictures, and for B pictures, I pictures and P pictures can be used as reference pictures, but this definition above can also be changed depending on the encoding / decoding settings.

[0041] Here, the picture referred to for encoding / decoding is called a reference picture, and the block or pixel referred to is called a reference block or reference pixel. In addition, the reference data may be not only a pixel value in the spatial domain, but also a coefficient value in the frequency domain, and various encoding / decoding information generated and determined during the encoding / decoding process.

[0042] The smallest unit that constitutes an image can be a pixel, and the number of bits used to express one pixel is called bit depth. Generally, the bit depth can be 8 bits, and other bit depths can be supported depending on the encoding settings. The bit depth can support at least one bit depth depending on the color space. In addition, it can be composed of at least one color space depending on the color format of the image. Depending on the color format, it can be composed of one or more pictures with a certain size or one or more pictures with different sizes. For example, in the case of YCbCr 4:2:0, it can be composed of one luminance component (Y in this example) and two chrominance components (Cb / Cr in this example), and in this case, the composition ratio of the chrominance component and the luminance component can have a horizontal and vertical ratio of 1:2. As another example, in the case of 4:4:4, it can have the same horizontal and vertical composition ratio. When a picture is composed of one or more color spaces as in the above example, division into each color space can be performed.

[0043] In the present invention, some color spaces (Y in this example) of some color formats (YCbCr in this example) will be described as references, and the same or similar application (settings dependent on a specific color space) can be made to other color spaces (Cb, Cr in this example) according to the color format. However, it may also be possible to have partial differences (settings independent of a specific color space) for each color space. That is, the settings dependent on each color space can mean having settings proportional to or dependent on the composition ratio of each component (e.g., determined according to 4:2:0, 4:2:2, or 4:4:4, etc.), and the settings independent on each color space can mean having settings only for the corresponding color space regardless of or independently of the composition ratio of each component. In the present invention, some components may have independent settings or dependent settings depending on the encoder / decoder.

[0044] The configuration information or syntax elements required during the video encoding process can be determined at the unit level such as video, sequence, picture, slice, tile, block, etc., and can be included in the bitstream as units such as VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), Slice Header, Tile Header, or Block Header and transmitted to the decoder. The decoder can parse at the same level of unit to restore the configuration information transmitted from the encoder and use it in the video decoding process. Each parameter set has a unique ID value, and a lower parameter set can have an ID value of a higher parameter set to reference. For example, a lower parameter set can reference information of a higher parameter set with a matching ID value among one or more higher parameter sets. Among the various examples of units mentioned above, if any one unit includes one or more other units, the corresponding unit can be called a higher unit, and the included units can be called lower units.

[0045] In the case of setting information occurring in the above unit, it may include content about independent settings for each unit, or content about settings that are dependent on previous, subsequent, or higher units. Here, the dependent setting may be understood as indicating the setting information of the unit with flag information indicating that it follows the settings of the previous, subsequent, or higher unit (for example, a 1-bit flag that follows when it is 1, and does not follow when it is 0). The setting information in the present invention will be explained focusing on examples of independent settings, but examples of addition or replacement with content about a dependent relationship with setting information of a previous or subsequent unit or higher unit of the current unit may also be included.

[0046] Video encoding / decoding may generally be performed according to the input size, but there may also be cases where encoding / decoding is performed through resizing. For example, in a hierarchical encoding method (scalability video coding) to support spatial, temporal, and image quality scalability, there may be cases where the overall resolution is adjusted, such as expansion and reduction of the video, and there may also be cases where partial expansion and reduction of the video is performed. Information about this can be switched by assigning selection information in units such as the VPS, SPS, PPS, and Slice Header described above. In this case, the hierarchy between each unit can be set as VPS > SPS > PPS > Slice header, etc.

[0047] The video encoding device and the video decoding device may be user terminals such as a personal computer (PC), a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a PlayStation Portable (PSP), a wireless communication terminal, a smart phone, or a TV, or server terminals such as an application server and a service server, and may include various devices including a communication device such as a communication modem for communicating with various devices or wired / wireless communication networks, a memory for storing various programs and data for inter- or intra-predicting to encode or decode an image, or a processor for executing a program and performing calculations and controls.

[0048] In addition, an image encoded into a bitstream by an image encoding device can be transmitted to an image decoding device through a wired or wireless communication network such as the Internet, a local area network, a wireless LAN, a WiBro network, or a mobile communication network in real time or non-real time, or through various communication interfaces such as a cable or a universal serial bus (USB), and can be decoded in the image decoding device, restored into an image, and played back. In addition, an image encoded into a bitstream by an image encoding device can be transmitted from the image encoding device to the image decoding device through a computer-readable recording medium.

[0049] The video encoding device and the video decoding device may be separate devices, but may be implemented as a single video encoding / decoding device. In this case, some components of the video encoding device may be implemented to have at least the same structure or perform at least the same function as some components of the video decoding device, as substantially the same technical elements.

[0050] Therefore, in the detailed description of the technical elements and their operating principles below, overlapping descriptions of corresponding technical elements will be omitted. In addition, since the image decoding device corresponds to a computing device that applies the image encoding method performed in the image encoding device to decoding, the description below will focus on the image encoding device.

[0051] A computing device may include a memory storing a program or software module implementing an image encoding method and / or an image decoding method, and a processor connected to the memory to execute the program. Here, the image encoding device may be referred to as an encoder, and the image decoding device may be referred to as a decoder, respectively.

[0052] Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0053]

[0054] FIG. 1 is a block diagram showing an image encoding device according to an embodiment of the present invention.

[0055] Referring to FIG. 1, an image encoding device (100) may include an image segmentation unit (101), an intra-screen prediction unit (102), an inter-screen prediction unit (103), a subtraction unit (104), a transformation unit (105), a quantization unit (106), an entropy encoding unit (107), an inverse quantization unit (108), an inverse transformation unit (109), an increase unit (110), a filter unit (111), and a memory (112).

[0056] Each component shown in Fig. 1 is independently depicted to indicate different characteristic functions in the video encoding device, and does not mean that each component is composed of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two components among each component may be combined to form a single component, or one component may be divided into multiple components to perform a function, and such integrated and separate embodiments of each component are also included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

[0057] Additionally, some components may not be essential components that perform essential functions of the present invention, but may be optional components merely used to enhance performance. The present invention may be implemented by including only components essential to implementing the essence of the present invention, excluding components used solely for performance enhancement. A structure that includes only essential components, excluding optional components used solely for performance enhancement, is also within the scope of the present invention.

[0058] The image segmentation unit (100) can segment an input image into at least one block. At this time, the input image can have various shapes and sizes such as pictures, slices, tiles, and segments. The block can mean a coding unit (CU, or coding block), a prediction unit (PU, or prediction block), or a transformation unit (TU, or transformation block). All or some of the coding units, prediction units, and transformation units can be the same. For example, the coding unit and the prediction unit can be the same, but the transformation unit can be a lower unit of the prediction unit. Alternatively, the coding unit, prediction unit, and transformation unit can all be different. The segmentation can be performed based on at least one of quadtree division, binary tree division, ternary tree division, and geometric division. Quadtree division is a method of dividing an upper block into four lower blocks whose width and height are half of those of the upper block. Binary tree partitioning divides a parent block into two sub-blocks, each of which is half the width or height of the parent block. Ternary tree partitioning divides a parent block into three sub-blocks, each of which is one-third the width or height of the parent block. Geometric partitioning divides a parent block into sub-blocks using partition lines of varying angles based on slope and / or distance from the center. Through the various partitioning methods described above, blocks can have shapes not only of square and non-square rectangles, but also of polygons (e.g., triangles).

[0059] The prediction unit (102, 103) may include an inter-screen prediction unit (103) that performs inter-prediction and an intra-screen prediction unit (102) that performs intra-prediction. It may be determined whether to use inter-prediction or intra-prediction for a prediction unit, and specific information (e.g., intra-prediction mode, motion vector, reference picture, etc.) according to each prediction method may be determined. At this time, the processing unit where prediction is performed may be different from the processing unit where the prediction method and specific contents are determined.

[0060] For example, the prediction method and prediction mode, etc. may be determined in prediction units, and prediction may be performed in transformation units. At this time, considering the block division structure, the transformation unit in which prediction is performed may be the sub-block with the largest division depth (=the smallest sub-block in the block division structure), and the prediction unit in which the prediction method and prediction mode, etc. are determined may be the upper block immediately above the sub-block with the largest division depth.

[0061] In the present disclosure, the current block may be a target block of each step of the encoding / decoding method or each unit of the encoding / decoding device. For example, if the prediction unit in which the prediction method and prediction mode are determined in the prediction unit is viewed as the current block, the transformation unit in which the prediction is performed may be expressed as a sub-block of the current block. For example, if the transformation unit in which the prediction is performed is viewed as the current block, the prediction unit in which the prediction method and prediction mode are determined in the prediction unit may be expressed as a block above the current block.

[0062] The residual value (residual block) between the generated prediction block and the original block can be input to the transformation unit (105). In addition, the prediction mode information, motion vector information, etc. used for prediction can be encoded together with the residual value by the entropy encoding unit (107) and transmitted to the decoder. When using a specific encoding mode, it is also possible to encode the original block as is and transmit it to the decoder without generating the prediction block through the prediction unit (102, 103).

[0063] The prediction unit (102) within the screen can generate a prediction block based on reference pixel information surrounding the current block, which is pixel information within the current picture. If the prediction mode of a block surrounding the current block on which intra prediction is to be performed is inter prediction, the reference pixel included in the surrounding block to which inter prediction is applied can be replaced with a reference pixel within another block surrounding which intra prediction is applied. That is, if the reference pixel is not available, the unavailable reference pixel information can be used by replacing it with at least one reference pixel among the available reference pixels.

[0064] In intra prediction, the prediction mode can have a directional prediction mode that uses reference pixel information according to the prediction direction, and a non-directional mode that does not use directional information when performing prediction. The mode for predicting luminance information and the mode for predicting chrominance information can be different, and the intra prediction mode information used to predict luminance information or the predicted luminance signal information can be utilized to predict chrominance information.

[0065] The prediction unit (102) within the screen may include an Adaptive Intra Smoothing (AIS) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter is a filter that performs filtering on the reference pixels of the current block and can adaptively determine whether to apply the filter depending on the prediction mode of the current prediction unit. If the prediction mode of the current block is a mode that does not perform AIS filtering, the AIS filter may not be applied.

[0066] The reference pixel interpolation unit of the prediction unit (102) within the screen can interpolate the reference pixel to generate a reference pixel at a fractional unit position when the intra prediction mode of the prediction unit is a prediction unit that performs intra prediction based on the pixel value interpolated for the reference pixel. When the prediction mode of the current prediction unit is a prediction mode that generates a prediction block without interpolating the reference pixel, the reference pixel may not be interpolated. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is the DC mode. The current block for which the prediction mode of the present disclosure is determined may be a sub-block having the largest division depth (=the smallest sub-block in the block division structure) or a block above the sub-block having the largest division depth. Here, the sub-block having the largest division depth may include a block ISP-divided from the upper block.

[0067] The inter-screen prediction unit (103) generates a prediction block using the previously restored reference image and motion information stored in the memory (112). The motion information may include, for example, a motion vector, a reference picture index, a list 1 prediction flag, a list 0 prediction flag, etc.

[0068] A residual block containing residual value information, which is the difference value between the prediction unit generated in the prediction unit (102, 103) and the original block of the prediction unit, can be generated. The generated residual block can be input to the transformation unit (130) and transformed.

[0069] The inter-screen prediction unit (103) can derive a prediction block based on information about at least one picture from among the previous or subsequent pictures of the current picture. Furthermore, the prediction block of the current block can also be derived based on information about a portion of the current picture in which encoding has been completed. The inter-screen prediction unit (103) according to one embodiment of the present invention can include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.

[0070] The reference picture interpolation unit can receive reference picture information from the memory (112) and generate pixel information less than or equal to an integer pixel from the reference picture. In the case of luminance pixels, a DCT-based 8-tap interpolation filter (DCT-based Filter) with different filter coefficients can be used to generate pixel information less than or equal to an integer pixel in units of 1 / 4 pixels. In the case of a chrominance signal, a DCT-based 4-tap interpolation filter (DCT-based Interpolation Filter) with different filter coefficients can be used to generate pixel information less than or equal to an integer pixel in units of 1 / 8 pixels.

[0071] The motion prediction unit can perform motion prediction based on a reference picture interpolated by the reference picture interpolation unit. Various methods can be used to derive a motion vector, such as FBMA (Full search-based Block Matching Algorithm), TSS (Three Step Search), and NTS (New Three-Step Search Algorithm). The motion vector can have a motion vector value in units of 1 / 2 or 1 / 4 pixels based on the interpolated pixel. The motion prediction unit can predict the prediction block of the current block by using different motion prediction methods. Various methods can be used as motion prediction methods, such as the Skip method, the Merge method, and the AMVP (Advanced Motion Vector Prediction) method.

[0072] The subtraction unit (104) subtracts the block to be currently encoded from the prediction block generated from the intra-screen prediction unit (102) or inter-screen prediction unit (103) to generate a residual block of the current block.

[0073] In the transformation unit (105), the residual block including the residual data can be transformed using a transformation method such as DCT, DST, KLT (Karhunen Loeve Transform), etc. At this time, the transformation method can be determined based on the intra prediction mode of the prediction unit used to generate the residual block. For example, depending on the intra prediction mode, DCT may be used in the horizontal direction and DST may be used in the vertical direction.

[0074] The quantization unit (106) can quantize the values ​​converted to the frequency domain by the transformation unit (105). The quantization coefficients can vary depending on the block or the importance of the image. The values ​​produced by the quantization unit (106) can be provided to the inverse quantization unit (108) and the entropy encoding unit (107).

[0075] The above-described transformation unit (105) and / or quantization unit (106) may be optionally included in the image encoding device (100). That is, the image encoding device (100) may encode the residual block by performing at least one of transformation or quantization on the residual data of the residual block, or by skipping both transformation and quantization. Even if neither transformation nor quantization is performed in the image encoding device (100), or neither transformation nor quantization is performed, a block that enters the input of the entropy encoding unit (107) is typically referred to as a transformation block. The entropy encoding unit (107) entropy-encodes the input data. Entropy encoding may use various encoding methods, such as, for example, Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC).

[0076] The entropy encoding unit (107) can encode various information such as coefficient information of a transform block, block type information, prediction mode information, division unit information, prediction unit information, transmission unit information, motion vector information, reference frame information, block interpolation information, and filtering information. The coefficients of a transform block can be encoded in units of sub-blocks within the transform block.

[0077] For encoding the coefficients of a transform block, various syntax elements can be encoded, such as Last_sig, a syntax element indicating the position of the first non-zero coefficient in reverse scan order, Coded_sub_blk_flag, a flag indicating whether there is at least one non-zero coefficient in the subblock, Sig_coeff_flag, a flag indicating whether the coefficient is non-zero, Abs_greater1_flag, a flag indicating whether the absolute value of the coefficient is greater than 1, Abs_greater2_flag, a flag indicating whether the absolute value of the coefficient is greater than 2, and Sign_flag, a flag indicating the sign of the coefficient. The residual value of the coefficient that is not encoded by the above syntax elements alone can be encoded through the syntax element remaining_coeff.

[0078] The inverse quantization unit (108) and the inverse transformation unit (109) inversely quantize the values ​​quantized in the quantization unit (106) and inversely transform the values ​​transformed in the transformation unit (105). The residual values ​​generated in the inverse quantization unit (108) and the inverse transformation unit (109) can be combined with the prediction units predicted through the motion estimation unit, motion compensation unit, and the intra-screen prediction unit (102) included in the prediction unit (102, 103) to generate a reconstructed block. The multiplier (110) multiplies the prediction blocks generated in the prediction units (102, 103) and the residual blocks generated through the inverse transformation unit (109) to generate a reconstructed block.

[0079] The filter unit (111) may include at least one of a deblocking filter, an offset correction unit, and an ALF (Adaptive Loop Filter).

[0080] A deblocking filter can remove block distortion caused by boundaries between blocks in a reconstructed picture. To determine whether to perform deblocking, a deblocking filter can be applied to the current block based on the pixels contained in several columns or rows within the block. When applying a deblocking filter to a block, a strong filter or a weak filter can be applied depending on the required deblocking filtering strength. Furthermore, when applying a deblocking filter, horizontal and vertical filtering can be processed in parallel when performing vertical and horizontal filtering.

[0081] The offset correction unit can correct the offset from the original image on a pixel-by-pixel basis for an image that has undergone deblocking. To perform offset correction for a specific picture, the pixels contained in the image can be divided into a certain number of regions, the regions to be offset can be determined, and the offset can be applied to those regions. Alternatively, the offset can be applied by considering the edge information of each pixel.

[0082] Adaptive Loop Filtering (ALF) can be performed based on the comparison of the filtered restored image with the original image. After dividing the pixels included in the image into predetermined groups, a filter to be applied to each group can be determined, and filtering can be performed differentially for each group. Information regarding whether to apply ALF can be transmitted by luminance signal for each coding unit (CU), and the shape and filter coefficients of the ALF filter to be applied can vary depending on each block. Furthermore, an ALF filter of the same form (fixed form) can be applied regardless of the characteristics of the target block.

[0083] The memory (112) can store a restoration block or picture produced through the filter unit (111), and the stored restoration block or picture can be provided to the prediction unit (102, 103) when performing inter-screen prediction.

[0084] Next, an image decoding device according to an embodiment of the present invention will be described with reference to the drawings. Fig. 2 is a block diagram illustrating an image decoding device (200) according to an embodiment of the present invention.

[0085] Referring to FIG. 2, the image decoding device (200) may include an entropy decoding unit (201), an inverse quantization unit (202), an inverse transformation unit (203), an amplification unit (204), a filter unit (205), a memory (206), and a prediction unit (207, 208).

[0086] When an image bitstream generated by an image encoding device (100) is input to an image decoding device (200), the input bitstream can be decoded according to a process opposite to the process performed in the image encoding device (100).

[0087] The entropy decoding unit (201) can perform entropy decoding in a procedure opposite to that of the entropy encoding unit (107) of the video encoding device (100). For example, various methods such as Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC) can be applied in response to the method performed in the video encoder. The entropy decoding unit (201) can decode the syntax elements described above, namely, Last_sig, Coded_sub_blk_flag, Sig_coeff_flag, Abs_greater1_flag, Abs_greater2_flag, Sign_flag, and remaining_coeff. Additionally, the entropy decoding unit (201) can decode information related to intra prediction and inter prediction performed in the image encoding device (100).

[0088] The inverse quantization unit (202) performs inverse quantization on a quantized transform block to generate a transform block. It operates substantially the same as the inverse quantization unit (108) of Fig. 1.

[0089] The inverse transform unit (203) performs an inverse transform on the transform block to generate a residual block. At this time, the transform method can be determined based on information regarding the prediction method (inter or intra prediction), the size and / or shape of the block, the intra prediction mode, etc. It operates substantially the same as the inverse transform unit (109) of FIG. 1.

[0090] The multiplication unit (204) multiplies the prediction block generated from the intra-screen prediction unit (207) or inter-screen prediction unit (208) and the residual block generated through the inverse transformation unit (203) to generate a restored block. It operates substantially the same as the multiplication unit (110) of Fig. 1.

[0091] The filter unit (205) reduces various types of noise occurring in restored blocks.

[0092] The filter unit (205) may include a deblocking filter, an offset correction unit, and an ALF.

[0093] Information on whether a deblocking filter has been applied to the current block or picture from the video encoding device (100) and, if a deblocking filter has been applied, information on whether a strong filter or a weak filter has been applied can be provided. The deblocking filter of the video decoding device (200) can receive information related to the deblocking filter provided from the video encoding device (100) and perform deblocking filtering on the current block in the video decoding device (200).

[0094] The offset correction unit can perform offset correction on the restored image based on the type of offset correction applied to the image during encoding and offset value information.

[0095] ALF can be applied to an encoding unit based on information on whether ALF is applied, ALF coefficient information, etc. provided from a video encoding device (100). This ALF information can be provided by being included in a specific parameter set. The filter unit (205) operates substantially the same as the filter unit (111) of FIG. 1.

[0096] The memory (206) stores the recovery block generated by the increase unit (204). It operates substantially the same as the memory (112) of FIG. 1.

[0097] The prediction unit (207, 208) can generate a prediction block based on the prediction block generation related information provided by the entropy decoding unit (201) and the previously decoded block or picture information provided by the memory (206).

[0098] The prediction unit (207, 208) may include an intra-screen prediction unit (207) and an inter-screen prediction unit (208). Although not separately illustrated, the prediction unit (207, 208) may further include a prediction unit determination unit. The prediction unit determination unit may receive various information such as prediction unit information input from the entropy decoding unit (201), prediction mode information of an intra-prediction method, and motion prediction-related information of an inter-prediction method, and may distinguish a prediction unit from a current encoding unit and determine whether the prediction unit performs inter-prediction or intra-prediction. The inter-screen prediction unit (208) may perform inter-screen prediction on the current prediction unit based on information included in at least one of a previous picture or a subsequent picture of the current picture including the current prediction unit, using information necessary for inter-prediction of the current prediction unit provided from the video encoding device (100). Alternatively, inter-screen prediction can be performed based on information from some previously reconstructed region within the current picture containing the current prediction unit.

[0099] In order to perform inter-screen prediction, it is possible to determine whether the motion prediction method of the prediction unit included in the encoding unit is Skip Mode, Merge Mode, or AMVP Mode based on the encoding unit.

[0100] The prediction unit (207) within the screen generates a prediction block using pixels located around the block to be encoded and previously restored.

[0101] The prediction unit (207) within the screen may include an Adaptive Intra Smoothing (AIS) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter is a filter that performs filtering on the reference pixels of the current block and can adaptively determine whether to apply the filter depending on the prediction mode of the current prediction unit. AIS filtering may be performed on the reference pixels of the current block using the prediction mode and AIS filter information of the prediction unit provided by the image encoding device (100). If the prediction mode of the current block is a mode that does not perform AIS filtering, the AIS filter may not be applied. Here, the current block may be the same as described in the encoding device.

[0102] The reference pixel interpolation unit of the prediction unit within the screen (207) can interpolate the reference pixel to generate a reference pixel at a fractional unit position when the prediction mode of the prediction unit is a prediction unit that performs intra prediction based on the pixel value interpolated from the reference pixel. The generated reference pixel at the fractional unit position can be used as a prediction pixel of a pixel within the current block. When the prediction mode of the current prediction unit is a prediction mode that generates a prediction block without interpolating the reference pixel, the reference pixel may not be interpolated. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is the DC mode. Here, the current block may be the same as described in the encoding device.

[0103] The on-screen prediction unit (207) operates substantially the same as the on-screen prediction unit (102) of FIG. 1.

[0104] The inter-screen prediction unit (208) generates an inter-screen prediction block using reference pictures and motion information stored in the memory (206). The inter-screen prediction unit (208) operates substantially the same as the inter-screen prediction unit (103) of FIG. 1.

[0105]

[0106] Figure 3 shows a flowchart of intra-subpartition (ISP)-based intra-screen prediction.

[0107] The Luma block of the present disclosure can be divided into K subpartitions in the horizontal or vertical direction. The sizes of the subpartitions can be the same, but in some cases, they can be different from each other. The encoding process of prediction and transformation can be performed individually for each subpartition, but the prediction mode and transformation kernel can be shared between the subpartitions. For example, each subpartition can perform prediction by sharing the prediction mode of the upper block, the Luma block. ISP-based intra-screen prediction can be applied to blocks with sizes ranging from 4x8 and 8x4 to 64x64.

[0108] The above K may be a natural number greater than or equal to 1. In addition, the above K may be determined according to the size of the block. For example, in the case of 4x8 and 8x4, the above K may be set to 2, and in other cases, the above K may be determined to 4. In this case, each subpartition may have at least 16 samples (or pixels).

[0109] If the size of the block to which the ISP-based on-screen prediction technology is applied is WxH, then in the case of horizontal partitioning, the size of the partitioned subpartition can be WxH / K, and in the case of vertical partitioning, the size of the partitioned subpartition can be W / KxH.

[0110] In the prediction and / or transformation process, the processing order of each subpartition may proceed from top to bottom in the case of horizontal partitioning, or from left to right in the case of vertical partitioning. According to the processing order, the subpartition restored first can be used for the prediction and transformation of the next subpartition.

[0111] In the conversion process, if ISP is used, the conversion kernel can be implicitly determined based on the block size. For example, if W is greater than or equal to 4 and H is greater than or equal to 16, DST7 is used, but otherwise, DCT2 can be used.

[0112] Low Frequency Non-Separable Transform (LFNST) can be transmitted globally across the entire ISP block. LFNST can only be used if all subpartitions satisfy the zero-out condition.

[0113] The flags and indices in FIG. 3 are described based on the operation in the decoding device for convenience of explanation, but the flags and indices can be encoded into the bitstream in the encoding device in the opposite order.

[0114] The intra_subpartitions_mode_flag signaled from the bitstream may indicate whether the ISP technology is applied to the current block. Accordingly, if the intra_subpartitions_mode_flag is true, the intra_subpartitions_split_flag may be additionally signaled. Here, the intra_subpartitions_split_flag may determine at least one of the splitting direction, the number of splits, or the splitting type to which the ISP technology is applied to the current block. For example, vertical splitting or horizontal splitting may be determined depending on the intra_subpartitions_split_flag value.

[0115] Even if the intra_subpartitions_mode_flag value is true, since the subpartitions (or sub-blocks) of the current block share the prediction mode of the current block, the prediction mode of the current block can be determined by applying the Most Probable Modes (MPM) technology of the current block regardless of the intra_subpartitions_mode_flag value. Specifically, the intra_luma_mpm_flag can be examined, and if the intra_luma_mpm_flag is true, the intra_luma_not_planar_flag can be examined. Here, the intra_luma_mpm_flag can indicate whether the intra-screen prediction mode of the current block is determined using any one of the MPM candidates. The intra_luma_not_planar_flag can indicate whether the intra-screen prediction mode of the current block is the Planar mode. That is, if intra_luma_not_planar_flag is true, the intra-screen prediction mode of the current block is determined as the Planar mode, and if intra_luma_not_planar_flag is false, the intra-screen prediction mode of the current block can be determined from MPM candidates excluding the Planar mode.

[0116] If intra_luma_not_planar_flag is false, the intra_luma_mpm_index can be additionally examined. The intra_luma_mpm_index indicates the prediction mode of one of the MPM candidates, and the prediction mode can be determined as the intra-screen prediction mode of the current block.

[0117] If intra_luma_mpm_flag is false, the intra-screen prediction mode of the current block can be determined as any one of the remaining prediction modes excluding the MPM candidates among the available intra-screen prediction modes. Any one of the above prediction modes can be determined as intra_luma_mpm_remainder. The intra_luma_mpm_remainder can indicate any one of the remaining prediction modes.

[0118] Figure 4 is a diagram showing adjacent blocks used to generate an MPM list.

[0119] The on-screen prediction mode of the current block may be determined based on an MPM list. Here, the MPM list may include at least one MPM candidate. The MPM candidates may be determined based on blocks surrounding the current block. The blocks surrounding the current block may be at least one of a block adjacent to the left of the current block or a block adjacent to the top of the current block. Here, the block adjacent to the left of the current block may be the lowermost block among the blocks adjacent to the left of the current block if there are multiple blocks adjacent to the left of the current block. Additionally, the block adjacent to the top of the current block may be the rightmost block among the blocks adjacent to the top of the current block if there are multiple blocks adjacent to the top of the current block.

[0120] For convenience of explanation, the prediction mode of the left-bottommost block adjacent to the current block in Fig. 4 is called L, the prediction mode of the top-rightmost block adjacent to the current block is called A, the vertical mode is called V, and the horizontal mode is called H. In addition, Max can mean max(A, L), and Min can mean min(A, L).

[0121] First, if the prediction mode (A, L) of the adjacent block is not an intra-screen prediction mode or if matrix-based intra-prediction (MIP) is applied, the Planar mode may be assigned. In this case, candidates derived from the adjacent block may be added to the candidate list in the Planar mode.

[0122] As an example, if neither A nor L are directional prediction modes, the MPM list may consist of {Planar, DC, V, H, V-4, V+4}.

[0123] In one embodiment, if A or L is a directional mode, the MPM list may consist of {Planar, Max, Max-1, Max+1, Max-2, Max+2}.

[0124] In one embodiment, if both A and L are directional modes and A and L are different modes, then: i) if Max-Min = 1, the MPM list is {Planar, L, A, min-1, Max+1, min-2}; ii) if Max-Min ≥ 62, the MPM list is {Planar, L, A, min+1, Max-1, min+2}; iii) if Max-Min = 2, the MPM list is {Planar, L, A, min+1, min-1, Max+1}; and iv) otherwise, the MPM list can be {Planar, L, A, min-1, min+1, Max-1}.

[0125] As an example, if A and L are both directional modes and A and L are the same mode, the MPM list may be {Planar, L, L-1, L+1, L-2, L+2}.

[0126] The ISP-based intra-screen prediction mode of the present disclosure may be based on the ISP-based intra-screen prediction mode described above.

[0127] The ISP-based intra-picture prediction mode of the present disclosure can use different intra-picture prediction modes between subpartitions (sub-blocks). Alternatively, as described above, the same intra-picture prediction mode can be used between subpartitions. In this regard, a flag indicating whether different intra-picture prediction modes are used between subpartitions can be transmitted through a bitstream. Alternatively, whether different intra-picture prediction modes are used between subpartitions can be determined based on attributes of a current block. Here, the attributes of the current block can mean the position, size, shape, area, width / height ratio, length of either width or height, segmentation depth, image type (I / P / B), color components (e.g., luminance, chrominance), value of the intra-picture prediction mode, whether the intra-picture prediction mode is a non-directional mode, angle of the intra-picture prediction mode, location of a reference pixel, etc.

[0128] For example, if the current block is in a vertical intra prediction mode, the prediction modes of the subpartitions are prediction modes spaced apart from the vertical intra prediction mode, and the two prediction modes can be symmetrical with respect to each other with respect to the vertical intra prediction mode.

[0129] For example, if the current block is in a horizontal intra prediction mode, the prediction modes of the subpartitions are prediction modes spaced apart from the horizontal intra prediction mode, and the two prediction modes can be symmetrical with respect to each other with respect to the horizontal intra prediction mode.

[0130] For example, if the size of the current block is equal to or smaller than 4x4, the prediction modes of the subpartitions follow the prediction mode of the current block, but if the size of the current block is larger than 4x4, the prediction modes of the subpartitions may be different from each other as the prediction modes of the current block are adjusted by adding an offset value to the prediction mode of the current block.

[0131] For example, if the direction of the directional prediction mode of the current block points to the decimal position of the surrounding reference sample of the current block, the prediction mode of the first subpartition may be the first directional prediction mode that points to the reference sample at the nearest integer position that has a larger angle around the directional prediction mode of the current block, and the prediction mode of the second subpartition may be the second directional prediction mode that points to the reference sample at the nearest integer position that has a smaller angle around the directional prediction mode of the current block.

[0132] A subpartition is obtained by dividing the current block using a method other than vertical or horizontal partitioning. The subpartitions may have different sizes. Alternatively, some subpartitions may be the same size while others may be different. Alternatively, all subpartitions may be the same size.

[0133] In prediction and transformation, the processing order of each subpartition of the ISP technology of the present disclosure may be at least one of a first order proceeding from top to bottom, a second order proceeding from bottom to top, a third order proceeding from left to right, or a fourth order proceeding from right to left. For example, the processing order of each subpartition may be performed in a diagonal direction by combining the first order and the third or fourth order. For example, the processing order of each subpartition may be performed in a diagonal direction by combining the second order and the third or fourth order.

[0134] If it is determined that different prediction modes are to be used between subpartitions, information for obtaining the prediction mode of each subpartition may be encoded / decoded for each subpartition. For example, the information may be an index indicating one of the prediction modes available to the current block. For example, the information may include at least one of the aforementioned MPM-related flags or indices. In other words, the method for obtaining the prediction mode within a screen of an existing CU (coding unit) can be applied to subpartitions (or sub-blocks) as is.

[0135] The prediction mode of the current subpartition can be determined using at least one of information from the current coding unit (CU), adjacent CUs, or a previous subpartition. The previous subpartition may refer to a subpartition that precedes the current subpartition in terms of processing order. In other words, it may be a subpartition that performed at least one of prediction and transformation before the current subpartition.

[0136] The prediction mode of the current subpartition may be determined based on the intra-screen prediction mode of the current block. For example, the prediction mode of the first subpartition may be a prediction mode obtained by adding a value K to the intra-screen prediction mode of the current block, and the prediction mode of the second subpartition may be a prediction mode obtained by subtracting a value K from the intra-screen prediction mode of the current block. Alternatively, the prediction mode of the subpartition may be determined as a mode within a range of ±K of the intra-screen prediction mode of the current block. The determination may be performed based on the aforementioned attributes of the current block or index information signaled from the bitstream.

[0137] The prediction mode of the current subpartition can be determined based on the prediction mode of the previous subpartition.

[0138] The MPM list of the current subpartition can be constructed based on the prediction mode of the previous subpartition. For example, the MPM list of the current subpartition can include a prediction mode obtained by adding or subtracting a value m from the prediction mode of the previous subpartition. Here, m can be a natural number greater than or equal to 1. Here, the value m can be explicitly signaled from the bitstream or implicitly determined from the attributes of the current block described above. As another example, the MPM list of the current subpartition can include the prediction mode of the previous subpartition. As another example, the MPM list of the current subpartition can be constructed excluding the previous subpartition.

[0139] As another example, the prediction modes available for the current block are divided into a plurality of groups, and a first group used to obtain the prediction mode of the first subpartition among the plurality of groups may be excluded from obtaining the prediction mode of the second subpartition. That is, the prediction mode of the second subpartition may be obtained from any one of the remaining groups excluding the first group among the plurality of groups. The plurality of groups may be natural numbers greater than 1, such as 2, 3, and 4, and the number of prediction modes in each group may be a natural number greater than 1, such as 1, 2, 3, 4, 5, and 6.

[0140] As another example, if the prediction mode of the first subpartition is determined, the prediction mode of the second subpartition can be determined as a prediction mode that adds or subtracts a value m from the prediction mode of the first subpartition. Here, m can be a natural number greater than or equal to 1. Here, the value m can be explicitly signaled from the bitstream or implicitly determined from the properties of the current block described above.

[0141] As another example, the prediction mode of the current subpartition can be determined from the prediction mode (K) of the current block, excluding the prediction mode of the previous subpartition, within a certain range (Km) to (K+m). Here, m can be a natural number greater than or equal to 1. Here, the value of m can be explicitly signaled from the bitstream or implicitly determined from the properties of the current block described above.

[0142] In performing prediction of the current subpartition based on the determined prediction mode, prediction of the current subpartition can be performed using the restored sample of the previous subpartition.

[0143]

[0144] Figure 5 is a diagram showing an example of an on-screen prediction mode.

[0145] The subpartitions of the present disclosure can be independently encoded in at least one of transformation, quantization, or entropy encoding processes after ISP-based intra-picture prediction.

[0146] Each subpartition can be encoded using a different transformation kernel depending on at least one of the size or prediction mode of the subpartition during the first transformation.

[0147] For example, if the size of the subpartition is less than 8x16 or 16x8 (=128), the 1D (1 Dimension) horizontal transformation kernel can be DST-7 and the 1D vertical transformation kernel can be DST-7.

[0148] For example, if the size of the subpartition is greater than or equal to 8x16 or 16x8 (=128), and if the number of prediction modes within the screen is within the range of 28 to 40, the 1D horizontal translation kernel can be DST-7 and the 1D vertical translation kernel can be DST-7.

[0149] For example, if the size of the subpartition is greater than or equal to 8x16 or 16x8 (=128), and if the prediction mode within the screen is a non-directional mode (e.g., Planar mode, DC mode), the 1D horizontal transform kernel can be DCT-2 and the 1D vertical transform kernel can be DCT-2.

[0150] For example, if the size of the subpartition is greater than or equal to 8x16 or 16x8 (=128), if the number of prediction modes within the screen is not in the range of 28 to 40 and the prediction mode within the screen is not a non-directional mode (= otherwise), the 1D horizontal transform kernel can be DCT-2 and the 1D vertical transform kernel can be DCT-2.

[0151] For example, if the prediction mode within the screen is Planar mode or DC mode, the transformation kernel derived for each mode may be KLT as a 1D horizontal transformation kernel and KLT as a 1D vertical transformation kernel.

[0152] For example, if the prediction mode within the screen is Planar mode or DC mode, the transformation kernel derived for each mode may be a 2D KLT with a 2D horizontal transformation kernel.

[0153] When different transformation kernels are used between subpartitions of the present disclosure, a combination of transformation kernels of each subpartition can be transmitted via a bitstream. That is, the transformation kernel of a subpartition can be explicitly determined based on information transmitted via the bitstream, or implicitly determined based on at least one of the attributes of the current block or the prediction mode within the screen as described above.

[0154] When the size of the subpartition of the present disclosure is 1xH or Wx1, a 1D transform (1D DST-2, 1D DST-7 or 1D-KLT) can be used.

[0155] Each subpartition of the present disclosure can be grouped into CU units and then encoded using different secondary transform kernels.

[0156] Figure 6 is an example diagram illustrating a CU division method in ISP-based on-screen prediction.

[0157] As illustrated in FIG. 6, the method for dividing a CU (Coding Unit) in the ISP-based intra-screen prediction of the present disclosure may include 2-partitioning, 3-partitioning, 4-partitioning, or M-partitioning. Here, M may be a natural number greater than or equal to 5. In addition, the method may include dividing by a vertical dividing line, a horizontal dividing line, a combination of vertical dividing lines and horizontal dividing lines, etc. In addition, the method may further include dividing by a diagonal dividing line. In addition, the dividing may be performed by combining diagonal dividing lines, vertical dividing lines, and horizontal dividing lines. For example, the current block may be divided into subpartitions using the above dividing methods.

[0158] The segmentation method can be transmitted in the bitstream via a flag or index, or determined based on the properties of the current block as described above.

[0159] For example, if the size of the CU being split is Wx1 or 1xH, it can only be split in the vertical or horizontal direction, and the maximum number of subpartitions being split can be 4.

[0160] Although Fig. 6 illustrates only the division of a square rectangle as an example, division similar to Fig. 6 may also be possible for non-square rectangle blocks. However, the number of block division methods for non-square rectangles may be smaller than that for square rectangles.

[0161] FIG. 7 is a diagram illustrating an example of using a previous subpartition in configuring an MPM list of the current subpartition.

[0162] Referring to Fig. 7, the current subpartition can construct the MPM list of the current subpartition using the prediction mode P of the previous subpartition.

[0163] Specifically, the MPM list of the current subpartition can be constructed using at least one of L, which is a prediction mode of the left neighboring block of the current subpartition, or A, which is a prediction mode of the upper neighboring block.

[0164] FIG. 8 is a diagram illustrating an embodiment of implicitly determining a combination of transformation kernels according to a splitting direction and a prediction mode of a subpartition.

[0165] In the case of Fig. 8A, the size of the subpartition is 32x8 (=256), which is larger than 128, and the numbers of the prediction modes within the screen of each subpartition are 35, 36, and 37, which are within the range of 28 to 40, so the combination of the transformation kernels of each subpartition can be DST-7 for the 1D horizontal transformation kernel and DST-7 for the 1D vertical transformation kernel.

[0166] In the case of Fig. 8B, since the size of the subpartition is 8x4 (=32), which is less than 128, the combination of transformation kernels of each subpartition can be a 1D horizontal transformation kernel of DST-7 and a 1D vertical transformation kernel of DST-7.

[0167] FIG. 9 is a diagram illustrating an example of performing a transformation by making a non-rectangular subpartition into a rectangular shape.

[0168] In the case of Figure 9A, the subpartition is an example in which a triangular shape is expanded to form a square shape. In this case, the coefficient value of the expanded empty area can be determined as 0.

[0169] In the case of Figure 9B, the CU is a subpartition located at the upper part of the CU, which is divided into four parts in an X shape. In this case, the transformation can be performed by expanding the part adjacent to the two short lines of the triangle to create a rectangular shape. At this time, the coefficient value of the expanded empty area can be determined as 0.

[0170] The expanded rectangular shape of the first subpartition located in the upper portion of the four-partitioned CU of FIG. 9B may be a rectangle whose width is greater than its height. Conversely, the expanded rectangular shape of the second subpartition located in the left portion of the four-partitioned CU of FIG. 9B may be a rectangle whose height is greater than its width.

[0171]

[0172] The present disclosure can propose a new interpolation filter and an interpolation filter application method for effectively performing ME (Motion Estimation) and MC (Motion Compensation) in inter prediction.

[0173] Inter-screen prediction of the VVC standard adopts AMVR (Adaptive Motion Vector Resolution), so in the case of normal AMVP mode, a 6-tab Gaussian interpolation filter (IF) (hpelIfIdx=1) or 8-tab DCT-IF (hpelIfIdx=0) is applied to 1 / 2 luma sample positions depending on the hpelIfIdx value, and an 8-tab DCT-IF can be applied to 1 / 4 luma sample positions.

[0174] The 6-tab IF applied when hpelIfIdx=1 at the 1 / 2 luma sample location is derived based on the Gaussian function below and has the characteristic of well restoring low frequency components, which can enable effective ME / MC when the reference block has low frequency characteristics.

[0175] Figure 10 shows an example of an equation for filter coefficients.

[0176] Referring to Figure 10, N represents the total number of taps of IF and n represents the sample position. w[n] can represent the filter coefficient.

[0177] The 8-tap DCT-IF applied to the 1 / 2 luma sample positions and 1 / 4 luma sample positions is derived based on the Discrete Cosine Transform (DCT)-II and Inverse DCT (IDCT)-II below, and can have the characteristics of well restoring low and high frequency components. Therefore, effective ME / MC can be possible when the reference block has high frequency characteristics. However, the current VVC applies the same DCT-IF regardless of the degree of high frequency of the reference block.

[0178] Figure 11 shows examples of formulas for DCT-II coefficients and IDCT-II coefficients.

[0179] Referring to Fig. 11, X(k) may represent DCT-II coefficients and x(n) may represent IDCT-II coefficients.

[0180] For example, in the case of 8-tab DCT-IF, N=8, and when creating filter coefficients at the position of 1 / 2 sample, as in Equation (3), by substituting X(k) of Equation (1) into Equation (2), and then substituting n=1 / 2 of Equation (3), the coefficients can be obtained by expressing the terms on the right as a linear combination of x(0), x(1),…

[0181] Fig. 12 is a diagram showing the relationship between the Magnitude and Frequency graphs representing the frequency characteristics at the 1 / 2 Luma sample position of a 6-tap Gaussian-based IF, an 8-tap DCT-IF, and the proposed 12-tap DCT-IF.

[0182] Referring to Fig. 12, it can be seen that the 6-tap Gaussian-based IF has the characteristic of restoring low frequency components well, the 8-tap and 12-tap DCT-IF have the characteristic of restoring low frequency and high frequency components well, and among them, the proposed 12-tap DCT-IF has the characteristic of restoring high frequency components better.

[0183] The proposed method can improve encoding efficiency by applying an interpolation filter that restores high frequency characteristics more strongly than the conventionally applied IF when the reference block has strong high frequency characteristics when DCT-IF is used at 1 / 2 luma sample positions and 1 / 4 luma sample positions in the existing VVC (i.e., when hpelIfIdx=0 at 1 / 2 luma sample positions).

[0184] Figure 13 shows an example of an auto-correlation formula.

[0185] Referring to Fig. 13, the frequency characteristics of a block can be obtained by applying the auto-correlation formula to the reference block. Here, N represents the height of the reference block, M represents the width, and x represents the samples of the reference block. may mean the average of reference block samples.

[0186] At this time, the frequency characteristics of the block can be found by using another formula that represents the similarity of data within the block rather than auto-correlation.

[0187] When the auto-correlation formula is applied to find out the frequency characteristics of a block, a larger absolute value of the correlation value means that the reference block has low frequency characteristics, and a smaller absolute value means that the reference block has high frequency characteristics.

[0188] At this time, the threshold may be different if a different equation representing similarity other than auto-correlation is used.

[0189] Figure 14 shows an example of 12-tap DCT-IF coefficients.

[0190] The encoder and decoder can utilize the above information to not transmit flag information, and if the absolute value of the correlation is greater than or equal to a set threshold value, apply VVC's 8-tab DCT-IF, and if it is less than the threshold value, apply the proposed 12-tab DCT-IF, which also restores high frequency components well. Referring to Fig. 14, the total weight of the 12-tab DCT-IF coefficients is set to 128, and if the weight value changes, the filter coefficients can also change.

[0191] Figure 15 shows the locations of fractional samples and integer samples.

[0192] The index in Fig. 14 may represent integer sample positions (A0 to A11) to which filtering is applied to generate fractional samples (a0, b0, c0 in Fig. 15). A 1 / 4-pixel filter may be a filter coefficient for generating a sample at position a0, a 1 / 2-pixel filter may be a filter coefficient for generating a sample at position b0, and a 3 / 4-pixel filter may be a filter coefficient for generating a sample at position c0.

[0193] Figure 16 shows an example of generating a sample at position b0 as an example of generating a fractional sample.

[0194] In order to reduce the complexity of the decoder, the proposed method can be applied by sending a flag to the decoder indicating whether to use a filter that restores high frequency characteristics well in the encoder, rather than finding out the frequency characteristics in both the encoder and the decoder.

[0195] For example, if long_tap_filter_flag=1, 12-tab DCT-IF can be applied, which restores high frequency components well.

[0196] The proposed method has been implemented in VTM 16.0, the VVC reference software, and can be implemented by selecting the interpolation filter through correlation calculations in both the encoder and decoder without flag transmission. It can also be implemented using the flag transmission method.

[0197] Figure 17 shows the experimental results of the performance of the proposed method.

[0198] The experiments were conducted in RA (Random Access) configurations and used classes A1, A2, B, C, and D. QPs of 22, 27, 32, and 37 were used for each sequence, and 32 frames were encoded for classes A1 and A2, and 64 frames for classes B, C, and D. The results of each experiment are expressed in BD-rate, and Fig. 17 shows that the proposed method improves encoding efficiency.

[0199] The proposed method can be applied to the normal AMVP mode for inter-screen prediction. The block to which the proposed method is applied can have a size of WxH. For example, W and H can be greater than or equal to 4. The proposed method has been shown to improve performance in VTM16.0, the VVC reference software.

[0200] Below, we describe a new application method that utilizes Mean SAD rather than the correlation of the proposed method.

[0201] Figure 18 shows an example of the Mean SAD formula.

[0202] To determine the frequency characteristics of a reference block, the Mean Sum of Absolute Difference (SAD) of the reference block can be utilized instead of correlation. The equation in Fig. 18 represents the Mean SAD equation, where N represents height, M represents width, and x(i,j) represents the sample value at position (i, j) of the reference block.

[0203] To find out the frequency characteristics of a reference block, it may be possible to use correlation or SAD information using only the information of the first row and first column within the reference block.

[0204] Figure 19 illustrates a formula for a method using correlation to find out the frequency characteristics of a reference block.

[0205] In the correlation-based method of Fig. 19 Select the interpolation filter to apply by comparing the value to the threshold, means the average of the first row could mean the average of the first column.

[0206] Figure 20 shows a formula for a method using Mean SAD to find out the frequency characteristics of a reference block.

[0207] In the Mean SAD-based method of Fig. 20, the interpolation filter to be applied is selected by comparing MeanSAD with the threshold, and MeanSAD 1st row of x (0, i) is the samples of the first row, MeanSAD 1st row of x (i, 0) can represent the samples of the first column.

[0208] A larger Mean SAD value indicates that the reference block has high-frequency characteristics. Therefore, if the calculated Mean SAD value of the reference block is greater than a predefined threshold (including high-frequency components), the proposed 12-tap DCT-IF or an IF with a longer tap can be used. In other cases, the existing interpolation filter of the VVC standard can be used. This can also be applied to the Mean SAD.

[0209] If the reference threshold is determined in a 4x4 block (a block having 16 samples), the threshold in blocks of other sizes can be determined in proportion to the number of samples based on the 4x4 block.

[0210] For example, if it is determined that 12-tap DCT-IF is applied when the Mean SAD of the surrounding blocks is > the first threshold for a 4x4 block having 16 samples, the second threshold of a 4x8 or 8x4 block having 32 samples may be 2 * the first threshold. Accordingly, 12-tap DCT-IF may be applied when the Mean SAD of the surrounding blocks of a 4x8 or 8x4 block having 32 samples is > the second threshold (= 2 * the first threshold).

[0211] In blocks of other sizes than the examples above, the threshold can be determined in proportion to the number of samples in the block.

[0212] Additionally, in the example above, the block that serves as the threshold standard was considered to be a 4x4 block, but it is not limited to this, and a block of a size other than a 4x4 block can serve as the threshold standard.

[0213] Alternatively, thresholds can be defined and applied differently for each block size. The examples below illustrate different definitions for blocks of 4x4, 4x8, and 16x16 sizes.

[0214] 1) In a 4x4 block, when the ref block's Mean SAD > threshold_1, 12-tap DCT-IF can be applied.

[0215] 2) In a 4x8 block, 12-tap DCT-IF can be applied when ref block's Mean SAD > threshold_2.

[0216] 3) In a 16x16 block, 12-tap DCT-IF can be applied when ref block's Mean SAD > threshold_n.

[0217] In the examples above, only blocks of sizes 4x4, 4x8, and 16x16 are described, but thresholds can be defined differently for blocks of other sizes.

[0218] As an example, if BLOCK_SIZE > Threshold, a Gaussian-based 8-tap or 6-tap interpolation filter, which is a low-pass filter that emphasizes low frequencies, can be applied. Conversely, if BLOCK_SIZE <= Threshold, a 12-tap DCT-IF with 7-bit or 8-bit precision that emphasizes low and high frequency components can be applied.

[0219] Video resolution may be additionally considered when applying the filter of the present disclosure.

[0220] For example, if the video is less than 4K (video resolution < 3840x2160) and correlation is used, 1) if the absolute value of the correlation is greater than a set threshold value, the 8-tap DCT-IF or 6-tap SIF of the VVC standard can be applied, and 2) if the absolute value of the correlation is less than the threshold value, the proposed 12-tap DCT-IF, which also restores high frequency components well, can be applied (performance can be improved by applying long-tap DCT-IF, which is longer than 8-tap DCT-IF).

[0221] For example, if the video is less than 4K (video resolution < 3840x2160) and Mean SAD is used, 1) if the absolute value of Mean SAD is less than a set threshold value, VVC's 8-tap DCT-IF or 6-tap SIF can be applied, and 2) if the absolute value of Mean SAD is greater than the threshold value, the proposed 12-tab DCT-IF, which also restores high frequency components well, can be applied (performance can be improved by applying a long-tab DCT-IF that is longer than 8-tab DCT-IF).

[0222] For example, in videos below 4K (video resolution < 3840x2160), it may be possible to select an interpolation filter using gradient in addition to correlation and Mean SAD.

[0223] For example, for videos of 4K or higher (video resolution ₃ 3840x2160), there is no need to use long tap DCT-IF because the correlation between samples is large and the Mean SAD value is small, so only 8-tap DCT-IF or 6-tap SIF can be used.

[0224] For example, for videos of 4K or higher (video resolution ₃ 3840x2160), only Smoothing IF can be used because the correlation between samples is high and the Mean SAD value is small, which means that the video itself has strong low frequency characteristics.

[0225]

[0226] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps included, or some steps may be excluded and additional steps included.

[0227] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combinations of two or more.

[0228] Additionally, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the embodiments may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.

[0229] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and executable on the device or computer.

[0230] The present disclosure has industrial applicability in the fields of video encoding / decoding methods, devices, and recording media.

Claims

1. In inter-screen prediction of the current video, a step of determining a filter for interpolating a reference picture of the current block; A step of obtaining an interpolated reference picture by interpolating the reference picture using the determined filter; A step of performing motion prediction based on the above interpolated reference picture, A method for decoding an image, characterized in that the filter for interpolating the reference picture is determined by comparing the size of the current block, the absolute correlation value of the reference block of the current block, or the Mean SAD (Sum of Absolute Difference) value of the reference block with a threshold value.

2. In paragraph 1, A video decoding method, characterized in that, in a case where a filter for interpolating the reference picture is determined by comparing the size of the current block with a threshold value, the filter for interpolating the reference picture in response to the size of the current block being greater than the threshold value is determined as an 8-tap or 6-tap interpolation filter, and the filter for interpolating the reference picture in response to the size of the current block being less than or equal to the threshold value is determined as a 12-tap interpolation filter.

3. In paragraph 1, An image decoding method, characterized in that the threshold value is different depending on the size of the current block.

4. In paragraph 1, An image decoding method, characterized in that the MeanSAD value of the above reference block is calculated based on the width and height of the above reference block and the sample values ​​of the above reference block.

5. In paragraph 4, A method for decoding an image, characterized in that, in response to the MeanSAD value of the above reference block being greater than a threshold value, a filter for interpolating the above reference picture is determined to be a 12-tap interpolation filter or an interpolation filter longer than 12 taps.

6. In paragraph 1, If the resolution of the current video is less than 4K, the filter for interpolating the reference picture is determined by comparing the absolute value of the correlation of the reference block with a threshold value, A video decoding method, characterized in that when the resolution of the current video is 4K or higher, the filter for interpolating the reference picture is determined as an 8-tap interpolation filter or a 6-tap interpolation filter.

7. In paragraph 6, If the absolute value of the correlation of the above reference block is greater than or equal to the threshold, the filter interpolating the above reference picture is determined as an 8-tap interpolation filter or a 6-tap interpolation filter, An image decoding method, characterized in that when the absolute value of the correlation of the above reference block is less than a threshold value, the filter interpolating the above reference picture is determined as a 12-tap interpolation filter.

8. In inter-screen prediction of the current video, a step of determining a filter for interpolating a reference picture of the current block; A step of obtaining an interpolated reference picture by interpolating the reference picture using the determined filter; A step of performing motion prediction based on the above interpolated reference picture, A video encoding method, characterized in that the filter for interpolating the reference picture is determined by comparing the size of the current block, the absolute correlation value of the reference block of the current block, or the Mean SAD (Sum of Absolute Difference) value of the reference block with a threshold value.

9. In paragraph 8, A video encoding method, characterized in that, in a case where a filter for interpolating the reference picture is determined by comparing the size of the current block with a threshold value, the filter for interpolating the reference picture in response to the size of the current block being greater than the threshold value is determined as an 8-tap or 6-tap interpolation filter, and the filter for interpolating the reference picture in response to the size of the current block being less than or equal to the threshold value is determined as a 12-tap interpolation filter.

10. In paragraph 8, A method for encoding an image, wherein the threshold value is different depending on the size of the current block.

11. In paragraph 8, A video encoding method, characterized in that the MeanSAD value of the above reference block is calculated based on the width and height of the above reference block and the sample values ​​of the above reference block.

12. In paragraph 11, A video encoding method, characterized in that, in response to the MeanSAD value of the above reference block being greater than a threshold value, a filter for interpolating the above reference picture is determined to be a 12-tap interpolation filter or an interpolation filter longer than 12 taps.

13. In paragraph 8, If the resolution of the current video is less than 4K, the filter for interpolating the reference picture is determined by comparing the absolute value of the correlation of the reference block with a threshold value, A video encoding method, characterized in that when the resolution of the current video is 4K or higher, the filter for interpolating the reference picture is determined as an 8-tap interpolation filter or a 6-tap interpolation filter.

14. In paragraph 13, If the absolute value of the correlation of the above reference block is greater than or equal to the threshold, the filter interpolating the above reference picture is determined as an 8-tap interpolation filter or a 6-tap interpolation filter, A video encoding method, characterized in that when the absolute value of the correlation of the above reference block is less than a threshold value, the filter interpolating the above reference picture is determined as a 12-tap interpolation filter.

15. A computer-readable recording medium storing a bitstream generated by an encoding method that performs inter-screen prediction of a current video, The above encoding method comprises the steps of: determining a filter for interpolating a reference picture of a current block; A step of obtaining an interpolated reference picture by interpolating the reference picture using the determined filter; A step of performing motion prediction based on the above interpolated reference picture, A computer-readable recording medium, characterized in that the filter for interpolating the reference picture is determined by comparing the size of the current block, the absolute correlation value of the reference block of the current block, or the Mean SAD (Sum of Absolute Difference) value of the reference block with a threshold value.

Images