Method and apparatus for coding video by using selective interpolation

Abstract

The present embodiment discloses a video coding method and an apparatus selectively using an interpolation scheme. In the present embodiment, an image decoding apparatus sets areference region of the current block and obtains encoding information of the reference region. The image decoding apparatus selects an interpolation scheme from an interpolation scheme pool on the basis of the reference regionand the encoding information. The image decoding apparatus predicts the current block by applying the selected interpolation scheme to the reference region.

Description

Method and device for video coding using optional interpolation

[0001] The present disclosure relates to a video coding method and device that optionally utilize an interpolation method.

[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.

[0003] Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.

[0004] Therefore, when storing or transmitting video data, the encoder compresses the video data and stores or transmits it, and the decoder receives the compressed video data, decompresses it, and plays it back. These video compression technologies include H.264 / AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves encoding efficiency by about 30% compared to HEVC.

[0005] However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.

[0006] Interpolation utilizes pixel values ​​at adjacent integer positions to calculate values ​​between integer positions that do not exist within a block. Existing HEVC or VVC technologies interpolate reference regions based on a predefined interpolation scheme, regardless of the type of video content, to generate predictors. However, because video properties can vary significantly depending on the content, predictors generated using this predefined interpolation scheme may not be optimal. Therefore, a method for efficiently interpolating reference regions is needed when generating a predictor for the current block.

[0007] The present disclosure aims to provide a video coding method and device that selectively uses various interpolation schemes based on content analysis of a reference region.

[0008] An embodiment according to the present disclosure aims to provide a video coding method and device for inferring an interpolation method by analyzing a reference region when generating a predictor.

[0009] One embodiment of the present disclosure aims to provide a video coding method and device for signaling and parsing an interpolation method related to a reference region when generating a predictor.

[0010] According to an embodiment of the present disclosure, a method for restoring a current block, performed by an image decoding device, is provided, comprising: setting a reference region of the current block and obtaining encoding information of the reference region; selecting an interpolation scheme from an interpolation scheme pool based on the reference region and the encoding information; and applying the selected interpolation scheme to the reference region to predict the current block.

[0011] According to another embodiment of the present disclosure, a method for encoding a current block, performed by a video encoding device, is provided, comprising: obtaining a reference region of the current block and encoding information of the reference region; selecting an interpolation scheme from an interpolation scheme pool based on the reference region and the encoding information; and applying the selected interpolation scheme to the reference region to predict the current block.

[0012] According to another embodiment of the present disclosure, a method for providing video data to a video decoding device is provided, comprising: encoding the video data into a bitstream; and transmitting the bitstream to the video decoding device, wherein the encoding the video data comprises: obtaining a reference region of a current block and encoding information of the reference region; selecting an interpolation scheme from an interpolation scheme pool based on the reference region and the encoding information; and applying the selected interpolation scheme to the reference region to predict the current block.

[0013] As described above, according to the present embodiment, there is provided a video coding method and device that selectively uses various interpolation methods based on content analysis of a reference area, thereby making it possible to improve video encoding efficiency and enhance video quality.

[0014] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.

[0015] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.

[0016] FIGS. 3A and 3B are diagrams illustrating multiple intra prediction modes, including wide-angle intra prediction modes.

[0017] Figure 4 is an example diagram of the surrounding blocks of the current block.

[0018] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.

[0019] Figure 6 is an example diagram showing a case where interpolation is applied in intra prediction.

[0020] Figure 7 is an example diagram showing a case where interpolation is applied in inter prediction or IBC (Intra Block Copy).

[0021] Figure 8 is an example diagram showing a reference area in inter prediction.

[0022] Figure 9 is an example diagram showing a reference area in IBC.

[0023] Figure 10 is an example diagram showing a reference area in intraTMP (intra Template Matching Prediction).

[0024] Figure 11 is an example diagram showing the location of subpixels.

[0025] Figure 12 is an example diagram showing legacy screen content.

[0026] Figure 13 is an example diagram showing interpolation of a reference region using a sinc function-based FIR (Finite Impulse Response) filter.

[0027] FIG. 14 is an exemplary diagram showing selection of an interpolation method according to one embodiment of the present disclosure.

[0028] FIG. 15 is an exemplary diagram showing a Sobel filter according to one embodiment of the present disclosure.

[0029] FIG. 16 is an exemplary diagram showing a classification network according to one embodiment of the present disclosure.

[0030] FIG. 17 is an exemplary diagram illustrating a reference area including areas encoded in different modes according to one embodiment of the present disclosure.

[0031] FIG. 18 is a flowchart illustrating a method of encoding a current block performed by an image encoding device according to one embodiment of the present disclosure.

[0032] FIG. 19 is a flowchart illustrating a method for restoring a current block performed by an image decoding device according to one embodiment of the present disclosure.

[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.

[0034] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to the illustration in FIG. 1.

[0035] The video encoding device may be configured to include a picture segmentation unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0036] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0037] A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). And, information commonly referenced by one or more SPS is encoded in a Video Parameter Set (VPS). In addition, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.

[0038] The picture segmentation unit (110) determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as the syntax of SPS or PPS and transmitted to the image decoding device.

[0039] The picture segmentation unit (110) divides each picture constituting an image into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a CU, which is a basic unit of encoding.

[0040] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).

[0041] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.

[0042] As illustrated in FIG. 2, a CTU may first be split into a QT structure. The quadtree splitting may be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further split into one or more of the BT structure or the TT structure. There may be multiple splitting directions in the BT structure and / or the TT structure. For example, there may be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As illustrated in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0043] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the CU split flag (split_cu_flag) value indicates that the node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is a basic unit of encoding. If the CU split flag (split_cu_flag) value indicates that the node is split, the video encoding device starts encoding from the first flag in the above-described manner.

[0044] As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.

[0045] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.

[0046] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0047] In general, each current block within a picture can be predictively coded. Prediction of the current block can typically be performed using either intra-prediction (using data from the picture containing the current block) or inter-prediction (using data from a picture coded before the picture containing the current block). Inter-prediction encompasses both unidirectional and bidirectional prediction.

[0048] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3a, the multiple intra prediction modes may include two non-directional modes including the Planar mode and the DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently depending on each prediction mode.

[0049] For efficient directional prediction for a rectangular current block, directional modes (intra prediction modes 67 to 80 and -1 to -14) indicated by dotted arrows in Fig. 3b may be additionally used. These may be referred to as "wide-angle intra-prediction modes." In Fig. 3b, the arrows point to corresponding reference samples used for prediction, and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide-angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra-prediction modes, some wide-angle intra-prediction modes available for the current block may be determined based on the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (intra prediction modes 67 to 80) having an angle less than 45 degrees are available when the current block is a rectangular shape whose height is smaller than its width, and wide-angle intra prediction modes (intra prediction modes -1 to -14) having an angle greater than -135 degrees are available when the current block is a rectangular shape whose width is larger than its height.

[0050] The intra prediction unit (122) can determine an intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate bit-rate distortion values ​​using rate-distortion analysis for multiple tested intra prediction modes and select an intra prediction mode with the best bit-rate distortion characteristics among the tested modes.

[0051] The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0052] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma component and the chroma component. The motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0053] The inter prediction unit (124) may perform interpolation on a reference picture or a reference block to improve prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision or resolution of the motion vector can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.

[0054] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the entropy encoding unit (155). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.

[0055] Various methods can be used to minimize the number of bits required to encode motion information.

[0056] For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the image decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.

[0057] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

[0058] As the surrounding blocks for deriving merge candidates, all or part of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the block at the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.

[0059] The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, the merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the video decoding device.

[0060] Merge Skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high encoding efficiency can be achieved for low-motion images, still images, and screen content images.

[0061] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0062] Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.

[0063] In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block in the current picture as shown in FIG. 4 may be used. In addition, a block located in a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block co-located with the current block in the reference picture or blocks adjacent to the block in the co-located block may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.

[0064] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.

[0065] The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information to identify the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.

[0066] Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.

[0067] The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.

[0068] The transformation unit (140) transforms residual signals within a residual block having pixel values ​​in a spatial domain into transform coefficients in a frequency domain. The transformation unit (140) may transform the residual signals within the residual block using the entire size of the residual block as a transformation unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transformation units to perform the transformation. Alternatively, the residual signals may be transformed using only the transformation domain sub-block as a transformation unit by dividing the sub-blocks into two sub-blocks, that is, a transformation domain and a non-transform domain. Here, the transformation domain sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only a sub-block has been converted, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. In addition, the size of the conversion area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0069] Meanwhile, the transformation unit (140) can individually perform transformations on the residual block in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation function pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation function pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0070] The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.

[0071] The rearrangement unit (150) can perform rearrangement of coefficient values ​​for quantized residual values.

[0072] The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region by using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, the scanning method to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan.

[0073] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0074] In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about an intra prediction mode) or inter prediction information (information about an encoding mode of motion information (merge mode or AMVP mode), a merge index in the case of a merge mode, and a reference picture index and a differential motion vector in the case of an AMVP mode) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about a quantization parameter and information about a quantization matrix.

[0075] The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0076] An adder (170) adds the restored residual block and the predicted block generated by the prediction unit (120) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting the next block.

[0077] The loop filter unit (180) performs filtering on restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filter unit (180) may include all or part of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186) as an in-loop filter.

[0078] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter (184) and the ALF (186) perform additional filtering on the deblocking-filtered image. The SAO filter (184) and the ALF (186) are filters used to compensate for the differences between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in units of CTUs. In contrast, the ALF (186) performs block-based filtering, and compensates for distortion by applying different filters by distinguishing the edges and degrees of variation of the corresponding block. Information on filter coefficients to be used in the ALF can be encoded and signaled to an image decoding device.

[0079] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0080] The video encoding device can store the bitstream of encoded video data on a non-transitory storage medium or transmit it to the video decoding device using a communication network.

[0081] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.

[0082] The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

[0083] Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0084] The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information, information on residual signals, etc. required to restore the current block.

[0085] The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.

[0086] For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (mtt_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Accordingly, each node below the leaf nodes of QT are split recursively into the BT or TT structure.

[0087] As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.

[0088] As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.

[0089] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded by using the division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.

[0090] Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about residual signals.

[0091] The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.

[0092] The inverse quantization unit (520) inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.

[0093] The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.

[0094] In addition, when the inverse transform unit (530) inversely transforms only a portion of a transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only a sub-block of the transform block has been transformed, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block, and inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to restore residual signals, and fills “0” values ​​with residual signals for areas that have not been inversely transformed, thereby generating a final residual block for the current block.

[0095] In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions using MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.

[0096] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.

[0097] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0098] The inter prediction unit (544) uses the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.

[0099] An adder (550) adds the residual block output from the inverse transform unit (530) and the predicted block output from the inter prediction unit (544) or the intra prediction unit (542) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting a block to be decoded later.

[0100] The loop filter unit (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as in-loop filters. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) and the ALF (566) perform additional filtering on restored blocks after deblocking filtering to compensate for differences between restored pixels and original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about filter coefficients decoded from the non-stream.

[0101] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0102] The present embodiment relates to the encoding and decoding of images (videos) as described above. More specifically, the present invention provides a video coding method and device that selectively utilizes various interpolation methods based on content analysis of a reference region.

[0103] The following embodiments may be performed by a prediction unit (120) within a video encoding apparatus. Additionally, the following embodiments may be performed by a prediction unit (540) within a video decoding apparatus.

[0104] The video encoding device can generate signaling information related to the present embodiment in terms of rate distortion optimization in encoding the current block. The video encoding device can encode the signaling information using the entropy encoding unit (155) and then transmit it to the video decoding device. The video decoding device can decode the signaling information related to the decoding of the current block from the bitstream using the entropy decoding unit (510).

[0105] In the following description, the term "target block" may be used interchangeably with the current block or coding unit (CU). Alternatively, the term "target block" may also refer to a portion of a coding unit.

[0106] Also, a value of a flag being true indicates that the flag is set to 1. Also, a value of a flag being false indicates that the flag is set to 0.

[0107] The decoder side includes all or part of the inverse quantization unit (160), the inverse transform unit (165), the prediction unit (120), the adder (170), the loop filter unit (180), and the memory (190) in the video encoding device illustrated in FIG. 1. Alternatively, the decoder side includes all or part of the inverse quantization unit (520), the inverse transform unit (530), the prediction unit (540), the adder (550), the loop filter unit (560), and the memory (570) in the video decoding device illustrated in FIG. 5. With respect to a series of decoding processes, the decoder side of the video encoding device and the decoder side of the video decoding device perform the same operation. The video encoding device determines information related to the operation of the decoder side and signals the determined information to the video decoding device. The video decoding device can decode the signaled information and operate the decoder side based on the decoded information.

[0108] I. Interpolation technology

[0109] Interpolation calculates values ​​between integer positions that do not exist in a block by using pixel values ​​at adjacent integer positions.

[0110] For example, as shown in Fig. 6, when generating a predictor according to a directional mode in intra prediction, if the prediction direction does not indicate the integer position of the reference pixel, interpolation between the reference pixels is required. As shown in Fig. 7, in inter prediction or IBC (Intra Block Copy), if there is a fractional part in the horizontal and / or vertical component of the motion vector (MV) or block vector (BV), interpolation between adjacent integer positions within the reference area is required.

[0111] A video decoding device can refer to various regions for predicting the current block. In intra prediction, if the intra mode is a directional mode and is less than 34, the video decoding device can generate a predictor by configuring reference pixels based on the left region of the current block. If the intra mode is 34 or more, the video decoding device can generate a predictor by configuring reference pixels based on the upper region of the current block. In the case of inter prediction, the video decoding device can generate a predictor of the current block by referring to the region indicated by the MV in the reference picture, as shown in FIG. 8. In the case of IBC mode, the video decoding device can generate a predictor by referring to the region indicated by the BV, as shown in FIG. 9. In the case of intraTMP (intra Template Matching Prediction), the video decoding device can generate a predictor of the current block by referring to the region with the smallest difference between templates in the restored region, as shown in FIG. 10.

[0112] When calculating pixel sample values ​​corresponding to sub-pel locations, interpolation filters with various numbers of taps or various sub-pel precisions can be used. In VVC, in order to obtain sample values ​​of reference pixel locations used for intra prediction, the interpolation filters applied, such as linear filters, Gaussian filters, and cubic filters, may vary depending on conditions such as whether the current block is a luma component or a chroma component, whether the reference pixel is filtered, and so on. In inter prediction, the interpolation filters applied may vary depending on conditions such as whether the current picture is a luma component or a chroma component, whether an affine motion model is used, and the ratio of Reference Picture Resampling (RPR) between the current picture and the reference picture.

[0113] Below, we illustrate interpolation using an 8-tap filter with 1 / 16-pixel precision for inter-prediction of the luma component when the affine motion model is not used in VVC and the RPR ratio is 1.0. The filter coefficients used for interpolation are illustrated in Table 1.

[0114]

[0115] p is A in Fig. 11 0,0 Wow A 1,0 The subpixel positions between positions are represented in 1 / 16-pixel units. f[p][0], ... f[p][7] are used to calculate the value at position p. 0,0 , A 1,0 These are filter coefficients that are multiplied by the pixel values ​​of adjacent integer positions. For example, the image decoding device, in Fig. 11, A 0,0 Wow A 1,0 p, the value of the 5 / 16 pixel position sample between positions 5 / 16 can be calculated as in mathematical formula 1.

[0116]

[0117] Video content that can be compressed using a video encoding device is very diverse, such as camera-captured content (CC), legacy screen content (LSC), photo-realistic screen content (PRSC), lenslet video, MPEG Immersive Video (MIV) atlas, depth map, network feature map, etc. Although the types of videos are very diverse depending on the video generation method, general video encoding technologies such as HEVC or VVC are used for encoding. Existing HEVC or VVC technologies generate a predictor by interpolating a reference region according to a predefined interpolation method regardless of the type of video content. Since the properties of the video may be very different depending on the video content, the predictor generated according to the predefined interpolation method may not be optimal.

[0118] As an example, a case where an LSC is included in a reference block during inter prediction is described. As shown in Fig. 12, unlike CC, a computer-generated LSC has parts where the distribution of pixel values ​​is flat or changes rapidly. Existing HEVC or VVC technology interpolates a reference region using a sinc function or a DCT (Discrete Cosine Transform)-based FIR (Finite Impulse Response) filter during inter prediction. When interpolating a reference region using a sinc function-based FIR filter, as shown in Fig. 13, the LSC image may be distorted due to overshooting and undershooting of the FIR filter itself. Due to the distortion of the LSC image, an incorrect predictor may be generated during inter prediction, which may cause a large error in the restored image. Fig. 13 shows the distribution of pixel values ​​corresponding to the part (1210) indicated in Fig. 12. In Fig. 13, circles represent integer pixel locations of Fig. 12, and solid lines conceptually represent interpolated values ​​when applying a sinc function-based FIR filter.

[0119] As another example, we describe a case where a reference block exists at the boundary between patches during inter prediction with a MIV atlas. MIV divides images from each viewpoint into rectangular patches to compress images from multiple views. Since there may be redundancy between patches, an atlas is created by stitching together non-overlapping patches, and the atlas is encoded using existing technologies such as HEVC or VVC. Since an atlas is created by stitching together patches cropped to different sizes at different locations from different views, there is discontinuity in pixel values ​​at the boundaries between patches. When predicting in inter prediction or IBC mode, if a reference block spans the boundary between patches of the atlas and an FIR filter is applied to the boundary between patches, the pixel values ​​used for interpolation may be mixed around the boundary between patches, which may result in an incorrect predictor.

[0120] The following embodiments are described with a focus on a video decoding device, but can be implemented in the same or similar manner in a video encoding device. Alternatively, the following embodiments are described with a focus on the decoder side of a video decoding device, but can also be implemented in the same or similar manner in the decoder side of a video encoding device.

[0121] II. Embodiments according to the present disclosure

[0122] The types of video content that must be compressed by a video encoding device vary greatly depending on the video generation method. However, existing technologies, regardless of the type of video content, can only interpolate reference regions based on predefined, limited interpolation methods to generate predictors. Therefore, encoding efficiency and reconstructed image generation may not be optimal. The problems of existing technologies can be solved by providing various methods capable of interpolating reference regions, as in the present disclosure, and by having video encoding devices and video decoding devices selectively use interpolation methods.

[0123] Depending on the type of image content in the reference area, various interpolation methods can be applied as follows.

[0124] As an example, a case where the reference area is CC is described. The interpolation filter coefficients in Table 1 can be utilized for interpolation during inter prediction. Between two adjacent integer pixel positions, such as A0 and A1, the sample value at the 5 / 16-pixel position can be calculated as in Equation 2. Hereinafter, the interpolation method for the case where the reference area is CC is referred to as the CC interpolation method.

[0125]

[0126] As another example, we describe a case where the reference region has a strong edge because it has the property of LSC. When predicting in inter prediction or IBC mode, a predictor can be generated using the pixel value of the integer pixel position closest to the position indicated by MV or BV without performing interpolation. When predicting in intra prediction, a predictor can be generated using the direction indicating the reference pixels of the integer pixel position closest to the position indicated by the prediction direction without interpolating the reference pixels. Hereinafter, the interpolation method for the case where the reference region is LSC is referred to as the LSC interpolation method.

[0127] As another example, the reference area is the MIV atlas and the boundaries between different patches are A -1 A case is described where A0 is located. The interpolation filter coefficients in Table 1 can be utilized during inter prediction. For example, the pixel values ​​in the patch including A0 can be extended to the values ​​of pixel locations outside the patch boundary, and the reference region can be interpolated. A -1 The sample values ​​at the location between A0 and A1 can be interpolated as in mathematical expression 3. Hereinafter, the interpolation method for the case where the reference area is the MIV atlas is referred to as the atlas interpolation method.

[0128]

[0129] The present disclosure solves problems according to existing technologies by selecting / applying an interpolation method suitable for the type of image content of a reference area used for prediction, as in the examples described above.

[0130] FIG. 14 is an exemplary diagram showing selection of an interpolation method according to one embodiment of the present disclosure.

[0131] A video decoding device can analyze video content using a reference region to be used for prediction and encoding information of the reference region. Based on the analyzed content, the video decoding device can select one of the methods available in the interpolation scheme pool and interpolate the reference region using the selected method.

[0132] As an example implementation of the present disclosure, a method for inferring an interpolation method based on analysis of a reference region is described. In this example implementation, an image decoding device infers an interpolation method by analyzing the reference region by applying a predefined filter to the reference region.

[0133] When a response map is obtained by applying a predetermined filter to an image of a reference area and the distribution of response map values ​​is analyzed, the analysis results may vary depending on the type of image. For example, a distribution of filter response maps can be generated by applying a Sobel filter, as shown in Fig. 15, to CC and LSC. In the case of CC, when the Sobel filters Gx and Gy are applied to the luma component, the proportion of zeros among the values ​​of the response map is observed to be low. For example, in the horizontal and vertical luma response maps related to a specific CC used in the experiment, the proportions of zeros are observed to be 48.29% and 47.31%, respectively. In LSC, since the distribution of pixel values ​​is flat or changes abruptly, the proportion of zeros among the values ​​of the response map is observed to be high. For example, in the horizontal and vertical luma response maps related to a specific LSC used in the experiment, the proportions of zeros are observed to be 98.23% and 96.60%, respectively.

[0134] According to the above-described observation results, if the ratio of zero values ​​in the filter response map is greater than a predetermined threshold (e.g., 80%), the image decoding device can determine the corresponding image as an LSC image. On the other hand, if the ratio of zero values ​​is less than the threshold, the image decoding device can determine the corresponding image as a CC. When interpolating a reference region, the image decoding device can select one interpolation method among various interpolation methods as described above according to the type of the image, such as CC or LSC. If determined as LSC, the image decoding device can use the LSC interpolation method. If determined as CC, the image decoding device can use the CC interpolation method as in mathematical expression 2.

[0135] An image decoding device can infer an interpolation method suitable for the current image using a deep learning-based classification network, as illustrated in FIG. 16. The image decoding device inputs the current image into the classification network, generates a classification result, and selects an interpolation method suitable for the current image based on the classification result. As an example, a multi-layer perceptron (MLP) can be utilized as the classification network. An activation function can be applied to the back of each layer constituting the multi-layer neural network. As another example, a convolutional neural network (CNN) suitable for image processing can be utilized as the classification network. The classification network can infer an interpolation method from the current image based on a plurality of convolutional layers constituting the CNN. An activation function can be applied to the back of each convolutional layer constituting the CNN. The image encoding device can update the parameters of the classification network by training the classification network using training images and labels indicating the types of training images (e.g., CC, LSC, MIV atlas). The parameters of the classification network can be shared in advance between the image encoding device and the image decoding device.

[0136] The video decoding device can select an interpolation method by utilizing encoding information of the reference area (e.g., pred_mode_ibc_flag, a syntax element indicating whether IBC mode is applied, intra_bdpcm_luma_flag, a syntax element indicating whether BDPCM (Block-based Delta Pulse Code Modulation) is applied, transform_skip_flag, a syntax element indicating whether TS (Transform Skip) is applied, etc.) and / or pixel values ​​of the reference area.

[0137] For example, IBC generates a predictor based on the pixel values ​​of the area indicated by the BV in the restored area of ​​the current picture. IBC is frequently used in image areas where many repetitive patterns appear, such as LSC and lenslet images. If the value of pred_mode_ibc_flag related to the reference area is 1 and the reference block is encoded with IBC, it may be advantageous to use an interpolation filter that causes less smoothing when encoding IBC using the block.

[0138] LSC and lenslet images generate different response maps when a Sobel filter is applied. For example, in the horizontal and vertical luma response maps associated with a specific LSC used in the experiment, the zero ratios are observed to be 54.95% and 41.66%, respectively. In the horizontal and vertical luma response maps associated with a specific lenslet image used in the experiment, the zero ratios are observed to be 1.84% and 1.60%, respectively. Therefore, when the reference region of the current block is encoded with IBC, the image decoding device can apply a predetermined filter, such as a Sobel filter, to the reference region and calculate the zero ratio in the response map. If the zero ratio is greater than a predetermined threshold (e.g., 45%), the region can be regarded as an LSC, and if the zero ratio is less than the predetermined threshold, the region can be regarded as a Lenslet image. The image decoding device can apply different interpolation methods to the reference region depending on the type of the identified image.

[0139] TS quantizes residual samples directly without transforming them and encodes the quantized residual samples to efficiently compress residual samples when non-zero transform coefficients of residual samples of the current block are not concentrated in low frequencies. BDPCM intra-predicts a block in the vertical and horizontal directions, and then signals and parses the differences between residual samples at adjacent pixel locations. BDPCM can be used in a block to which TS is applied. The residual sample value R at pixel location (i,j) within the block (0 ≤ i ≤ W - 1, 0 ≤ j ≤ H - 1, W and H are the width and height of the current transformed block) i,j Wow R i,j Q(R) is the quantized value of i,j ), the residual signal calculated by applying BDPCM in the vertical and horizontal directions for prediction are expressed as mathematical expressions 4 and 5, respectively.

[0140]

[0141]

[0142] TS and BDPCM are also frequently used in screen contents like IBC. If the values ​​of transform_skp_flag, intra_bdpcm_luma_flag, and intra_bdpcm_chroma_flag related to the reference region are 1, and the reference block is encoded with TS and BDPCM, it may be advantageous to use an interpolation filter that causes less smoothing when encoding TS and BDPCM using the block. If the reference region of the current block does not use both TS and BDPCM, the video decoding device may regard the reference region as a CC and use the CC interpolation method. If the reference region of the current block uses both TS and BDPCM, the video decoding device may regard the reference region as an LSC and may not perform interpolation. If the reference region of the current block uses TS but does not use BDPCM, the video decoding device may apply a predetermined filter to the reference region to generate a response map, and analyze the response map to select an interpolation method.

[0143] According to this implementation example, as shown in Fig. 17, when a reference region to be used for inter prediction of a current block is encoded according to different modes such as intra prediction and IBC, the video decoding device can divide the reference region according to the boundaries between blocks in the reference picture and separately perform video content analysis for each divided region. The video decoding device can infer an interpolation method suitable for each divided region.

[0144] As another example, additionally, the encoding information of the current block can be utilized to select different interpolation methods, for example, different interpolation filters.

[0145] For example, when inter-predicting a current block, depending on whether the prediction block is generated based on bidirectional prediction or unidirectional prediction, the video decoding device may apply different interpolation methods, for example, different interpolation filters, to the reference region. The interpolation filters exemplified in Table 1 are utilized for unidirectional prediction, and an interpolation filter with a sharper property on the time axis than the filters exemplified in Table 1 may be utilized for bidirectional prediction.

[0146] The Template-based Intra Mode Prediction (TIMD) mode predicts templates by applying candidate intra prediction modes to reference samples of the template of the current block, and compares the matching costs between the predicted templates and the restored template to derive the optimal intra prediction mode. Prediction modes included in the Most Probable Mode (MPM) list are used as candidate intra prediction modes. The matching cost can be SAD (Sum of Absolute Differences) or SATD (Sum of Absolute Transformed Differences). The candidate mode with the minimum matching cost can be selected as the optimal intra prediction mode.

[0147] If the current block is intra-predicted, the image decoding device can select an interpolation method, i.e., an interpolation filter, based on the MDIS (Mode Dependent Intra reference sample Smoothing) condition and apply the selected interpolation filter to the reference samples of the current block. A 6-tap cubic filter or a 4 / 6-tap Gaussian filter can be used as the interpolation filter. The MDIS condition includes the size of the current block, the directionality of the prediction mode, etc.

[0148] As an example, assume that the intra prediction mode of the current block is derived according to the TIMD mode. Based on the derived intra prediction mode, a template is derived, and an image decoding device can predict the template by applying different interpolation methods, i.e., interpolation filters (e.g., a 6-tap cubic filter, a 4 / 6-tap Gaussian filter, etc.) to reference samples of the template, and calculate a matching cost between the predicted template and the restored template. The image decoding device can compare matching costs according to the types of interpolation filters to select the interpolation filter of the derived intra prediction mode. If the matching costs according to the types of interpolation filters are the same, the image decoding device can select the interpolation filter of the derived intra prediction mode based on the existing MDIS.

[0149] As another example, when the current block is intra-predicted, the image decoding device may select an interpolation method, i.e., an interpolation filter, based on conditions such as the size of the current block, the directionality of the prediction mode, etc. At this time, the applied conditions may be different from the MDIS conditions described above. The interpolation filter may additionally include a 2-tap interpolation filter in addition to a 6-tap cubic filter and a 4 / 6-tap Gaussian filter. For example, when the size of the current block is greater than 32 and the directionality of the prediction mode is an integer angle, a 2-tap interpolation filter may be applied to reference samples first. Thereafter, when the size of the current block is greater than 32 and the directionality of the prediction mode is not an integer angle, a 4 / 6-tap Gaussian filter may be applied to the reference samples. On the other hand, when the size of the current block is less than or equal to 32 or the directionality of the prediction mode is an integer angle, a 6-tap cubic filter may be used.

[0150] According to the present implementation example, whether to selectively interpolate a reference region can be signaled at various levels such as SPS (Sequence Parameter Set), VPS (Video Parameter Set), PPS (Picture Parameter Set), PH (Picture Header), SH (Slice Header), CTU (Coding Tree Unit), CU (Coding Unit), etc. By parsing the signaled information, the video decoding device can determine whether to selectively interpolate a reference region.

[0151] A video encoding device can use a syntax element (e.g., selective_interp_flag) to indicate whether to selectively interpolate a reference region. When selective_interp_flag is 1, it indicates that an interpolation method is selectively used for the reference region based on a predetermined filter, a classification network, and a coding mode of the reference region. When selective_interp_flag is 0, it indicates that a conventional interpolation method is used for the reference region. A video decoding device can parse the syntax from the bitstream to determine whether to selectively interpolate the reference region. After selecting an interpolation method based on analysis of the reference region, the video encoding device and the video decoding device can interpolate the reference region according to the method to generate a predictor. The video decoding device can apply inverse quantization and inverse transformation to decoded quantized transform coefficients to generate a residual block, and reconstruct the current block by adding the predictor and the residual block.

[0152] As another implementation example according to the present disclosure, a method for signaling and parsing an interpolation method is described. In the implementation example, an image encoding device and an image decoding device signal and parse an interpolation method applied to a reference region using syntax elements.

[0153] A video encoding device can use a syntax element (e.g., interp_method_idx) to indicate an interpolation method to be applied during intra prediction, inter prediction, or IBC prediction. A video decoding device can parse the syntax from a bitstream to determine the interpolation method to be applied during intra prediction, inter prediction, or IBC prediction. The syntax can be signaled and parsed at various levels, such as SPS, VPS, PPS, PH, SH, CTU, and CU. The interpolation method corresponding to the syntax element can be expressed as shown in Table 2.

[0154]

[0155] When the value of interp_method_idx is 0, the image decoding device can interpolate the reference region by applying a filter predefined in the existing technology to the reference region. When the value of interp_method_idx is 1, the image decoding device may not apply interpolation to the reference region. When the value of interp_method_idx is 2, the image decoding device can use the interpolation filter predefined in the existing technology, but extend the pixel values ​​within the patch to the values ​​of pixel locations outside the patch boundary, and interpolate the reference region.

[0156] FIG. 18 is a flowchart illustrating a method of encoding a current block performed by an image encoding device according to one embodiment of the present disclosure.

[0157] The image encoding device obtains activation information indicating whether to selectively interpolate a reference area from a higher level (S1800).

[0158] Later, the video encoding device can encode activation information.

[0159] The video encoding device checks the activation information (S1802).

[0160] If the activation information indicates that the reference area is to be selectively interpolated (Yes in S1802), the image encoding device performs the following processes.

[0161] The image encoding device sets a reference area of ​​the current block and obtains encoding information of the reference area (S1804).

[0162] If the prediction mode of the current block is inter prediction, the video encoding device can set the region indicated by the motion vector as a reference region. If the prediction mode of the current block is intra prediction, the video encoding device can set reference samples around the current block as a reference region. If the prediction mode of the current block is IBC (Intra Block Copy), the video encoding device can set the region indicated by the block vector as a reference region. Alternatively, if the prediction mode of the current block is intraTMP (Intra Template Matching Prediction), the video encoding device can set the region calculated according to template matching as a reference region.

[0163] As an example, a video encoding device can obtain an interpolation method or an index indicating an interpolation method from a higher level.

[0164] As another example, the image encoding device selects an interpolation method from a pool of interpolation methods based on the reference area and encoding information (S1806).

[0165] An image encoding device can generate a response map by applying a preset filter to a reference region. The image encoding device can determine the type of the reference region by calculating the proportion of zeros among the values ​​in the response map and comparing the proportion of zeros with a preset threshold. The image encoding device can determine an interpolation method based on the type of the reference image.

[0166] As another example, if the encoding information of the reference region indicates that IBC is applied to the reference region, the video encoding device may generate a response map by applying a preset filter to the reference region. The video encoding device may determine the type of the reference region by calculating the ratio of zeros among the values ​​of the response map and comparing the ratio of zeros with a preset threshold. The video encoding device may determine an interpolation method based on the type of the reference region.

[0167] The video encoding device applies the selected interpolation method to the reference area to generate a prediction block of the current block (S1808).

[0168] If the activation information does not indicate that the reference area is to be selectively interpolated (No in S1802), the image encoding device applies the existing interpolation method to the reference area to generate a prediction block of the current block (S1820).

[0169] Thereafter, the video encoding device can generate a residual block by subtracting the predicted block from the current block. The video encoding device can apply transformation / quantization to the residual block to generate quantized transform coefficients, and encode the quantized transform coefficients to generate a bitstream.

[0170] FIG. 19 is a flowchart illustrating a method for restoring a current block performed by an image decoding device according to one embodiment of the present disclosure.

[0171] The video decoding device decodes activation information indicating whether to selectively interpolate a reference area from a bitstream (S1900).

[0172] The video decryption device checks the activation information (S1902).

[0173] If the activation information indicates that the reference area is to be selectively interpolated (Yes in S1902), the image decoding device performs the following processes.

[0174] The image decoding device sets a reference area of ​​the current block and obtains encoding information of the reference area (S1904).

[0175] If the prediction mode of the current block is inter prediction, the video decoding device can set the region indicated by the motion vector as a reference region. If the prediction mode of the current block is intra prediction, the video decoding device can set reference samples around the current block as a reference region. If the prediction mode of the current block is IBC (Intra Block Copy), the video decoding device can set the region indicated by the block vector as a reference region. Alternatively, if the prediction mode of the current block is intraTMP (Intra Template Matching Prediction), the video decoding device can set the region calculated according to template matching as a reference region.

[0176] As an example, a video decoding device can decode an interpolation method, or an index indicating an interpolation method, from a bitstream.

[0177] As another example, the image decoding device selects an interpolation method from the interpolation method pool based on the reference area and encoding information (S1906).

[0178] A video decoding device can generate a response map by applying a preset filter to a reference region. The video decoding device can determine the type of the reference region by calculating the ratio of zeros among the values ​​of the response map and comparing the ratio of zeros with a preset threshold. The video decoding device can determine an interpolation method based on the type of the reference image. For example, if the type of the reference region is determined to be camera-captured content, the video decoding device can use a preset interpolation filter as the interpolation method. If the type of the reference region is determined to be legacy screen content, the video decoding device can omit the use of a preset interpolation filter as the interpolation method.

[0179] As another example, if the reference area is an MIV (MPEG Immersive Video) atlas, based on the encoding information of the reference area, the image decoding device can extend the pixel values ​​within the patch to values ​​at pixel locations outside the patch boundary as an interpolation method and use a predefined interpolation filter.

[0180] As another example, if the encoding information of the reference region indicates that IBC is applied to the reference region, the image decoding device may generate a response map by applying a preset filter to the reference region. The image decoding device may determine the type of the reference region by calculating the ratio of zeros among the values ​​of the response map and comparing the ratio of zeros with a preset threshold. The image decoding device may determine the interpolation method based on the type of the reference region (e.g., camera-acquired image or legacy screen content).

[0181] As another example, a video decoding device can determine whether to apply TS (Transform Skip) and BDPCM (Block-based Delta Pulse Code Modulation) to a reference region based on encoding information of the reference region. If neither TS nor BDPCM is applied to the reference region, the video decoding device can determine the type of the reference region based on a camera-captured image. If both TS and BDPCM are applied to the reference region, the video decoding device can determine the type of the reference region as legacy screen content. The video decoding device can determine an interpolation method based on the type of the reference region.

[0182] Meanwhile, if TS is applied to the reference region but BDPCM is not applied, the video decoding device can generate a response map by applying a preset filter to the reference region. The video decoding device can determine the type of the reference region by calculating the ratio of zeros among the values ​​of the response map and comparing the ratio of zeros with a preset threshold. The video decoding device can determine the interpolation method based on the type of the reference region (e.g., camera-acquired image or legacy screen content).

[0183] The image decoding device applies the selected interpolation method to the reference area to generate a prediction block of the current block (S1908).

[0184] If the activation information does not indicate that the reference area is to be selectively interpolated (No in S1902), the image decoding device applies the existing interpolation method to the reference area to generate a prediction block of the current block (S1920).

[0185] Thereafter, the video decoding device can decode quantized transform coefficients from the bitstream and apply inverse quantization / inverse transformation to the quantized transform coefficients to generate a residual block. The video decoding device can generate a restored block of the current block by adding the residual block and the prediction block.

[0186] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0187] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.

[0188] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0189] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0190]

[0191]

[0192] CROSS-REFERENCE TO RELATED APPLICATION

[0193] This patent application claims priority to Korean Patent Application No. 10-2024-0050610, filed on April 16, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. In a method for restoring a current block performed by an image decryption device, A step of setting a reference area of ​​the current block and obtaining encoding information of the reference area; A step of selecting an interpolation scheme from an interpolation scheme pool based on the above reference area and the above encoding information; and A step of predicting the current block by applying the selected interpolation method to the reference area. A method comprising:

2. In paragraph 1, A step of decoding activation information indicating whether to selectively interpolate the above reference area from the bitstream; and Steps to check the above activation information Including more, If the above activation information indicates that the above reference area is to be selectively interpolated, A method for performing the step of selecting the above interpolation method.

3. In paragraph 1, The step of obtaining the above reference area is: A method for setting an area indicated by a motion vector as the reference area when the prediction mode of the current block is inter prediction, or setting reference samples around the current block as the reference area when the prediction mode of the current block is intra prediction, or setting an area indicated by a block vector as the reference area when the prediction mode of the current block is IBC (Intra Block Copy), or setting an area calculated according to template matching as the reference area when the prediction mode of the current block is intraTMP (Intra Template Matching Prediction).

4. In paragraph 1, The step of selecting the above interpolation method is: A step of generating a response map by applying a preset filter to the above reference area; A step of calculating the ratio of zero among the values ​​of the above reaction map; and A step of determining the type of the reference area based on the ratio of the above spirits A method comprising:

5. In paragraph 4, The step of selecting the above interpolation method is: A method of using a predefined interpolation filter as the interpolation method when the type of the above reference area is determined as a camera-captured content.

6. In paragraph 4, The step of selecting the above interpolation method is: A method for omitting the use of a predefined interpolation filter as the interpolation method when the type of the above reference area is determined to be legacy screen content.

7. In paragraph 1, The step of selecting the above interpolation method is: A method of expanding pixel values ​​within a patch of the MIV atlas to values ​​at pixel locations outside the patch boundary, and using a predefined interpolation filter, according to encoding information of the reference area, when the reference area is an MIV (MPEG Immersive Video) atlas, as the interpolation method.

8. In paragraph 1, The step of selecting the above interpolation method is: If the encoding information of the above reference area indicates that IBC (Intra Block Copy) is applied to the above reference area, A step of generating a reaction map by applying a preset filter to the above reference area; A step of calculating the ratio of zero among the values ​​of the above reaction map; A step of determining the type of the reference area based on the ratio of the above spirit; and A step of determining the interpolation method based on the type of the above reference area. A method comprising:

9. In paragraph 1, The step of selecting the above interpolation method is: A step of determining whether to apply TS (Transform Skip) and BDPCM (Block-based Delta Pulse Code Modulation) in the reference area based on the encoding information of the reference area, A step of determining the type of the reference area by a camera-acquired image when neither the above TS nor the above BDPCM is applied; When both the above TS and the above BDPCM are applied, a step of determining the type of the reference area as legacy screen content; and A step of determining the interpolation method based on the type of the above reference area. A method further comprising:

10. In paragraph 9, The step of selecting the above interpolation method is: If the above TS applies but the above BDPCM does not apply, A step of generating a reaction map by applying a preset filter to the above reference area; A step of determining the type of the reference area based on the values ​​of the reaction map; and A step of determining the interpolation method based on the type of the above reference area. A method further comprising:

11. In a method for encoding a current block performed by a video encoding device, A step of obtaining a reference area of ​​the current block and encoding information of the reference area; A step of selecting an interpolation scheme from an interpolation scheme pool based on the above reference area and the above encoding information; and A step of predicting the current block by applying the selected interpolation method to the reference area. A method comprising:

12. In paragraph 11, A step of obtaining activation information from a higher level indicating whether to selectively interpolate the above reference area; Step of checking the above activation information; and Step of encoding the above activation information Including more, If the above activation information indicates that the above reference area is to be selectively interpolated, A method for performing the step of selecting the above interpolation method. A method further comprising:

13. In paragraph 11, The step of selecting the above interpolation method is: A step of generating a response map by applying a preset filter to the above reference area; A step of calculating the ratio of zero among the values ​​of the above reaction map; A step of determining the type of the reference area based on the ratio of the above spirit; and A step of determining the interpolation method based on the type of the above reference area. A method comprising:

14. In paragraph 11, The step of selecting the above interpolation method is: If the encoding information of the above reference area indicates that IBC (Intra Block Copy) is applied to the above reference area, A step of generating a reaction map by applying a preset filter to the above reference area; A step of determining the type of the reference area based on the values ​​of the reaction map; and A step of determining the interpolation method based on the type of the above reference area. A method comprising:

15. In a method for providing video data to a video decoding device, A step of encoding the above video data into a bitstream; and A step of transmitting the above bitstream to the image decoding device Including, The step of encoding the above video data is: A step of obtaining a reference area of ​​the current block and encoding information of the reference area; A step of selecting an interpolation scheme from an interpolation scheme pool based on the above reference area and the above encoding information; and A step of predicting the current block by applying the selected interpolation method to the reference area. A method comprising:

Images