Non-linear modeling-based template matching prediction
The use of nonlinear modeling-based template matching techniques in video coding addresses the inefficiencies of current compression methods by applying cross-component discrete mapping models or nonlinear filters, resulting in improved video quality and reduced computational complexity.
Patent Information
- Application Number
- PCT/KR2024/004426
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-04-03
- Filing Date
- 2024-04-04
- Publication Date
- 2024-11-28
AI Technical Summary
Current video compression technologies, such as H.264/AVC, HEVC, and VVC, face increased computational complexity and reduced coding efficiency due to the growing size, resolution, and frame rate of video data, necessitating a more efficient compression method that improves picture quality and reduces hardware resource requirements.
The implementation of a video coding method using a nonlinear modeling-based template matching technique, which involves cross-component discrete mapping models or nonlinear filters to enhance prediction accuracy and reduce computational complexity by applying discrete mapping models or nonlinear filters to the video data during the encoding and decoding processes.
This approach improves both objective video coding efficiency and subjective video quality by reducing computational complexity and enhancing prediction accuracy, thereby addressing the limitations of existing compression technologies.
Smart Images

Figure KR2024004426_28112024_PF_FP_ABST
Abstract
Description
Template matching prediction based on nonlinear modeling
[0001] The present disclosure relates to a video coding method and apparatus using optional multiple phrases.
[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.
[0003] Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.
[0004] Therefore, when storing or transmitting video data, an encoder is used to compress the video data and store or transmit it, and a decoder receives the compressed video data, decompresses it, and plays it back. Examples of such video compression technologies include H.264 / AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves encoding efficiency by about 30% compared to HEVC.
[0005] However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.
[0006] In encoding / decoding using template matching-based prediction, a reference block corresponding to a template most similar to the template of the current block is used as a matching block. During template matching-based decoding, the decoder searches for a matching block based on template matching between the templates of the reference blocks and the template of the current block, and determines the prediction block of the current block as the searched matching block. The decoder uses the Sum of Absolute Differences (SAD) or the Sum of Squared Errors (SSE) to estimate the similarity according to template matching. Since template matching is performed using multiple reconstructed reference blocks, template matching-based prediction increases the computational complexity of the image decoding process. To reduce the computational complexity, a search region for searching for a reference block is defined, and template matching is performed. As illustrated in Fig. 6, R1, R2, R3, and R4 correspond to search regions for template searching.
[0007] Meanwhile, filtering can be applied to intra-template matching prediction (intraTMP). The filter is applied to a reference block, and the filtered reference block is used as the prediction block of the current block. For example, the filter is a 6-tap linear filter, including 5 luma taps and 1 bias term. The filter coefficients are derived for each block using a regression model that minimizes the mean square error (MSE) between the output of the filter generated from the reference template and the samples of the current template. A filter flag indicating whether to use the linear filter is transmitted from the encoder to the decoder at the CU level. Furthermore, the use of the linear filter is considered a submode of intra-template matching prediction. That is, the filter flag is signaled when the template matching flag indicating whether to use intra-template matching prediction is true. Meanwhile, to improve video encoding efficiency and enhance video quality, it is necessary to consider utilizing a non-linear model in the aforementioned filtering process.
[0008] The present disclosure aims to provide a video coding method and device that utilize a discrete mapping model or nonlinear filter-based template matching technique, and a cross-component discrete mapping model or cross-component nonlinear model based on the template matching technique, in template matching prediction (TMP).
[0009] According to an embodiment of the present disclosure, a method for restoring a current block, performed by an image decoding device, is provided, characterized in that the method comprises: searching for a reference block based on template matching between templates of blocks in a search area and a template of the current block or a block vector of the current block, and determining a prediction block of the current block using the searched reference block; obtaining a shape of a filter of the current block, wherein the shape of the filter is a cross-component discrete mapping model or a cross-component nonlinear filter; calculating coefficients of the filter using the reference block; and applying filtering to the prediction block using the filter.
[0010] According to another embodiment of the present disclosure, a method for encoding a current block, performed by a video encoding device, is provided, comprising: generating a first prediction block of the current block using a prediction mode excluding template matching and an IBC (Intra Block Copy) mode; searching for a reference block based on template matching between templates of blocks in a search area and a template of the current block or a block vector of the current block, and determining a second prediction block of the current block using the searched reference block; determining a form of a filter of the current block, wherein the form of the filter is a cross-component discrete mapping model or a cross-component nonlinear filter; calculating coefficients of the filter using the reference block; and applying the filtering to the second prediction block using the filter to generate a filtered second prediction block.
[0011] According to another embodiment of the present disclosure, a method for providing video data to a video decoding device is provided, comprising: encoding the video data into a bitstream; and transmitting the bitstream to the video decoding device, wherein the encoding the video data comprises: generating a first prediction block of a current block using a prediction mode other than template matching and an Intra Block Copy (IBC) mode; searching for a reference block based on template matching between templates of blocks in a search region and a template of the current block or a block vector of the current block, and determining a second prediction block of the current block using the searched reference block; determining a form of a filter of the current block, wherein the form of the filter is a cross-component discrete mapping model or a cross-component nonlinear filter; calculating coefficients of the filter using the reference block; and applying the filtering to the second prediction block using the filter to generate a filtered second prediction block.
[0012] As described above, according to the present embodiment, in template matching prediction, a video coding method and device using a discrete mapping model or a nonlinear filter-based template matching technique and a cross-component discrete mapping model or a cross-component nonlinear model based on the template matching technique are provided, thereby making it possible to improve objective video encoding efficiency and subjective video quality.
[0013] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.
[0014] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.
[0015] FIGS. 3A and 3B are diagrams illustrating multiple intra prediction modes, including wide-angle intra prediction modes.
[0016] Figure 4 is an example diagram of the surrounding blocks of the current block.
[0017] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.
[0018] FIG. 6 is an exemplary diagram showing a search area for template search according to one embodiment of the present disclosure.
[0019] FIG. 7 is a flowchart illustrating restoration of a current block performed by an image decoding device according to one embodiment of the present disclosure.
[0020] FIG. 8 is an exemplary diagram showing a filter for filtering a prediction block according to template matching, according to one embodiment of the present disclosure.
[0021] Figure 9 is an example diagram showing the MMLM (Multi-model Linear Model) mode.
[0022] Figure 10 is an example diagram showing the DIMD (Decoder-side Intra Mode Derivation) chroma mode.
[0023] Figure 11 is an example diagram showing a reference area used for deriving filter coefficients.
[0024] Figure 12 is an example diagram showing Sobel filters for gradient calculation.
[0025] Figure 13 is an example diagram showing candidate luma blocks in DBV (Direct Block Vector) mode.
[0026] Figure 14 is an example diagram showing prediction of a chroma block in DBV mode.
[0027] Figure 15 is an example diagram showing co-located luma sample values according to the application of candidate downsampling filters.
[0028] FIG. 16 is an exemplary diagram illustrating nonlinear modeling-based template matching prediction according to one embodiment of the present disclosure.
[0029] FIG. 17 is an exemplary diagram showing the order of checking luma samples in a discrete mapping model according to one embodiment of the present disclosure.
[0030] FIG. 18 is an exemplary diagram showing a nonlinear filter in the shape of an ALF (Adaptive Loop Filter) according to one embodiment of the present disclosure.
[0031] FIG. 19 is an exemplary diagram illustrating nonlinear modeling-based template matching prediction according to another embodiment of the present disclosure.
[0032] FIG. 20 is a flowchart illustrating a method of encoding a current block performed by an image encoding device according to one embodiment of the present disclosure.
[0033] FIG. 21 is a flowchart illustrating a method of restoring a current block performed by an image decoding device according to one embodiment of the present disclosure.
[0034] Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.
[0035] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to the illustration in FIG. 1.
[0036] The video encoding device may be configured to include a picture segmentation unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).
[0037] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.
[0038] A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). And, information commonly referenced by one or more SPS is encoded in a Video Parameter Set (VPS). In addition, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.
[0039] The picture segmentation unit (110) determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as the syntax of SPS or PPS and transmitted to the image decoding device.
[0040] The picture segmentation unit (110) divides each picture constituting an image into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a CU, which is a basic unit of encoding.
[0041] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).
[0042] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.
[0043] As illustrated in FIG. 2, a CTU may first be split into a QT structure. The quadtree splitting may be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further split into one or more of the BT structure or the TT structure. There may be multiple splitting directions in the BT structure and / or the TT structure. For example, there may be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As illustrated in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.
[0044] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the CU split flag (split_cu_flag) value indicates that the node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is a basic unit of encoding. If the CU split flag (split_cu_flag) value indicates that the node is split, the video encoding device starts encoding from the first flag in the above-described manner.
[0045] As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.
[0046] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.
[0047] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).
[0048] In general, each current block within a picture can be predictively coded. Prediction of the current block can typically be performed using either intra-prediction (using data from the picture containing the current block) or inter-prediction (using data from a picture coded before the picture containing the current block). Inter-prediction encompasses both unidirectional and bidirectional prediction.
[0049] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3a, the multiple intra prediction modes may include two non-directional modes including the Planar mode and the DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently depending on each prediction mode.
[0050] For efficient directional prediction for a rectangular current block, directional modes (intra prediction modes 67 to 80 and -1 to -14) indicated by dotted arrows in Fig. 3b may be additionally used. These may be referred to as "wide-angle intra-prediction modes." In Fig. 3b, the arrows point to corresponding reference samples used for prediction, and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide-angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra-prediction modes, some wide-angle intra-prediction modes available for the current block may be determined based on the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (intra prediction modes 67 to 80) having an angle less than 45 degrees are available when the current block is a rectangular shape whose height is smaller than its width, and wide-angle intra prediction modes (intra prediction modes -1 to -14) having an angle greater than -135 degrees are available when the current block is a rectangular shape whose width is larger than its height.
[0051] The intra prediction unit (122) can determine an intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate bit-rate distortion values using rate-distortion analysis for multiple tested intra prediction modes and select an intra prediction mode with the best bit-rate distortion characteristics among the tested modes.
[0052] The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.
[0053] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma component and the chroma component. The motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.
[0054] The inter prediction unit (124) may perform interpolation on a reference picture or a reference block to improve prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision or resolution of the motion vector can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.
[0055] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the entropy encoding unit (155). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.
[0056] Various methods can be used to minimize the number of bits required to encode motion information.
[0057] For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the image decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.
[0058] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.
[0059] As the surrounding blocks for deriving merge candidates, all or part of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the block at the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.
[0060] The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, the merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the video decoding device.
[0061] Merge Skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high encoding efficiency can be achieved for low-motion images, still images, and screen content images.
[0062] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.
[0063] Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.
[0064] In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block in the current picture as shown in FIG. 4 may be used. In addition, a block located in a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block co-located with the current block in the reference picture or blocks adjacent to the block in the co-located block may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.
[0065] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.
[0066] The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information to identify the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.
[0067] Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.
[0068] The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.
[0069] The transformation unit (140) transforms residual signals within a residual block having pixel values in a spatial domain into transform coefficients in a frequency domain. The transformation unit (140) may transform the residual signals within the residual block using the entire size of the residual block as a transformation unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transformation units to perform the transformation. Alternatively, the residual signals may be transformed using only the transformation domain sub-block as a transformation unit by dividing the sub-blocks into two sub-blocks, that is, a transformation domain and a non-transform domain. Here, the transformation domain sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only a sub-block has been converted, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. In addition, the size of the conversion area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.
[0070] Meanwhile, the transformation unit (140) can individually perform transformations on the residual block in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation function pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation function pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.
[0071] The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.
[0072] The rearrangement unit (150) can perform rearrangement of coefficient values for quantized residual values.
[0073] The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region by using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, the scanning method to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan.
[0074] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.
[0075] In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about an intra prediction mode) or inter prediction information (information about an encoding mode of motion information (merge mode or AMVP mode), a merge index in the case of a merge mode, and a reference picture index and a differential motion vector in the case of an AMVP mode) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about a quantization parameter and information about a quantization matrix.
[0076] The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.
[0077] An adder (170) adds the restored residual block and the predicted block generated by the prediction unit (120) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting the next block.
[0078] The loop filter unit (180) performs filtering on restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filter unit (180) may include all or part of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186) as an in-loop filter.
[0079] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter (184) and the ALF (186) perform additional filtering on the deblocking-filtered image. The SAO filter (184) and the ALF (186) are filters used to compensate for the differences between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in units of CTUs. In contrast, the ALF (186) performs block-based filtering, and compensates for distortion by applying different filters by distinguishing the edges and degrees of variation of the corresponding block. Information on filter coefficients to be used in the ALF can be encoded and signaled to an image decoding device.
[0080] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.
[0081] The video encoding device can store the bitstream of encoded video data on a non-transitory storage medium or transmit it to the video decoding device using a communication network.
[0082] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.
[0083] The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).
[0084] Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.
[0085] The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information, information on residual signals, etc. required to restore the current block.
[0086] The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.
[0087] For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (mtt_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Accordingly, each node below the leaf nodes of QT are split recursively into the BT or TT structure.
[0088] As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.
[0089] As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.
[0090] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded by using the division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.
[0091] Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about residual signals.
[0092] The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.
[0093] The inverse quantization unit (520) inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.
[0094] The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.
[0095] In addition, when the inverse transform unit (530) inversely transforms only a portion of a transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only a sub-block of the transform block has been transformed, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block, and inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to restore residual signals, and fills “0” values with residual signals for areas that have not been inversely transformed, thereby generating a final residual block for the current block.
[0096] In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions using MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.
[0097] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.
[0098] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.
[0099] The inter prediction unit (544) uses the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.
[0100] An adder (550) adds the residual block output from the inverse transform unit (530) and the predicted block output from the inter prediction unit (544) or the intra prediction unit (542) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting a block to be decoded later.
[0101] The loop filter unit (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as in-loop filters. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) and the ALF (566) perform additional filtering on restored blocks after deblocking filtering to compensate for differences between restored pixels and original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about filter coefficients decoded from the non-stream.
[0102] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.
[0103]
[0104] The present embodiment relates to encoding and decoding of images (videos) as described above. More specifically, in template matching prediction, a video coding method and device are provided that utilize a template matching technique based on a discrete mapping model or a nonlinear filter, and a cross-component discrete mapping model or a cross-component nonlinear model based on a template matching technique.
[0105] The following embodiments may be performed by an intra prediction unit (122) within a video encoding apparatus. In addition, the following embodiments may be performed by an intra prediction unit (542) within a video decoding apparatus.
[0106] The video encoding device can generate signaling information related to the present embodiment in terms of rate distortion optimization in encoding the current block. The video encoding device can encode the signaling information using the entropy encoding unit (155) and then transmit it to the video decoding device. The video decoding device can decode the signaling information related to the decoding of the current block from the bitstream using the entropy decoding unit (510).
[0107] In the following description, the term "target block" may be used interchangeably with the current block or coding unit (CU). Alternatively, the term "target block" may also refer to a portion of a coding unit.
[0108] Also, a value of a flag being true indicates that the flag is set to 1. Also, a value of a flag being false indicates that the flag is set to 0.
[0109] I. Template Matching-Based Prediction
[0110] In encoding / decoding using template matching-based prediction, a reference block corresponding to a template most similar to the template of a current block is used as a matching block. In template matching-based decoding, an image decoding device searches for a matching block based on template matching between the templates of reference blocks and the template of the current block, and determines the prediction block of the current block as the searched matching block. In other words, the matching block represents a (selected) reference block according to template matching. Hereinafter, the template of the reference block is used interchangeably with the reference template, and the template of the current block is used interchangeably with the current template.
[0111] Video decoding devices use measures such as Sum of Absolute Differences (SAD) and Sum Squared Errors (SSE) to estimate similarity based on template matching. Since template matching is performed using multiple reconstructed reference blocks, template matching-based prediction increases the computational complexity of the video decoding process. To reduce the computational complexity, a search area for searching for reference blocks is defined, and template matching is performed. As illustrated in Fig. 6, R1, R2, R3, and R4 correspond to search areas for template searching. In the example of Fig. 6, (x, y) represents the location of the upper left sample of the current block.
[0112] Hereinafter, the mode that predicts the current block based on template matching is referred to as the template matching-based prediction mode. Furthermore, the method of intra-predicting the current block using the template matching-based prediction mode is referred to as template matching-based intra-prediction or intra-template matching prediction (intraTMP). Hereinafter, the terms template matching-based intra-prediction and template matching-based prediction are used interchangeably. Template matching and intra-template matching are used interchangeably.
[0113] Hereinafter, a method for predicting and restoring a current block using a template matching-based prediction mode is described using the city of FIG. 7.
[0114] FIG. 7 is a flowchart illustrating restoration of a current block performed by an image decoding device according to one embodiment of the present disclosure.
[0115] The video decoding device obtains the size of the current block (S700).
[0116] The video decoding device determines whether the size of the current block satisfies the range conditions (S702). The video decoding device determines whether the size of the current block satisfies the range conditions (e.g., the minimum block size condition and / or the maximum block size condition) for decoding based on template matching-based prediction. Depending on whether the range conditions are satisfied, the restoration process is divided into a restoration process using template matching and a restoration process not using template matching.
[0117] If the range condition is satisfied (Yes in S702), the video decoding device decodes the template matching flag (S704). The template matching flag indicates whether intra-template matching is used for the current block.
[0118] The video decoding device checks the template matching flag (S706).
[0119] If the template matching flag is true (Yes in S706), the video decoding device generates a prediction block of the current block based on template matching (S708).
[0120] If the range condition is not satisfied (No in S702) or the template matching flag is false (No in S706), the video decoding device generates a prediction block of the current block using a prediction mode other than template matching (S720). For example, intra prediction mode or inter prediction may be used instead of template matching.
[0121] Afterwards, the video decoding device can decode the residual block and then add the residual block and the prediction block to generate a restoration block of the current block.
[0122] Meanwhile, filtering can be applied to intra-template matching prediction. The filter is applied to the matching block, and ultimately, the filtered matching block is used as the prediction block of the current block. For example, the filter is a 6-tap linear filter, which is a cross filter including 5 luma taps and 1 bias term. The input to the spatial 5-tap component of the filter consists of the center (C) sample, the upper / north (N) sample, the lower / south (S) sample, the left / west (W) sample, and the right / east (E) sample of the matching block at the corresponding position to the sample of the current block, as shown in the example of Fig. 8. Hereinafter, the 6-tap filter is also referred to as a 5-tap filter without considering the bias term.
[0123] The output of the filter is calculated as in mathematical expression 1.
[0124]
[0125] In mathematical expression 1, B represents a bias term.
[0126] The filter coefficients (c0 to c5) shown in Equation 1 are derived for each block using a regression model that minimizes the Mean Square Error (MSE) between the filter output and the samples in the template of the current block. MSE minimization involves calculating the autocorrelation matrix between the reference template and the current template, LDL decomposition of the autocorrelation matrix, and calculating the filter coefficients according to back-substitution.
[0127] Meanwhile, a filter flag indicating whether to use a filter is transmitted from the video encoding device to the video decoding device at the CU level. Furthermore, the use of a linear filter is considered a submode of intra-template matching prediction. That is, if the template matching flag indicating whether to use intra-template matching prediction is true, the filter flag is signaled.
[0128] II. Chroma prediction mode of ECM (Enhanced Compression Model)
[0129] In the ECM corresponding to Beyond VVC, a Multi-model LM (MMLM) mode is added, which is an extension of the Cross-component Linear Model (CCLM) included in the existing VVC. The MMLM mode derives a linear model by applying the Least Mean Square (LMS) method to luma samples and improves accuracy by adjusting the slope of the linear model. As shown in the example of Fig. 9, the slope is adjusted using a linear function that maps luma values to chroma values based on a center point determined according to the average luma value of the reference sample. At this time, the linear function of the MMLM mode is derived from the linear function of the CCLM mode as in Equation 2.
[0130]
[0131] In mathematical expression 2, a and b are parameters representing the linear model of the CCLM mode illustrated in Fig. 9, and a' and b' are parameters representing the linear function of the MMLM mode. y r represents the reference luma sample value corresponding to the center point.
[0132] DIMD (Decoder-side Intra Mode Derivation) chroma determines the intra prediction mode of the current chroma block. As shown in the example of Fig. 10, the DIMD chroma mode is used based on the surrounding luma samples and chroma samples decoded in the surrounding rows and columns. The horizontal gradient and the vertical gradient are calculated for the luma samples, Cb, and Cr samples decoded at the same position in the current chroma block to derive a histogram of gradients. The prediction mode with the largest histogram value is used for intra prediction of the current chroma block. If the intra prediction mode derived according to the DIMD chroma mode is the same as the intra prediction mode according to the DM mode, the intra prediction mode with the second largest histogram value is used for intra prediction of the current chroma block. A CU level flag may be signaled to indicate whether the DIMD chroma mode is applied.
[0133] In ECM, two chroma prediction blocks can be fused together to generate final prediction signals as in Equation 3.
[0134]
[0135] In Equation 3, one of the two chroma prediction blocks is generated according to the Derived Mode (DM), the DIMD chroma mode, and one of the four basic modes (non-LM mode). The other prediction block may be generated according to the cross-component linear prediction mode. For example, pred0(i,j) may be predicted according to the non-LM (Linear Model) mode, and pred1(i,j) may be predicted according to the MMLM_LT mode. The four basic modes represent DC, Planar, horizontal mode (No. 18), and vertical mode (No. 50). The MMLM_LT mode represents the MMLM mode that uses the left and top samples to derive a linear model.
[0136] In Equation 3, the weights w0 and w1 are determined according to the intra prediction mode of the adjacent chroma blocks, and the shift is set to 2. If both the upper and left adjacent blocks are encoded in the LM mode, {w0, w1} = {1, 3}. If neither the upper and left adjacent blocks are encoded in the LM mode, {w0, w1} = {3, 1}. In other cases, {w0, w1} = {2, 2}.
[0137] As another example, final prediction signals can be generated as in Equation 4.
[0138]
[0139] In Equation 4, pred0(i,j) is predicted according to non-LM mode, and rec' L (i,j) represents the downsampled luma sample at the same location.
[0140] The Convolutional Cross-Component Model (CCCM) mode, similar to the CCLM mode, predicts chroma samples from reconstructed luma samples. When chroma subsampling is used, the CCCM mode compensates for the resolution difference between luma and chroma samples by downsampling the luma samples to align their positions. The CCCM mode computes the model using top, left, or top and left reference samples.
[0141] The convolutional 7-tap filter in CCCM mode consists of a 5-tap cross-shaped spatial component, a nonlinear term, and a bias term. The 5-tap component of the filter can be expressed as an example in Fig. 8. That is, the 5-tap component of the filter consists of the center (C) luma sample, below / south (S), left / west (W), right / east (E), and above / north (N) samples.
[0142] The nonlinear term P is expressed as a power of the central luma sample C as in Equation 5 and is scaled by the bit depth (bitDepth).
[0143]
[0144] For an image with a depth of 10 bits, the nonlinear term P is expressed as in mathematical expression 6.
[0145]
[0146] The bias term B represents the scalar offset between the input and output, and is the intermediate chroma value (512 for images with 10-bit depth). The output of the convolution filter is the filter coefficient c i It is calculated by convolution between the input values and clipped to the valid chroma sample range. The convolution filter finally calculates the predicted value as in Equation 7.
[0147]
[0148] Filter coefficient c i is calculated to minimize the Mean Square Error (MSE) between the predicted chroma sample and the reconstructed chroma sample in the reference region. In the example of Fig. 11, the six lines located at the top and left of the prediction unit (PU) represent the reference region of the chroma sample. The reference region extends to a size equal to the PU width and height, respectively, toward the lower left and upper right of the PU boundary. The hashed region around the reference region and the prediction unit is used when deriving a cross-shaped spatial filter, and the unusable region can be used using padding.
[0149] MSE minimization is calculated using the autocorrelation matrix for luma samples and the cross-correlation vector between the luma input and chroma output. The autocorrelation matrix is LDL decomposed, and the final filter coefficients are calculated by back-substitution. MSE minimization roughly follows the process of calculating ALF filter coefficients in ECM, but LDL decomposition is used instead of Cholesky decomposition to avoid square root operations.
[0150] The Gradient Linear Model (GLM) is used to predict chroma samples from luma sample gradients in the YUV 4:2:0 color format. GLM modes use two or three variables. When two variables are used, GLM derives a linear model using the luma sample gradient. The CCLM parameter derivation method and the linear transformation of the predicted samples remain unchanged. In a GLM using two variables, the chroma sample C is expressed as in Equation 8.
[0151]
[0152] In mathematical expression 8, α and β represent parameters of the linear model derived according to the CCLM mode, and G represents the gradient.
[0153] The GLM mode using three variables predicts chroma samples based on luma sample gradients and downsampled luma values. The model parameters of the three-variable GLM are derived using the LDL decomposition-based MSE minimization method used in CCCM, using adjacent six-sample rows and columns. In the three-variable GLM, the chroma sample C is expressed as in Equation 9.
[0154]
[0155] In Equation 9, G and L represent the luma sample gradient and the downsampled luma value, and α0, α1, and β represent the parameters of the GLM model.
[0156] In terms of signaling, if CCLM mode is enabled for the current CU, a flag indicating whether GLM is enabled for both the Cb and Cr components is signaled. If GLM is enabled, another flag indicating one of the two GLM modes is signaled. In addition, a filter index indicating one of the four Sobel filters for gradient computation is also signaled, as illustrated in FIG. 12 . In the example of FIG. 12 , the circle in the center of each filter indicates the location of the chroma sample corresponding to the six luma samples.
[0157] DBV (Direct Block Vector) mode is used for chroma blocks in a dual tree slice. When dual tree is enabled, a flag is displayed indicating whether the chroma block is encoded using IBC (Intra Block Copy) mode. When DBV mode is enabled, the chroma positions and the positions of candidate luma blocks are as shown in the example of Fig. 13. In the example of Fig. 13, the candidate luma blocks are located at the center, top left, top right, bottom left, and bottom right in the corresponding luma region of the chroma block. When one of the five candidate luma blocks corresponding to the chroma block encoded in DBV mode is encoded in IBC mode or intraTMP mode, the block vector bvC (block vector Chroma) of the chroma block is derived using the block vector bvL (block vector Luma) of the corresponding block. The block vector is resized and used as the block vector of the chroma block. When template matching is used for block vector scaling and the luma block is encoded with RR-IBC (Reconstruction Reordered IBC), horizontal or vertical BV (Block Vector) adjustment is performed for bvL in the same manner as RR-IBC.
[0158] As shown in the example in Fig. 14, the DBV mode uses the current chroma block (xCb, yCb) and the position of bvC to determine the corresponding offset position (xCb+bvC[0], yCb+bvC[1]) and predicts the chroma block.
[0159] Compared to natural video, where linear filters operate effectively, screen content coding uses discrete models to improve encoding efficiency. The Cross-Component Discrete Mapping Model (CCDMM) derives a discrete mapping that memorizes pairs of luma and chroma values from reconstructed surrounding samples and maps them according to the memorized pairs, unlike the CCLM mode that uses a continuous linear model. The CCDMM mode predicts chroma samples from co-located luma samples using the memorized pairs. In a pair of luma and chroma values, the luma value corresponds to the input, and the chroma value corresponds to the output. Hereinafter, the set of pairs of luma and chroma values is defined as the mapping set of the discrete mapping model, or the discrete mapping set.
[0160] The CCDMM mode derives a mapping function between luma and chroma using the decoded region. Compared to the continuous model of the CCLM or CCCM mode, the discrete mapping model of the CCDMM mode directly memorizes luma chroma pairs without a continuous function. In the CCDMM mode, to predict a chroma sample in the current chroma block, the corresponding luma value Y is used. c The chroma prediction value is derived according to the discrete mapping model by referring to Y in the input of the mapping set of the discrete mapping model. c If is not found, the CCDMM mode is {Y c+1 , Y c-1 , Y c+2 , Y c-2} to check the luma chroma pair in the order of c = 3, i.e. Y c=Y3 is applied. If none of the five values match the inputs of the discrete mapping set, the CCDMM mode predicts the chroma sample by calculating the average value of adjacent reconstructed chroma samples.
[0161] As an example, five candidate downsampling filters are introduced as in the example of Fig. 15 to generate optimal co-located luma sample values for the YCbCr 4:2:0 color format. In the example of Fig. 15, circles represent luma samples, and triangles (Y1 to Y5) represent co-located luma sample values according to the application of the candidate downsampling filters. An image encoding apparatus selects an index of an optimal filter by generating co-located luma sample values using the candidate downsampling filters. In a manner similar to GLM, the image encoding apparatus signals the selected index to an image decoding apparatus.
[0162] The following embodiments are described with a focus on an image decoding device, but can be implemented identically or similarly in an image encoding device.
[0163] Hereinafter, in the technology related to the process after template matching, the matching block (a reference block selected based on template matching) and the reference block can be used interchangeably. Therefore, the reference template can represent the template of the matching block. Furthermore, in the technology related to the application of the IBC mode below, the matching block (a reference block selected based on a block vector) and the reference block can be used interchangeably.
[0164] III. Embodiments according to the present disclosure
[0165] FIG. 16 is an exemplary diagram illustrating nonlinear modeling-based template matching prediction according to one embodiment of the present disclosure.
[0166] As shown in the example of Fig. 16, the video decoding device according to the present disclosure derives a nonlinear model using a current template and a reference template, and filters a reference block using the derived model, thereby generating a final current block. Hereinafter, a template matching technique based on a discrete mapping model (Embodiment 1), a template matching technique based on a nonlinear filter (Embodiment 2), a cross-component discrete mapping model based on a template matching technique (Embodiment 3), and a cross-component nonlinear model based on a template matching technique (Embodiment 4) are described in order. The present disclosure can be applied only to special image compression such as screen content. In terms of not being a linear filter, the discrete mapping model can be included in a nonlinear filter.
[0167] (Example 1) Template matching technique based on discrete mapping model
[0168] In this implementation example, a template matching technique utilizing a discrete mapping model is described. An image decoding device derives a discrete mapping model between the template of the current block and the template of the reference block, and applies the derived model to the reference block to generate the current block.
[0169] In order to derive a discrete mapping model, a video decoding device directly stores luma pairs between a reference template and a current template without calculating a continuous function such as a linear function. That is, a discrete mapping model is implemented as a nonlinear filter. The video decoding device according to the present disclosure derives a discrete mapping model by storing luma pairs representing luma signal relationships between luma signals of a template of a current block and a template of a reference block in a decoded area, instead of storing a mapping relationship between a luma signal and a chroma signal at the same location according to a derivation technique of a discrete mapping model of a conventional CCDMM. In a luma pair, the luma value of the reference template / reference block corresponds to an input, and the luma value of the current template / current block corresponds to an output. Hereinafter, a set including luma pairs is defined as a mapping set of a discrete mapping model, or a discrete mapping set.
[0170] The video decoding device derives a prediction value from the luma value of a reference block based on a pre-derived discrete mapping model to predict a sample of the current block. When deriving the prediction value, the video decoding device may select one of the following methods.
[0171] As an example, an image decoding device applies the derived discrete mapping model to samples of a reference block to predict samples of a current block. For example, the image decoding device may, based on the discrete mapping model, derive a luma value Y of the reference block at the same location as the current block. c Derive the predicted value of the current block from the input of the discrete mapping set Y c If is not found, the image decoding device uses Y, which is the value without applying discrete mapping. c You can use Y as the predicted value or cThe most similar value is used as the predicted value. The image decoding device uses an arbitrary value (e.g., a median value according to bit depth) as the predicted value or as the input of the discrete mapping set. Alternatively, the image decoding device uses a luma sample with a value for which mapping can be applied among the closest reconstructed luma samples as the input of the discrete mapping set.
[0172] As another example, the five samples {Y c , Y c+1 , Y c-1 , Y c+2 , Y c-2} is not found in the input of the discrete mapping set, the image decoding device generates the mean, median, or predefined weighted sum of adjacent luma samples as the predicted value. Alternatively, the image decoding device can use the mean, median, or predefined weighted sum of adjacent luma samples as the input of the discrete mapping set. Alternatively, the image decoding device can generate the predefined value as the predicted value. In the example of Fig. 17, c = 3, i.e., Y c =Y3 is applied. The predefined weighted sum represents a weighted sum in which the number of samples that are the target of the weighted sum and the corresponding weights are defined in advance.
[0173] As another example, Y c If Y is not found in the input of the discrete mapping set, the image decoding device c A predicted value is generated by interpolating a predetermined number of adjacent luma samples.
[0174] Implementation Example 1 relates to a method for establishing a discrete mapping model between the luma signals of the current block and the luma signals of the reference block during the template matching process. Furthermore, Implementation Example 1 can also be applied between the chroma signals of the current block and the chroma signals of the reference block.
[0175] Embodiment 1 relates to a method for establishing a discrete mapping model between luma signals of a current block and luma signals of a reference block, and between chroma signals of the current block and chroma signals of the reference block, in a template matching process. When Embodiment 1 is applied to the IBC mode of a single tree, a reference block may be obtained according to an existing block vector, rather than being obtained in the template matching process. In addition, when Embodiment 1 is applied to the IBC mode of a dual tree, a reference block of a chroma component may be obtained according to a chroma block vector derived from a DBV (Direct Block Vector) mode. Thereafter, the video decoding device derives a luma signal relationship (or a chroma signal relationship if it is also applied to a chroma component) using the template of the current template and the template of the reference block.
[0176] (Example 2) Template matching technique using nonlinear filters
[0177] In this implementation example, a template matching technique utilizing a nonlinear filter is described. An image decoding device derives a nonlinear model between a template of a current block and a template of a reference block, and applies the derived model to the reference block to predict the current block. At this time, instead of deriving a mapping relationship between a luma signal and a chroma signal at the same location as a nonlinear function according to a derivation technique of a conventional nonlinear model such as GLM or CCCM, the image decoding device according to the present disclosure derives and stores a nonlinear model representing the relationship between luma signals of the template of the current block and the template of the reference block in a decoded area.
[0178] In addition to the derivation method used in existing GLM, CCCM, etc., the image decoding device implements a nonlinear filter using one of the following representation methods, a part of one representation method, or a combination of representation methods.
[0179] To predict a luma sample, a nonlinear filter according to the present embodiment can be implemented as in Equation 10.
[0180]
[0181] In mathematical expression 10, Y c represents the luma sample of the current template or current block, and Y r represents a luma sample of a reference block or reference template. (x, y) represents the position of the current sample, and (x c , y c ) represents the location of the reference sample corresponding to (x, y). (x0, y0) is (x c , y c ) represents the filter offset around S i Indicates the filtering target region for reference samples. The filter offset can be adjusted using the block vector used in the co-located luma block or the DBV for the chroma block.
[0182] c i (x0, y0) represents the filter coefficients. The shape of the filter can be of various shapes and sizes. As an example, a filter having the same shape as the ALF filter illustrated in FIG. 18 can be used. As another example, a filter having the same shape as illustrated in FIG. 8 can be used. As another example, in addition to the nonlinear terms illustrated in Equation 10, the nonlinear filter can include linear terms as illustrated in Equation 7.
[0183] A nonlinear filter generates nonlinear modeling values of luma samples within a filtering target region using nonlinear modeling (NM). The nonlinear filter can then generate luma sample values based on the product of the nonlinear modeling values and nonlinear filter coefficients.
[0184] At this time, one of the following methods can be used for nonlinear modeling.
[0185] As an example, a nonlinear model can be implemented using a polynomial function, as in Equation 11.
[0186]
[0187] As another example, the Michaelis-Menten hyperbolic model can be used as in Equation 12.
[0188]
[0189] As another example, a model based on Fourier transform, such as Equation 13, can be used.
[0190]
[0191] As another example, an exponential function model can be used, as in Equation 14.
[0192]
[0193] In mathematical expressions 11 to 14, Y r represents a luma sample of a reference block or reference template. (x c , y c ) represents the position of the reference sample corresponding to the position (x, y) of the current sample. (x0, y0) is (x c , y c ) represents the filter offset around α. Also, α j are parameters of the nonlinear model and may be preset values according to the nonlinear model. For example, if the nonlinear model is a polynomial function as in Equation 6, α2 is 1 and α0 is 512.
[0194] As an example, the image decoding device Y in the proposed nonlinear modeling c Wow Y rAfter setting the current template and the reference template respectively, the filter coefficients are calculated so that the MSE between the predicted value according to Equation 10 and the restored value of the current template is minimized, for example. After deriving the coefficients of the nonlinear filter, the image decoding device predicts the current block based on the derived nonlinear filter. Reference block Y r Current block Y in c By referring to the luma value at the same location, the image decoding device obtains the final predicted Y based on the nonlinear filter. c is calculated.
[0195] Implementation Example 2 relates to a method for establishing a nonlinear relationship between the luma signals of the current block and the luma signals of the reference block during the template matching process. Furthermore, Implementation Example 2 can also be applied between the chroma signals of the current block and the chroma signals of the reference block.
[0196] Implementation Example 2 relates to a method for establishing a nonlinear relationship between luma signals of a current block and luma signals of a reference block, and between chroma signals of the current block and chroma signals of a reference block, using template matching in the process of obtaining a reference block. When Implementation Example 2 is applied to the IBC mode of a single tree, in addition to using template matching, a reference block can be obtained according to an existing block vector. When Implementation Example 2 is applied to the IBC mode of a dual tree, a reference block of a chroma component can be obtained according to a chroma block vector derived from the DBV mode. In addition, in Implementation Example 2, reference signals can be obtained in an area where adjacent samples that have been decoded in advance are located.
[0197] (Example 3) Template matching-based cross-component discrete mapping model
[0198] As described above, the conventional CCDMM mode derives a discrete mapping model between luma and chroma in the restored surrounding area where the linear model is applied, and predicts chroma samples from luma samples in the current block using the derived discrete mapping model.
[0199] FIG. 19 is an exemplary diagram illustrating nonlinear modeling-based template matching prediction according to another embodiment of the present disclosure.
[0200] In this implementation example, a cross-component discrete mapping model based on template matching is described. As illustrated in FIG. 19, when deriving the discrete mapping model, the image decoding device according to the present disclosure derives and stores the mapping relationship between luma and chroma in a reference block based on template matching. In the example of FIG. 19, I L represents luma signals, and I C represents chroma signals. Omitting the calculation of the continuous function, the discrete mapping model directly remembers the luma chroma pair of the reference block according to template matching. The video decoding device derives a chroma sample by referring to the luma value at the position where the current block is downsampled based on the discrete mapping model. That is, the cross-component discrete mapping model derived from the reference block is applied between the luma signals and chroma signals of the current block. The luma value downsampled from the luma chroma pair corresponds to the input, and the chroma value corresponds to the output. Hereinafter, a set including luma chroma pairs is defined as a mapping set of the discrete mapping model, or a discrete mapping set.
[0201] Luma value Y at the location corresponding to the chroma sample of the current block in the input of the mapping set of the discrete mapping model c If is not found, the image decoding device may be configured to {Y c+1 , Y c-1 , Y c+2 , Y c-2} are checked in a given arbitrary order. As mentioned above, in the example of Fig. 17, c = 3, i.e. Y c =Y3 is applied. In the example of Fig. 17, the samples represent luma values at the locations where the current block is downsampled. If all five values are not found in the input of the discrete mapping set, the video decoding device sets the predicted value of the current chroma sample to a value (e.g., a mean value, a median value, etc.) calculated based on adjacent restored chroma samples.
[0202] Implementation Example 3 relates to a method for establishing a discrete mapping model between luma signals and chroma signals of a current block, and uses a reference block based on template matching as reference information to obtain information between luma signals and chroma signals. When Implementation Example 3 is applied to the IBC mode of a single tree, the reference block can be obtained according to an existing block vector. In addition, when Implementation Example 3 is applied to the IBC mode of a dual tree, the reference block of a chroma component can be obtained according to a chroma block vector derived from the DBV mode. In addition, in Implementation Example 3, reference information can be obtained in an area where adjacent samples that have been decoded in advance are located.
[0203] (Example 4) Template matching-based cross-component nonlinear model
[0204] In this implementation example, a cross-component nonlinear model based on template matching is derived. Instead of a cross-component discrete mapping model based on template matching, the image decoding device derives a cross-component nonlinear model based on template matching. At this time, the methods for calculating the nonlinear model described above can be utilized. The image decoding device implements the cross-component nonlinear model using one of the following methods.
[0205] As an example, an image decoding device derives a nonlinear function by applying a nonlinear model derivation technique used in conventional GLM, CCCM, etc. to a reference block. In other words, the image decoding device derives and stores a nonlinear function representing the relationship between luma signals and chroma signals of the reference block.
[0206] Alternatively, in addition to the derivation method used in existing GLM, CCCM, etc., a nonlinear filter is implemented using one of the following representation methods, a part of one representation method, or a combination of representation methods.
[0207] To predict chroma samples, a nonlinear filter according to the present embodiment can be implemented as in mathematical expression 15.
[0208]
[0209] In Equation 15, R Y represents the restored luma sample, and R i (i∈{Cb, Cr}) represents a predicted chroma sample. (x, y) represents the location of the chroma sample, and (x c , y c ) represents the location of the luma sample corresponding to (x, y). (x0, y0) is (x c , y c ) represents the filter offset around S i represents the filtering target area for reference samples. c i (x0, y0) represents the filter coefficient.
[0210] c i(x0, y0) represents the filter coefficients. The shape of the filter can be of various shapes and sizes. As an example, a filter having the same shape as the ALF filter illustrated in FIG. 18 may be used. In the example of FIG. 18, the C2 sample represents a luma sample at the same location as the chroma sample to be filtered. As another example, a filter having the same shape as illustrated in FIG. 8 may be used. As another example, in addition to the nonlinear terms illustrated in Equation 10, the nonlinear filter may include linear terms as illustrated in Equation 7.
[0211] A nonlinear filter generates nonlinear modeling values for luma samples within a filtering target region using nonlinear modeling. The nonlinear filter can then generate chroma sample values based on the product of the nonlinear modeling values and nonlinear filter coefficients.
[0212] At this time, one of the following methods can be used for nonlinear modeling.
[0213] As an example, a nonlinear model can be implemented using a polynomial function, as in Equation 16.
[0214]
[0215] As another example, the Michaelis-Menten hyperbolic model can be used as in Equation 17.
[0216]
[0217] As another example, a model based on Fourier transform, such as Equation 18, can be used.
[0218]
[0219] As another example, an exponential function model can be used, as in Equation 19.
[0220]
[0221] In mathematical expressions 16 to 19, R Yrepresents the restored luma sample. (x c , y c ) represents the position of the luma sample corresponding to the position (x, y) of the chroma sample. (x0, y0) is (x c , y c ) represents the filter offset around α. Also, α j are parameters of the nonlinear model, and may be, for example, preset values according to the nonlinear model. For example, if the nonlinear model is a polynomial function such as Equation 6, α2 is 1 and α0 is 512.
[0222] As an example, the image decoding device uses R in the proposed nonlinear modeling. Y WowR i After setting the luma sample and chroma prediction sample of the template matching-based reference block respectively, the filter coefficients are calculated so that the MSE between the prediction value according to Equation 15 and the chroma sample of the reference block is minimized, for example. After deriving the coefficients of the nonlinear filter, the image decoding device predicts the current chroma block based on the derived nonlinear filter. With reference to the restored luma sample values of the current block, the image decoding device predicts the final predicted R based on the nonlinear filter. i is calculated.
[0223] Implementation Example 4 relates to a method for establishing a nonlinear relationship between luma signals and chroma signals of a current block, and uses a reference block based on template matching as reference information to obtain information between luma signals and chroma signals. When Implementation Example 4 is applied to the IBC mode of a single tree, the reference block can be obtained according to an existing block vector. In addition, when Implementation Example 4 is applied to the IBC mode of a dual tree, the reference block of a chroma component can be obtained according to a chroma block vector derived from the DBV mode. In addition, in Implementation Example 4, reference information can be obtained in an area where adjacent samples that have been decoded in advance are located.
[0224] Meanwhile, when various types of filters are used in the template matching technique according to Realization Examples 1 to 4, the image decoding device may use one of the following methods.
[0225] For example, a nonlinear filter may be applied after applying a linear model-based filter. For example, by applying a nonlinear model F(·) after applying a linear model-based filter, the image decoding device calculates the current block as in Equation 20.
[0226]
[0227] In equation 20, L i curr represents the i-th luma sample of the current block, and L i ref represents the reference sample. α i represents the ith coefficient of the linear model, and β represents the offset. N represents the number of luma samples involved in linear filtering.
[0228] As another example, a nonlinear model F(·) can be applied after applying a model G(·) based on the luma sample gradient, as in Equation 21.
[0229]
[0230] In Equation 21, α and β represent parameters implementing the GLM. The gradient G can be estimated using one of a preset number of Sobel-based gradient patterns.
[0231] Hereinafter, a method for predicting a current block based on template matching is described using the cities of FIGS. 20 and 21.
[0232] FIG. 20 is a flowchart illustrating a method of encoding a current block performed by an image encoding device according to one embodiment of the present disclosure.
[0233] The video encoding device generates a first prediction block of the current block using a prediction mode that does not utilize template matching (S2000). For example, an intra prediction mode or an inter prediction mode other than template matching may be used.
[0234] The video encoding device obtains the size of the current block and determines whether the size of the current block satisfies a range condition (e.g., a minimum block size condition and / or a maximum block size condition) for encoding according to template matching-based prediction. If the range condition is satisfied, the video encoding device performs template matching-based prediction.
[0235] The video encoding device generates a second prediction block of the current block based on template matching (S2002).
[0236] An image encoding device predicts a current block using template matching according to a preset order (e.g., z-scan order). The image encoding device searches for a reference block based on template matching between templates of blocks within a search region and the template of the current block, and determines a second prediction block of the current block using the searched reference block.
[0237] The image encoding device determines the shape of the filter (S2004).
[0238] The shape of the filter can be a linear model, a discrete mapping model, or a nonlinear filter. Alternatively, the shape of the filter can be a composite of a linear model and a nonlinear filter. In terms of rate-distortion optimization, an image encoding device can determine the shape of the filter. The image encoding device encodes information indicating the shape of the filter.
[0239] The image encoding device calculates the coefficients of the filter (S2006).
[0240] As an example, the video encoding device derives filter coefficients of a linear model using a regression model that minimizes the MSE between samples in the template of the reference block and samples in the template of the current block. As another example, the video encoding device calculates a discrete mapping model or a nonlinear filter between the template of the reference block and the template of the current block. As another example, the video encoding device calculates a cross-component discrete mapping model or a cross-component nonlinear filter using luma signals and chroma signals of the reference block.
[0241] The video encoding device applies filtering to the second prediction block based on the filter to generate a filtered second prediction block (S2008).
[0242] As an example, an image encoding device generates a filtered second prediction block by applying a linear model to a reference block. As another example, an image encoding device generates a filtered second prediction block from a reference block by using a discrete mapping model or a nonlinear filter between templates.
[0243] As another example, a video encoding device predicts chroma signals of a second prediction block from luma signals of the second prediction block (or the filtered second prediction block) using a cross-component discrete mapping model or a cross-component nonlinear filter. At this time, it is assumed that the luma signals of the prediction block (or the filtered prediction block) are restored.
[0244] The video encoding device determines a template matching flag based on the first prediction block, the second prediction block, and the filtered second prediction block (S2010). Here, the template matching flag indicates whether template matching is used for the current block.
[0245] In terms of rate distortion optimization, the video encoding device determines a template matching flag. For example, if the first prediction block is optimal, the template matching flag is set to false. Conversely, if the second prediction block or the filtered second prediction block is optimal, the template matching flag is set to true.
[0246] The video encoding device encodes a template matching flag (S2012).
[0247] The video encoding device checks the template matching flag (S2014).
[0248] If the template matching flag is true (Yes in S2014), the video encoding device performs the following steps.
[0249] The video encoding device determines a filter flag based on the second prediction block and the filtered second prediction block (S2016). Here, the filter flag indicates whether filtering is applied to the prediction block based on template matching.
[0250] In terms of rate distortion optimization, the video encoding device determines a filter flag. For example, if the second prediction block is optimal, the filter flag is determined to be false. Conversely, if the filtered second prediction block is optimal, the filter flag is determined to be true.
[0251] The video encoding device encodes a filter flag (S2018).
[0252] Additionally, the image encoding device encodes information indicating the shape of the filter.
[0253] If the template matching flag is false (No in S2014), the image encoding device encodes information related to the first prediction block (S2030).
[0254] Thereafter, the video encoding device generates a residual block by subtracting the predicted block according to the template matching flag and the filter flag from the current block. The video encoding device applies transformation / quantization to the residual block to generate quantized transform coefficients, and then encodes the quantized transform coefficients.
[0255] In the example of Fig. 20, the video encoding device utilizes template matching in the process of obtaining a reference block. As another example, when the IBC mode is applied in a single tree, the video encoding device can obtain a reference block according to an existing block vector. When the IBC mode is applied in a dual tree, the video encoding device can obtain a reference block of a chroma component according to a chroma block vector derived from the DBV mode. In this case, the template matching flag can be replaced with a flag indicating the application of the IBC mode.
[0256] FIG. 21 is a flowchart illustrating a method of restoring a current block performed by an image decoding device according to one embodiment of the present disclosure.
[0257] The video decoding device obtains the size of the current block and checks whether the size of the current block satisfies a range condition (e.g., a minimum block size condition and / or a maximum block size condition) for encoding according to template matching-based prediction.
[0258] If the range condition is satisfied, the video decoding device decodes the template matching flag from the bitstream (S2100). The template matching flag indicates whether template matching is used for the current block.
[0259] The video decoding device checks the template matching flag (S2102).
[0260] If the template matching flag is true (Yes in S2102), the video decoding device performs the following steps.
[0261] The video decoding device generates a prediction block of the current block based on template matching (S2104).
[0262] The image decoding device predicts the current block using template matching according to a preset order (e.g., z-scan order). The image decoding device searches for a reference block based on template matching between templates of blocks in a search area and the template of the current block, and determines the prediction block of the current block as the searched reference block.
[0263] The video decoding device decodes a filter flag from the bitstream (S2106). Here, the filter flag indicates whether filtering is applied to a prediction block based on template matching.
[0264] The video decoding device checks the filter flag (S2108).
[0265] If the filter flag is true (Yes in S2108), the video decoding device performs the following steps.
[0266] The video decoding device obtains the shape of the filter of the current block (S2110).
[0267] The shape of the filter may be a linear model, a discrete mapping model, or a nonlinear filter. Alternatively, the shape of the filter may be a composite of a linear model and a nonlinear filter. As an example, to obtain the shape of the filter, an image decoding device may decode information indicating the shape of the filter from a bitstream.
[0268] The image decoding device calculates the coefficients of the filter (S2112).
[0269] For example, the video decoding device derives filter coefficients of a linear model using a regression model that minimizes the MSE between samples in the template of the reference block and samples in the template of the current block. As another example, the video decoding device calculates a discrete mapping model or a nonlinear filter between the template of the reference block and the template of the current block. As another example, the video decoding device calculates a cross-component discrete mapping model or a cross-component nonlinear filter using luma signals and chroma signals of the reference block.
[0270] The video decoding device applies filtering to the prediction block using a filter (S2114).
[0271] As an example, an image decoding device generates a filtered prediction block by applying a linear model to a reference block. As another example, an image decoding device generates a filtered prediction block from a reference block by using a discrete mapping model or a nonlinear filter between templates.
[0272] As another example, a video decoding device predicts chroma signals of a prediction block from luma signals of the prediction block (or filtered prediction block) using a cross-component discrete mapping model or a cross-component nonlinear filter. At this time, it is assumed that the luma signals of the prediction block (or filtered prediction block) are pre-restored.
[0273] If the filter flag is false (No in S2108), the video decoding device does not perform filtering of the prediction block according to template matching.
[0274] If the range condition is not satisfied or the template matching flag is false (No in S2102), the video decoding device generates a prediction block of the current block using a prediction mode other than template matching (S2130). For example, intra prediction mode or inter prediction may be used instead of template matching.
[0275] Afterwards, the image decoding device decodes the quantized transform coefficients and then applies inverse quantization / inverse transformation to the quantized transform coefficients to generate a residual block. The image decoding device can add the residual block and the prediction block to generate a restored block of the current block.
[0276] In the example of Fig. 21, the video decoding device utilizes template matching in the process of obtaining a reference block. As another example, when the IBC mode is applied in a single tree, the video decoding device can obtain the reference block according to an existing block vector. When the IBC mode is applied in a dual tree, the video decoding device can obtain the reference block of the chroma component according to the chroma block vector derived from the DBV mode. In this case, the template matching flag can be replaced with a flag indicating the application of the IBC mode.
[0277] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.
[0278] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.
[0279] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).
[0280] The above description is merely an example of the technical idea of the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.
[0281] (Explanation of symbols)
[0282] 122: Intra prediction unit
[0283] 155: Entropy decryption unit
[0284] 510: Entropy decryption unit
[0285] 542: Intra prediction unit
[0286]
[0287] CROSS-REFERENCE TO RELATED APPLICATION
[0288] This patent application claims priority to Korean patent application No. 10-2023-0065621, filed in Korea on May 22, 2023, and Korean patent application No. 10-2024-0045374, filed in Korea on April 3, 2024, the entire contents of which are incorporated herein by reference.
Claims
1. A method for restoring a current block performed by a video decoding device, A step of searching for a reference block based on template matching between templates of blocks in a search area and templates of the current block or a block vector of the current block, and determining a prediction block of the current block using the searched reference block; A step of obtaining the form of a filter of the current block, wherein the form of the filter is a cross-component discrete mapping model or a cross-component nonlinear filter; A step of calculating coefficients of the filter using the above reference block; and A step of applying filtering to the prediction block using the above filter. A method, characterized by including:
2. In paragraph 1, A step of decoding a template matching flag from a bitstream, wherein the template matching flag indicates whether the template matching is used for the current block; and Step for checking the above template matching flag Including, A method characterized in that a step of determining a prediction block of the current block is performed when the above template matching flag is true.
3. In paragraph 1, A step of decoding a flag indicating whether to apply IBC (Intra Block Copy) mode; and Steps to check the above flags Including, A method characterized in that, if the above flag is true, a step of determining a prediction block of the current block is performed.
4. In paragraph 3, The step of determining the predicted block of the current block is: A method characterized in that, when the IBC mode is applied in a single tree, luma samples and chroma samples of the reference block are determined based on the block vector of the current block.
5. In paragraph 3, The step of determining the predicted block of the current block is: A method characterized in that, when the IBC mode is applied in a dual tree, the chroma samples of the reference block are determined based on a chroma block vector derived from a DBV (Direct Block Vector) mode.
6. In paragraph 1, A step of decoding a filter flag from a bitstream, wherein the filter flag indicates whether to apply the filtering to the prediction block; and Step to check the above filter flag Including, A method characterized by performing a step of obtaining the shape of the filter when the above filter flag is true.
7. In paragraph 1, The step of calculating the coefficients of the above filter is: a step of inputting luma samples of the above reference block into the cross-component nonlinear filter; and A step of calculating coefficients of the cross-component nonlinear filter so that the difference between the output of the cross-component nonlinear filter and the corresponding chroma sample in the reference block is minimized. A method, characterized by including:
8. In paragraph 7, The steps for applying the above filtering are: A method characterized in that the coefficients of the cross-component nonlinear filter are applied to luma samples around a position corresponding to a chroma sample of the current block in the luma block of the current block to generate a filtered prediction sample of the chroma sample.
9. In paragraph 1, The step of calculating the coefficients of the above filter is: A step of deriving luma chroma pairs representing the relationship between luma samples and chroma samples of the above reference block; and A step of storing the above luma chroma pairs in the mapping set of the above cross-component discrete mapping model. A method, characterized by including:
10. In paragraph 9, The steps for applying the above filtering are: A method characterized in that the filtered chroma prediction sample of the current block is generated from a downsampled luma sample of the current block based on the luma chroma pairs of the cross-component discrete mapping model.
11. In paragraph 9, The steps for applying the above filtering are: A method characterized in that, if the value of a luma sample at a position corresponding to a chroma sample of the current block and the values of surrounding samples of the luma sample are not found in the input of the mapping set, a value calculated based on restored chroma samples adjacent to the chroma sample is generated as a filtered chroma prediction sample of the chroma sample.
12. A method for encoding a current block performed by a video encoding device, A step of generating a first prediction block of the current block using a prediction mode other than template matching and IBC (Intra Block Copy) mode; A step of searching for a reference block based on a template matching between templates of blocks in a search area and a template of the current block or a block vector of the current block, and determining a second prediction block of the current block using the searched reference block; A step of determining the form of the filter of the current block, wherein the form of the filter is a cross-component discrete mapping model or a cross-component nonlinear filter; A step of calculating coefficients of the filter using the above reference block; and A step of applying the filtering to the second prediction block using the filter to generate a filtered second prediction block. A method, characterized by including:
13. In paragraph 12, A step of determining a template matching flag based on the first prediction block, the second prediction block, and the filtered second prediction block, wherein the template matching flag indicates whether to use the template matching for the current block; a step of encoding the above template matching flag; and Step for checking the above template matching flag A method characterized by further comprising:
14. In paragraph 13, If the above template matching flag is true, A step of determining a filter flag based on the second prediction block and the filtered second prediction block, wherein the filter flag indicates whether to apply filtering to the prediction block; a step of encoding the above filter flag; and A step of encoding information indicating the form of the above filter. A method, characterized in that it further comprises:
15. In paragraph 12, A step of determining a flag indicating whether to apply the IBC mode based on the first prediction block, the second prediction block, and the filtered second prediction block; and A step of encoding the above flag; A method characterized by further comprising:
16. A method for providing video data to a video decoding device, A step of encoding the above video data into a bitstream; and A step of transmitting the above bitstream to the image decoding device Including, The step of encoding the above video data is: A step of generating a first prediction block of a current block using a prediction mode other than template matching and IBC (Intra Block Copy) mode; A step of searching for a reference block based on a template matching between templates of blocks in a search area and a template of the current block or a block vector of the current block, and determining a second prediction block of the current block using the searched reference block; A step of determining the form of the filter of the current block, wherein the form of the filter is a cross-component discrete mapping model or a cross-component nonlinear filter; A step of calculating coefficients of the filter using the above reference block; and A step of applying the filtering to the second prediction block using the filter to generate a filtered second prediction block. A method, characterized by including:
Citation Information
Patent Citations
Methods and devices for encoding and decoding transform domain filters
KR101506372B1
Method and apparatus of video decoder with extended intra block copy
KR1020170122350A
Book holder for bookshelf
KR1020230031613A
Boat Fastening Pole
KR102345102B1
Data transmission apparatus in semiconductor system for reducing current consumption
KR102589534B1
Cited By
Video encoding method and apparatus, video decoding method and apparatus, computer-readable medium, and electronic device
WO2026179774A1
