Method and device for video coding using matrix-based intra prediction

Abstract

The present embodiment provides a video coding method and device using matrix-based intra prediction. In the present embodiment, an image decoding device obtains a weight matrix for applying matrix-based intra prediction (MIP) to a current block, and generates a prediction block of the current block by applying the weight matrix to reference pixels of the current block. The image decoding device derives a virtual intra prediction mode on the basis of adjacent / non-adjacent samples of the current block, and selects a separable / non-separable inverse transform kernel of the current block on the basis of the virtual intra prediction mode. The image decoding device reconstructs transform coefficients of the current block from a bitstream, and generates a residual block of the current block by applying the separable / non-separable inverse transform kernel of the current block to the transform coefficients.

Description

Method and device for video coding using matrix-based intra prediction

[0001] The present disclosure relates to a video coding method and device using matrix-based intra prediction.

[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.

[0003] Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.

[0004] Therefore, when storing or transmitting video data, the encoder compresses the video data and stores or transmits it, and the decoder receives the compressed video data, decompresses it, and plays it back. These video compression technologies include H.264 / AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves encoding efficiency by about 30% compared to HEVC.

[0005] However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.

[0006] In particular, the demand for high-resolution, high-quality images, such as HD (High Definition) images and UHD (Ultra High Definition) images, has been increasing in various fields. As image data becomes higher in resolution and quality, the amount of information or bits transmitted increases relatively compared to existing image data. This increase in the amount of information or bits transmitted leads to an increase in transmission and storage costs. In addition, interest in and demand for immersive media, such as VR (Virtual Reality), AR (Artificial Reality) content, and holograms, has been increasing recently, and the broadcasting of videos / images with different image characteristics from real images, such as game images, is increasing. Accordingly, a highly efficient image compression technology is required to effectively compress, transmit, store, and play information of high-resolution, high-quality images with various characteristics as described above.

[0007] The present disclosure aims to provide a video coding method and apparatus for performing intra prediction using matrix-based intra prediction (MIP) having multiple candidates.

[0008] One embodiment of the present disclosure aims to provide a video coding method and apparatus for performing intra prediction using MIP having a plurality of weight matrices.

[0009] One embodiment of the present disclosure aims to provide a video coding method and device for selecting a separable / non-separable transform kernel of MIP.

[0010] One embodiment of the present disclosure provides a method for encoding / decoding an image, which is performed by an image encoding / decoding device. One embodiment of the present disclosure provides a device for encoding / decoding an image.

[0011] One embodiment of the present disclosure provides a computer-readable digital storage medium storing encoded image information generated by an image encoding method disclosed in at least one of the embodiments of the present disclosure. One embodiment of the present disclosure provides a computer-readable digital storage medium storing encoded information or encoded image information that causes an image decoding device to perform an image decoding method disclosed in at least one of the embodiments of the present disclosure.

[0012] In addition, one embodiment according to the present disclosure provides a method for transmitting image data including a bitstream generated based on an image encoding method disclosed in at least one of the embodiments according to the present disclosure, or a transmission device for transmitting image data.

[0013] According to an embodiment of the present disclosure, a method for restoring a current block, performed by an image decoding device, is provided, comprising: obtaining a weight matrix for applying matrix-based intra prediction (MIP) to the current block; applying the weight matrix to reference pixels of the current block to generate a prediction block of the current block; deriving a virtual intra prediction mode based on adjacent / non-adjacent samples of the current block; selecting a separable / non-separable inverse transform kernel of the current block based on the virtual intra prediction mode; restoring transform coefficients of the current block from a bitstream; and applying the separable / non-separable inverse transform kernel of the current block to the transform coefficients to generate a residual block of the current block.

[0014] According to another embodiment of the present disclosure, a method for encoding a current block, performed by a video encoding apparatus, is provided, comprising: obtaining a weight matrix for applying matrix-based intra prediction (MIP) to the current block; applying the weight matrix to reference pixels of the current block to generate a prediction block of the current block; deriving a virtual intra prediction mode based on adjacent / non-adjacent samples of the current block; selecting a separable / non-separable transform kernel of the current block based on the virtual intra prediction mode; generating a residual block of the current block by subtracting the prediction block from the current block; and applying the separable / non-separable transform kernel of the current block to the residual block to generate transform coefficients.

[0015] According to another embodiment of the present disclosure, a method for providing video data to a video decoding device is provided, comprising: encoding the video data into a bitstream; and transmitting the bitstream to the video decoding device, wherein the encoding the video data comprises: obtaining a weight matrix for applying Matrix-based Intra Prediction (MIP) to a current block; applying the weight matrix to reference pixels of the current block to generate a prediction block of the current block; deriving a virtual intra prediction mode based on adjacent / non-adjacent samples of the current block; selecting a separable / non-separable transform kernel of the current block based on the virtual intra prediction mode; generating a residual block of the current block by subtracting the prediction block from the current block; and applying the separable / non-separable transform kernel of the current block to the residual block to generate transform coefficients.

[0016] As described above, according to the present embodiment, by providing a video coding method and device for performing intra prediction using a MIP having a plurality of candidates, performing intra prediction using a MIP having a plurality of weight matrices, and selecting a separable / non-separable transform kernel of the MIP, it is possible to improve video encoding efficiency and enhance video quality.

[0017] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.

[0018] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.

[0019] FIGS. 3A and 3B are diagrams illustrating multiple intra prediction modes, including wide-angle intra prediction modes.

[0020] Figure 4 is an example diagram of the surrounding blocks of the current block.

[0021] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.

[0022] Figure 6 is an example diagram showing a search area used in IntraTMP (Intra Template Matching Prediction) technology.

[0023] Figure 7 is an example diagram showing a method for deriving one or more intra prediction modes in TIMD (Template-based intra mode derivation) technology.

[0024] Figure 8 is an example diagram showing a method for signaling a combination of a partitioning mode and an intra prediction mode of SGPM (Spatial Geometric Partitioning Mode).

[0025] Figure 9 is an example diagram showing a template of a current block to which SGPM mode is applied.

[0026] Figure 10 is an example diagram showing a reference template of MIP (Matrix-based Intra Prediction).

[0027] FIG. 11 is a flowchart illustrating a method of encoding a current block performed by an image encoding device according to one embodiment of the present disclosure.

[0028] FIG. 12 is a flowchart illustrating a method for restoring a current block performed by an image decoding device according to one embodiment of the present disclosure.

[0029] Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.

[0030] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to the illustration in FIG. 1.

[0031] The video encoding device may be configured to include a picture segmentation unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0032] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0033] A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). And, information commonly referenced by one or more SPS is encoded in a Video Parameter Set (VPS). In addition, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.

[0034] The picture segmentation unit (110) determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as the syntax of SPS or PPS and transmitted to the image decoding device.

[0035] The picture segmentation unit (110) divides each picture constituting an image into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a CU, which is a basic unit of encoding.

[0036] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).

[0037] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.

[0038] As illustrated in FIG. 2, a CTU may first be split into a QT structure. The quadtree splitting may be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further split into one or more of the BT structure or the TT structure. There may be multiple splitting directions in the BT structure and / or the TT structure. For example, there may be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As illustrated in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0039] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the CU split flag (split_cu_flag) value indicates that the node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is a basic unit of encoding. If the CU split flag (split_cu_flag) value indicates that the node is split, the video encoding device starts encoding from the first flag in the above-described manner.

[0040] As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.

[0041] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.

[0042] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0043] In general, each current block within a picture can be predictively coded. Prediction of the current block can typically be performed using either intra-prediction (using data from the picture containing the current block) or inter-prediction (using data from a picture coded before the picture containing the current block). Inter-prediction encompasses both unidirectional and bidirectional prediction.

[0044] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3a, the multiple intra prediction modes may include two non-directional modes including the Planar mode and the DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently depending on each prediction mode.

[0045] For efficient directional prediction for a rectangular current block, directional modes (intra prediction modes 67 to 80 and -1 to -14) indicated by dotted arrows in Fig. 3b may be additionally used. These may be referred to as "wide-angle intra-prediction modes." In Fig. 3b, the arrows point to corresponding reference samples used for prediction, and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide-angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra-prediction modes, some wide-angle intra-prediction modes available for the current block may be determined based on the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (intra prediction modes 67 to 80) having an angle less than 45 degrees are available when the current block is a rectangular shape whose height is smaller than its width, and wide-angle intra prediction modes (intra prediction modes -1 to -14) having an angle greater than -135 degrees are available when the current block is a rectangular shape whose width is larger than its height.

[0046] The intra prediction unit (122) can determine an intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate bit-rate distortion values ​​using rate-distortion analysis for multiple tested intra prediction modes and select an intra prediction mode with the best bit-rate distortion characteristics among the tested modes.

[0047] The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0048] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma component and the chroma component. The motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0049] The inter prediction unit (124) may perform interpolation on a reference picture or a reference block to improve prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision or resolution of the motion vector can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.

[0050] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the entropy encoding unit (155). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.

[0051] Various methods can be used to minimize the number of bits required to encode motion information.

[0052] For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the image decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.

[0053] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

[0054] As the surrounding blocks for deriving merge candidates, all or part of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the block at the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.

[0055] The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, the merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the video decoding device.

[0056] Merge Skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high encoding efficiency can be achieved for low-motion images, still images, and screen content images.

[0057] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0058] Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.

[0059] In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block in the current picture as shown in FIG. 4 may be used. In addition, a block located in a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block co-located with the current block in the reference picture or blocks adjacent to the block in the co-located block may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.

[0060] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.

[0061] The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information to identify the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.

[0062] Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.

[0063] The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.

[0064] The transformation unit (140) transforms residual signals within a residual block having pixel values ​​in a spatial domain into transform coefficients in a frequency domain. The transformation unit (140) may transform the residual signals within the residual block using the entire size of the residual block as a transformation unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transformation units to perform the transformation. Alternatively, the residual signals may be transformed using only the transformation domain sub-block as a transformation unit by dividing the sub-blocks into two sub-blocks, that is, a transformation domain and a non-transform domain. Here, the transformation domain sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only a sub-block has been converted, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. In addition, the size of the conversion area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0065] Meanwhile, the transformation unit (140) can individually perform transformations on the residual block in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation function pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation function pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0066] The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.

[0067] The rearrangement unit (150) can perform rearrangement of coefficient values ​​for quantized residual values.

[0068] The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region by using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, the scanning method to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan.

[0069] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0070] In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about the prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about the intra prediction mode) or inter prediction information (information about the encoding mode of motion information (merge mode or AMVP mode), a merge index in the case of the merge mode, and a reference picture index and a differential motion vector in the case of the AMVP mode) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about quantization parameters and information about a quantization matrix.

[0071] The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0072] An adder (170) adds the restored residual block and the predicted block generated by the prediction unit (120) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting the next block.

[0073] The loop filter unit (180) performs filtering on restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filter unit (180) may include all or part of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186) as an in-loop filter.

[0074] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter (184) and the ALF (186) perform additional filtering on the deblocking-filtered image. The SAO filter (184) and the ALF (186) are filters used to compensate for the differences between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in units of CTUs. In contrast, the ALF (186) performs block-based filtering, and compensates for distortion by applying different filters by distinguishing the edges and degrees of variation of the corresponding block. Information on filter coefficients to be used in the ALF can be encoded and signaled to an image decoding device.

[0075] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0076] The video encoding device can store the bitstream of encoded video data on a non-transitory storage medium or transmit it to the video decoding device using a communication network.

[0077] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.

[0078] The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

[0079] Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0080] The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information, information on residual signals, etc. required to restore the current block.

[0081] The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.

[0082] For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (mtt_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Accordingly, each node below the leaf nodes of QT are split recursively into the BT or TT structure.

[0083] As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.

[0084] As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.

[0085] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded by using the division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.

[0086] Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about residual signals.

[0087] The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.

[0088] The inverse quantization unit (520) inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.

[0089] The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.

[0090] In addition, when the inverse transform unit (530) inversely transforms only a portion of a transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only a sub-block of the transform block has been transformed, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block, and inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to restore residual signals, and fills “0” values ​​with residual signals for areas that have not been inversely transformed, thereby generating a final residual block for the current block.

[0091] In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions using MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.

[0092] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.

[0093] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0094] The inter prediction unit (544) uses the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.

[0095] An adder (550) adds the residual block output from the inverse transform unit (530) and the predicted block output from the inter prediction unit (544) or the intra prediction unit (542) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting a block to be decoded later.

[0096] The loop filter unit (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as in-loop filters. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) and the ALF (566) perform additional filtering on restored blocks after deblocking filtering to compensate for differences between restored pixels and original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about filter coefficients decoded from the non-stream.

[0097] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0098] The present embodiment relates to encoding and decoding of images (videos) as described above. More specifically, a video coding method and device for performing intra prediction using matrix-based intra prediction (MIP) having a plurality of candidates are provided. A video coding method and device for performing intra prediction using MIP having a plurality of weight matrices are provided. In addition, a video coding method and device for selecting a separable / non-separable transform kernel of MIP are provided.

[0099] The following embodiments may be performed by an intra prediction unit (122) within a video encoding apparatus. In addition, the following embodiments may be performed by an intra prediction unit (542) within a video decoding apparatus.

[0100] The video encoding device can generate signaling information related to the present embodiment in terms of rate-distortion optimization in encoding the current block. The video encoding device can encode the signaling information using an entropy encoding unit (155) and then transmit it to an image decoding device. The video decoding device can decode signaling information related to decoding the current block from a bitstream using an entropy decoding unit (510).

[0101] In the following description, the term "target block" may be used interchangeably with the current block or coding unit (CU). Alternatively, the term "target block" may also refer to a portion of a coding unit.

[0102] A flag having a value of true indicates that the flag is set to 1. A flag having a value of false indicates that the flag is set to 0.

[0103] The decoder side includes all or part of the inverse quantization unit (160), the inverse transform unit (165), the prediction unit (120), the adder (170), the loop filter unit (180), and the memory (190) in the video encoding device illustrated in FIG. 1. Alternatively, the decoder side includes all or part of the inverse quantization unit (520), the inverse transform unit (530), the prediction unit (540), the adder (550), the loop filter unit (560), and the memory (570) in the video decoding device illustrated in FIG. 5. With respect to a series of decoding processes, the decoder side of the video encoding device and the decoder side of the video decoding device perform the same operation. The video encoding device determines information related to the operation of the decoder side and signals the determined information to the video decoding device. The video decoding device can decode the signaled information and operate the decoder side based on the decoded information.

[0104] I-1. VVC's intra prediction technology

[0105] In intra prediction of VVC, the angular prediction direction is subdivided into 65, as shown in the example of Fig. 3a. Depending on the prediction angle of the intra prediction mode, prediction modes (predModeIntra) from 2 to 66 can be used. By introducing Wide-Angle Intra Prediction (WAIP), prediction modes of -14 to -1 and 67 to 80, which are directional modes with larger angles, can be used depending on the aspect ratio of the block. In intra prediction, a prediction block can be generated based on 67 Intra Prediction Modes (IPMs). The 67 IPMs refer to 67 intra prediction modes that can be signaled according to the aspect ratio of the block among prediction modes -14 to 80, including non-directional prediction modes such as Planar and DC modes.

[0106] VVC's intra prediction technology can generate intra prediction blocks using technologies such as MRLP (Multiple Reference Line intra Prediction), CCLM (Cross-Component Linear Model), PDPC (Position Dependent intra Prediction Combination), ISP (Intra Sub-Partitions), and MIP (Matrix-based Intra Prediction).

[0107] I-2. Template Matching Prediction

[0108] Template Matching Prediction (TMP) searches for a prediction block that minimizes the differences among templates within a predefined reconstruction region of the current frame, i.e., the search region. The template of the current block (hereinafter, "current template") consists of upper and left neighboring samples. The difference between the template of the current block and the template of the predicted block found in the search region is defined as a cost function. Template Matching Prediction determines the prediction block with the minimum cost as the prediction block of the current block. The Sum of Absolute Differences (SAD) is used as the cost function.

[0109] IntraTMP (Intra Template Matching Prediction) technology in ECM (Enhanced Compression Model), the next-generation technology of VVC, sets an L-shaped / left / upper template around the current block, searches for the template most similar to the current template in the restoration area of ​​the current frame, and then uses a block adjacent to the searched template and having the same size as the current block as the prediction block of the current block. IntraTMP technology searches for similar templates based on a cost function, and uses SAD (Sum of Absolute Differences) as the cost function. The video encoding device transmits whether to use IntraTMP mode, and the video decoding device can perform the same template matching task when applying IntraTMP mode.

[0110] Figure 6 is an example diagram showing a search area used in IntraTMP technology.

[0111] To save memory, the current CTU (Coding Tree Unit) where the current block is located, the upper left CTU, the upper CTU, and the left CTU can be limited to the possible search areas. In Fig. 6, the search areas are represented by R1 to R4. Among the possible search areas, an area generated according to the product of the width and height (w, h) of the current block and a preset constant a can be adaptively set as the search range. For example, by setting a = 5, the search ranges SearchRange_w and SearchRange_h can be determined as in Equation 1.

[0112]

[0113] In mathematical expression 1, BlkW and BlkH represent the width and height of the current block.

[0114] The template search area in IntraTMP mode can be predefined according to an agreement between the projection encoding device and the image decoding device.

[0115] IntraTMP technology may include sub-modes such as a mode using a single template (TMP single), a technology that fuses multiple templates (TMP fusion), a sub-pixel precision mode, and a linear filter model mode.

[0116] In single-template mode, the template search process involves two steps. In the first step, a search is performed at 3-pixel intervals, and block vectors (BVs) designating 30 candidate templates (i.e., templates of candidate reference blocks) are included in a candidate list. In the second step, an additional search is performed at 1-pixel intervals in a surrounding 3×3 region for the 30 BVs in the candidate list, resulting in a final selection of 19 BVs. Thereafter, an optimal reference block is selected in terms of rate-distortion optimization, and a candidate list index indicating the finally selected BV is signaled from the video encoding device to the video decoding device.

[0117] TMP fusion combines multiple reference templates and reference blocks using weights.

[0118] Sub-pixel precision mode searches for similar templates based on precisions of 1 / 2-Pel, 1 / 4-Pel, and 3 / 4-Pel.

[0119] The linear filter model mode utilizes a 6-tap filter to filter predicted blocks. The 6-tap filter is a cross-shaped 5-tap filter with an added bias. The filter coefficients can be calculated based on the relationship between the current template and the reference template. The current block is predicted using the reference block to which the generated filter is applied.

[0120] I-3. DIMD (Decoder-side Intra Mode Derivation) technology

[0121] The DIMD method sets reconstructed pixels adjacent to the current block as a template, and performs gradient analysis on the set template to derive directional intra prediction modes. The DIMD method extracts orientation and magnitude information by analyzing vertical and horizontal gradients within the template. The DIMD method can calculate the horizontal gradient Gx and the vertical gradient Gy, respectively, by applying a horizontal Sobel filter and a vertical Sobel filter to the pixel position at the center of the template. The orientation of the corresponding pixel position is calculated through atan(Gy / Gx), and the sum of the absolute values ​​of Gx and Gy can be calculated as the magnitude of the orientation. The size of the template may be equal to or larger than the size of the Sobel filter. The template may include adjacent samples at the lower left and upper right of the current block.

[0122] The angle and magnitude calculated for each pixel position in the center of the template region are used to generate a Histogram of Gradient (HoG), which is constructed by accumulating the angle along the x-axis and the magnitude along the y-axis. The DIMD method can derive or determine corresponding intra prediction modes and corresponding weights by mapping a plurality of angles (e.g., up to 5) with relatively high accumulated magnitude values ​​on the HoG to directional modes. The intra prediction modes determined by the DIMD method may be referred to as DIMD modes and can be subsequently utilized to generate a Most Probable Mode (MPM).

[0123] To form the final predicted block, the predictors of the intra prediction modes determined by DIMD can be weighted and combined with non-directional predictors (based on planar or block vectors).

[0124] Meanwhile, in intra prediction, MPM technology utilizes the intra prediction modes of surrounding blocks when intra-predicting the current block. The video encoding device generates an MPM list that includes intra-prediction modes derived from predefined locations spatially adjacent to the current block. By transmitting the index of the MPM list instead of the index of the prediction mode, the video encoding device can improve the encoding efficiency of the intra-prediction mode.

[0125] I-4. TIMD (Template-based intra mode derivation) technology

[0126] TIMD technology performs template prediction by applying the intra prediction mode of the MPM list to the template region surrounding the current block. TIMD technology calculates the Sum of Absolute Transformed Differences (SATD) cost between the predicted value in the template region and the reconstructed template value. Based on the estimated cost, TIMD technology selects the mode with the lowest cost (costMode1) (hereinafter, the first mode) and the mode with the second lowest cost (costMode2) (hereinafter, the second mode). The undirected mode with the lower SATD cost (costMode3) among DC or Planar is selected as the third mode. Undirected modes can also be used if the following conditions are met.

[0127] - If the third non-directional mode is different from the first and second modes.

[0128] - If costMode3 < 1.5×costMode1

[0129] When both of the above conditions are true, the predicted blocks according to the three intra prediction modes are combined using weights as shown in Equation 2.

[0130]

[0131] If either of the two conditions described above is false, the first and second modes are used. The TIMD technique determines whether to combine the weights of the two TIMD modes according to Equation 3.

[0132]

[0133] If Equation 3 is satisfied, prediction blocks according to the two modes are weighted and combined. Conversely, if Equation 3 is not satisfied, the prediction mode with the lower cost is used. When two modes are combined, a greater weight is assigned to the mode with the lower SATD cost according to Equation 4.

[0134]

[0135] Figure 7 is an example diagram showing a method for deriving one or more intra prediction modes in TIMD technology.

[0136] The size of the template area can be determined based on the size of the current block, as shown in Fig. 7. If the width (W) or height (H) of the current block is greater than 8, the template size L1 or L2 is set to 4. If the width or height of the current block is 8 or less, L1 or L2 is set to 2.

[0137] I-5. SGPM (Spatial Geometric partitioning mode) technology

[0138] SGPM technology divides the current block into two regions and performs different intra predictions on each region. As illustrated in Fig. 8, the current block can be divided into two parts with different intra prediction modes ("intra_pred_mode0" and "intra_pred_mode1") using a partition mode. This information can be expressed as a single candidate index ("sgpm_cand_idx") that indicates a combination of one partition mode ("partition mode idx") and two intra prediction modes ("intra_pred_mode0_idx" and "intra_pred_mode0_idx"), as illustrated in Fig. 8.

[0139] The order of candidates in the list is reorganized in order of lowest SAD cost in the template area, as shown in Fig. 9. That is, templates can be used to generate a candidate list. Fig. 9 illustrates a template (1120) represented by a current block (1110) and two template parts (1120a, 1120b).

[0140] For each possible combination of one partition mode and two intra prediction modes, a prediction for the template (1120) is generated, and blending weights for the current block are extended toward the template (1120). Alternatively, each template region can use a single value as a blending weight based on the partition boundary. For example, a value of 0 or 1 can be used as the weight of each template region. The combinations described above are ranked in ascending order according to the sum of absolute difference (SAD) cost between the template prediction and the reconstruction. The size of the candidate list can be set to, for example, 16, and these candidates are considered the most probable SGPM combinations of the current block. The video encoding device and the video decoding device construct the same candidate list using the template (1120). The video encoding device selects an optimal candidate and signals the index of the selected candidate to the video decoding device.

[0141] To reduce the complexity of candidate list generation, both the number of available partition modes and the number of available intra prediction modes can be limited. For example, in the SGPM adopted in ECM, 26 partition modes and 9 intra prediction modes are used to form combinations. The 9 intra prediction modes include some of the typical intra prediction modes illustrated in Figure 3a (e.g., a directional mode parallel to the partition boundary, a directional mode perpendicular to the partition boundary, and a PLANAR mode), and may further include prediction modes based on block vectors obtained from neighboring blocks (e.g., an IntraTMP mode or an IBC mode).

[0142] I-6. MIMD (Merged intramode derivation) technology

[0143] MIMD (Merged intra mode derivation) mode derives MHoG (Merged Histogram of Gradient) using information extracted from surrounding blocks.

[0144] The MIMD mode utilizes N neighboring blocks (where N is a positive integer greater than or equal to 2) predicted according to at least one directional intra prediction mode. If neighboring block i is predicted according to DIMD or MIMD, then the HoG or MHoG of that block is H i is considered. Hi(m) represents the amplitude corresponding to the directional prediction mode m. If the surrounding block i is predicted based on the directional intra prediction mode m, H i (m) is set to A, and H i (k)=0, k≠i. A may be a value that depends on the size of the current block. If the surrounding block i is predicted by SGPM or TIMD and one or more intra prediction modes are available, the corresponding intra prediction modes are H i can be considered in the derivation of .

[0145] MIMD mode is, as mentioned above, H extracted from surrounding blocks. i MHoG can be calculated using Equation 5.

[0146]

[0147] The MIMD mode can derive intra prediction modes and corresponding weights for MIMD based on the intra prediction modes with the highest amplitudes in MHoG (e.g., up to 5).

[0148] I-7. OBIC (Occurrence-based intra coding) technology

[0149] OBIC technology derives an intra prediction mode based on neighboring reference sample information of the current block at the decoder side, and combines it with an existing prediction mode (planar or non-directional prediction based on block vectors) to generate a final predicted block. The video decoding device can generate a HoC by counting the frequency of the intra prediction mode used to predict the surrounding reference samples for each sample. The video decoding device predetermines a neighboring block area that is immediately adjacent or not adjacent to the current block, collects the intra prediction mode of the corresponding block, and constructs a HoC based on the occurrence frequency of the intra prediction mode. The occurrence frequency is calculated based on the number of samples encoded with a specific intra prediction mode. For example, if a block ui having a length of horizontal uiWidth and a length of vertical uiHeight is encoded using a specific intra prediction mode IPM (Intra prediction mode), the HoC [IPM] is calculated as in Equation 6.

[0150]

[0151] In Equation 6, IPM can be one of the 67 intra prediction modes illustrated in Fig. 3a.

[0152] The video decoding device selects five IPMs according to the order of the most frequent values ​​of the HoC. The video decoding device generates prediction blocks based on the five selected prediction modes and weights and combines the generated prediction blocks to generate a prediction block. Similar to DIMD, the predictors of the intra prediction modes determined according to the HoC can be weighted and combined with non-directional predictors (based on planar or block vectors) to form the final prediction block. Meanwhile, the weights associated with each prediction mode can be set in descending order of the HoC value based on the HoC value.

[0153] The video decoding device applies the HoC-based intra prediction mode only to luma blocks. The HoC-based intra prediction mode is not applied to blocks containing fewer than 64 samples. The HoC-based intra prediction mode is signaled using a PU-level flag. The HoC-based intra prediction mode is used as a submode of DIMD, and this flag can be transmitted after the DIMD flag.

[0154] I-8. Matrix-based Intra Prediction (MIP)

[0155] Matrix-based intra prediction (MIP) derives prediction samples that replace conventional intra prediction by multiplying reference pixels by a weight matrix. The process of deriving the prediction samples can be expressed as shown in Equation 7.

[0156]

[0157] In Equation 7, the prediction sample P(x, y) is calculated by multiplying the reference pixel r(k) and the weight matrix F(x, y, k). k represents the index of the reference pixel.

[0158] A set of reference pixels is called a reference template. The reference template of a conventional MIP can be exemplified as shown in Fig. 10. The reference template corresponds to the input of the MIP. The reference template includes reconstructed 4-line samples (T2 = 4) adjacent to the left side of the current block and reconstructed 4-line samples (T1 = 4) adjacent to the top side of the current block, and has an L-shape. The width and height of the reference template have values ​​of 2W and 2H, respectively. Here, W represents the width of the current block, H represents the height of the current block, and r represents reference pixels or the reference template.

[0159] MIP is activated for blocks of a predetermined size or smaller, with respect to a preset intra prediction mode (e.g., intra prediction modes 0, 1, 2, 6, 10, …). When MIP is activated, intra prediction according to the corresponding intra prediction mode is replaced with MIP. Therefore, when MIP is applied, no additional information other than intra prediction mode information is signaled.

[0160] The following embodiments are described with a focus on a video decoding device, but can be implemented identically or similarly in a video encoding device. Alternatively, the following embodiments are described with a focus on the decoder side of a video decoding device, but can also be implemented identically or similarly in the decoder side of a video encoding device.

[0161] II. Embodiments according to the present disclosure

[0162] As one embodiment of the present disclosure, matrix-based intra prediction using multiple candidates is described.

[0163] In this embodiment, the image decoding device performs matrix-based intra prediction (MIP) with multiple candidates. MIP candidates can be derived in various ways depending on the location of the reference pixel, the characteristics of the reference pixel, the form of the reference pixels (e.g., a group of reference pixels or a reference template), the type of the weight matrix, etc.

[0164] For example, pixel level values ​​may be utilized as characteristics of reference pixels. An image decoding device may divide reference pixels into N groups according to predetermined reference level values, and derive MIP candidates for each of the N groups. As another example, integer / sub-pixel may be utilized as characteristics of reference pixels. When interpolation is applied to reference pixels, an image decoding device may derive one MIP candidate using only pixels at integer positions among reference pixels, derive another MIP candidate using only pixels at sub-pixel positions, and derive another MIP candidate based on reference pixels corresponding to integers and sub-pixels.

[0165] A video decoding device can construct a MIP candidate list by performing candidate derivation methods in a set order. For example, when constructing each candidate list in the order of three methods A, B, and C, the video decoding device can generate X MIP candidates based on method A (A1, A2, …, AX), Y MIP candidates based on method B (B1, B2, …, BY), and Z MIP candidates based on method C (C1, C2, …, CZ). The candidate list constructed according to the above-described method can be exemplified as shown in Table 1. The MIP candidate list construction method can be defined in advance according to an agreement between the video encoding device and the video decoding device.

[0166]

[0167] For example, the MIP candidate information used for the current block can be explicitly signaled. When there are four MIP candidates, the video encoding device can signal a MIP index encoded with 2 bits using a fixed length code (FLC). When configuring the front part of the MIP candidate list based on a method with higher efficiency or a method for deriving candidates likely to be selected frequently, the video encoding device can signal a MIP index encoded with a variable length code (VLC). Accordingly, fewer bits can be allocated to candidates located at the front of the candidate list.

[0168] As another embodiment according to the present disclosure, matrix-based intra prediction using multiple weight matrices is described.

[0169] In the present embodiment, the image decoding device performs MIP using a plurality of weight matrices. As an example, the image decoding device can use M weight matrices according to the positions of reference pixels for MIP. The M weight matrices can form a candidate list as described above. For example, a total of three weight matrices can be used, such as a weight matrix for an L-shaped reference pixel, a weight matrix for a left reference pixel, and a weight matrix for an upper reference pixel. Thereafter, the image decoding device can implicitly use the corresponding weight matrix depending on a reference pixel selection method, or explicitly use signaled weight matrix information (hereinafter, expressed as weightMatrixIdx).

[0170] Alternatively, the image decoding device may utilize N weight matrices based on reference values ​​based on surrounding block information. Reference values ​​based on surrounding block information may include the average of reference pixel level values, the median of reference pixel level values, and directional information of surrounding blocks of the current block (e.g., intra prediction mode, gradient, etc.).

[0171] For example, if the reference value is the average value of the reference pixel level values, the range of the level values ​​can be divided into three sections, and two threshold values ​​(th1, th2) for distinguishing the three sections can be defined in advance according to an agreement between the image encoding device and the image decoding device. The reference value and the two threshold values ​​can be utilized as in mathematical expression 8.

[0172]

[0173] In mathematical expression 8, when the average value avglev satisfies 0 ≤ avglev < th1, the image decoding device uses the first weight matrix (weightMatrixIdx = 0). When the average value satisfies th1 ≤ avglev < th2, the image decoding device uses the second weight matrix (weightMatrixIdx = 1). When the average value satisfies th2 ≤ avglev, the image decoding device can use the third weight matrix (weightMatrixIdx = 2). In the above-described method, since the weight matrix is ​​determined according to a reference value based on the surrounding block information, information indicating the weight matrix (hereinafter, weight matrix information) is not signaled.

[0174] As another example, when using multiple weight matrices, weight matrix information can be explicitly signaled. For example, multiple weight matrices can be configured regardless of the intra prediction mode. When the intra prediction mode to which MIP is applied is selected, the video encoding device can perform prediction of the current block based on the K weight matrices, and then signal weight matrix information (e.g., an index) that provides the smallest cost value according to a predetermined cost function. The video decoding device can use the signaled information to identify a weight matrix actually used for intra prediction among the K weight matrices.

[0175] As another example, multiple different weight matrices can be defined for each intra prediction mode. That is, for intra prediction mode A to which MIP is applied, K A The weight matrices of the dog are defined, and K for mode B B The weight matrices of K can be defined. In this case, K A The weight matrices of the dogs and K B The weight matrices of the dogs are different matrices. K A Wow K B can be the same value or different values.

[0176] As another example, multiple different weight matrices can be defined depending on the size of the current block. For example, for a 4×4 block to which MIP is applied, K 4x4 The weight matrices of K are defined, and for a 4×16 block, 4x16 Different weight matrices can be defined depending on the block size, such as the weight matrices of K 4x4 Wow K 4x16 can be the same value or different values.

[0177] As another embodiment according to the present disclosure, selection of separable / non-separable transform kernels in matrix-based intra prediction is described.

[0178] In the conventional MIP technology, when determining a separable / non-separable transform set / transform kernel from separable / non-separable transform sets, the image decoding device uses information of the intra prediction mode replaced by the MIP. However, a problem may arise in that the characteristics of the residual remaining after performing MIP prediction may be different from the characteristics of the residual according to the existing intra prediction. Therefore, in the present embodiment, instead of the intra prediction mode of the current block, the image decoding device newly derives / specifies a virtual intra prediction mode (VIM) used to select the separable / non-separable transform kernel of the MIP.

[0179] A separable transformation applies a set of transformations / kernels separately to the horizontal and vertical directions of a transformation block. A non-separable transformation transforms a transformation block into a one-dimensional vector and then applies a single set of transformations / kernels. An example of a non-separable transformation is the LFNST (Low-frequency non-separable transform), which is used for secondary transformations, and can be applied after the primary transformation, which is a separable transformation.

[0180] As one embodiment according to the present disclosure, a virtual intra prediction mode derivation using gradients of adjacent / non-adjacent samples is described.

[0181] In this embodiment, the image decoding device derives a virtual intra prediction mode for determining a separable / non-separable transform set / kernel using the gradient of adjacent / non-adjacent samples. Here, non-adjacent samples refer to samples of an area determined according to an agreement between the image encoding and image decoding devices, or samples of an arbitrary area indicated by the additional information of the current block.

[0182] As an example, an image decoding device applies a boundary detection filter, such as a Sobel filter, a Roberts cross filter, a Prewitt filter, a Scharr filter, a Laplacian filter, etc., to adjacent / non-adjacent samples to calculate all gradients (or slopes) of adjacent / non-adjacent samples, and generates a histogram of gradients (HoG) based on the gradients. The HoG can be generated by accumulating the magnitude (i.e., amplitude) of the gradients related to the direction of the gradients. The image decoding device can select a gradient with the largest amplitude from the gradient histogram, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode of the current block. Here, mapping is the process of matching or approximating the direction of the gradient to, for example, one of the directional intra prediction modes shown in Fig. 3a.

[0183] As another example, a predetermined number of gradients may be selected from a gradient histogram in an arbitrary order or in the order of increasing amplitude, and the directions of the selected gradients may be mapped to intra prediction modes, respectively, so that the mapped prediction modes may be used as candidates for a virtual intra prediction mode of the current block. The video encoding device may perform encoding, such as prediction / transformation, on the candidate modes (i.e., the intra prediction mode information of the current block and the virtual intra prediction mode), and then determine the candidate mode that is optimal in terms of rate-distortion or has the smallest cost value for a predetermined cost function as the virtual intra prediction mode of the current block. The video encoding device may signal related information (e.g., whether to use a virtual intra prediction mode, a transform set index / transform kernel index / virtual intra prediction mode, etc.) to the video decoding device.

[0184] As another example, the video decoding device selects a predetermined number of gradients from the gradient histogram in an arbitrary order or in ascending order of amplitude, and maps the directions of the selected gradients to intra prediction modes, respectively. Thereafter, the video decoding device can determine the intra prediction mode with the smallest mode number or the intra prediction mode with the largest mode number among the assigned intra prediction modes as the virtual intra prediction mode of the current block.

[0185] When applying methods for deriving a virtual intra prediction mode using gradients of adjacent / non-adjacent samples, if all amplitudes of the gradient histogram are below a specific value, the image decoding device can determine the virtual intra prediction mode of the current block as the Planar mode, the DC mode, or an arbitrary predetermined mode.

[0186] As another embodiment according to the present disclosure, a virtual intra prediction mode derivation based on gradient histograms of adjacent / non-adjacent blocks is described.

[0187] In this embodiment, the image decoding device derives a virtual intra prediction mode for determining a separable / non-separable transform set / kernel using gradient histograms of adjacent / non-adjacent blocks. Here, non-adjacent samples refer to samples of an area determined according to an agreement between the image encoding and image decoding devices, or samples of an arbitrary area indicated by the additional information of the current block.

[0188] The gradient histogram for determining the virtual intra prediction mode of the current block can be generated as follows.

[0189] For example, when the intra prediction mode of adjacent / non-adjacent blocks is the decoder-side intra mode ferivation (DIMD) mode or the merged intra mode derivation (MIMD) mode, the image decoding device can use the amplitude value of the current mode derived from the histogram of gradients (HoG) or the merged histogram of gradients (MHoG) of the corresponding blocks as is or after normalizing it.

[0190] As another example, if the intra prediction mode of adjacent / non-adjacent blocks is not a decoder-side intra mode derivation mode (non-DIMD) or not a merged intra mode derivation mode (non-MIMD), the image decoding device can use a value accumulated as a real number multiple of an arbitrary fixed value, such as a real number multiple of the number of pixels of the current block, a real number multiple of the number of pixels of adjacent / non-adjacent blocks, etc., as the amplitude value of the intra prediction mode of the corresponding block.

[0191] If adjacent / non-adjacent blocks are blocks predicted according to a mode having two or more intra prediction modes, such as Spatial Geometric Partitioning Mode (SGPM) or Template-based Intra Mode Derivation (TIMD) mode, the image decoding device may use a value accumulated as a real number multiple of an arbitrary fixed value, such as a real number multiple of the number of pixels of the current block, a real number multiple of the number of pixels of adjacent / non-adjacent blocks, as an amplitude value of each intra prediction mode.

[0192] After deriving multiple gradient histograms using adjacent / non-adjacent blocks, the image decoding device can use the accumulated histogram of the multiple gradient histograms as the gradient histogram / merged gradient histogram of the current block. Alternatively, the image decoding device can combine / normalize the accumulated histograms using an operation such as the average to generate the gradient histogram / merged gradient histogram of the current block.

[0193] From the gradient histogram / merged gradient histogram generated based on information of adjacent / non-adjacent blocks, the image decoding device can select a gradient with the largest amplitude, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode of the current block.

[0194] As another example, a video encoding device selects a predetermined number of gradients in an arbitrary order or in order of increasing amplitude from a gradient histogram / merged gradient histogram. Thereafter, the video encoding device can map the directions of the selected gradients to respective intra prediction modes and use the mapped prediction modes as candidates for a virtual intra prediction mode of the current block. The video encoding device can perform encoding, such as prediction / transformation, on the candidate modes (i.e., intra prediction mode information of the current block and the virtual intra prediction mode), and then determine the candidate mode that is optimal in terms of rate distortion or the candidate mode that has the smallest cost value for a predetermined cost function as the virtual intra prediction mode of the current block. The video encoding device can signal related information (e.g., whether to use a virtual intra prediction mode, a transform set index / transform kernel index / virtual intra prediction mode, etc.) to the video decoding device.

[0195] As another example, the video decoding device selects a predetermined number of gradients in ascending order of magnitude or amplitude from the gradient histogram / merged gradient histogram, and maps the directions of the selected gradients to intra prediction modes, respectively. Thereafter, the video decoding device can determine the intra prediction mode with the smallest mode number or the intra prediction mode with the largest mode number among the assigned intra prediction modes as the virtual intra prediction mode of the current block.

[0196] When applying methods for deriving a virtual intra prediction mode using a gradient histogram / merged gradient histogram of adjacent / non-adjacent samples, if all amplitudes of the gradient histogram / merged gradient histogram are below a specific value, the image decoding device can determine the virtual intra prediction mode of the current block as a Planar mode, a DC mode, or an arbitrary predetermined mode.

[0197] As another embodiment according to the present disclosure, a virtual intra prediction mode derivation based on an occurrence histogram of adjacent / non-adjacent blocks is described.

[0198] In this embodiment, the video decoding device derives a virtual intra prediction mode for determining a separable and non-separable transform set and kernel using a histogram of occurrence rates (HoC) of adjacent / non-adjacent blocks. Here, non-adjacent samples refer to samples of an area determined according to an agreement between the video encoding and video decoding devices, or samples of an arbitrary area indicated by the additional information of the current block.

[0199] The incidence histogram for determining the virtual intra prediction mode of the current block can be generated as follows.

[0200] As an example, an image decoding device can configure intra prediction modes of adjacent / non-adjacent blocks into a histogram and use the configured histogram as an occurrence rate histogram.

[0201] As another example, the image decoding device may accumulate the histogram amplitude of the bin corresponding to the intra prediction mode of adjacent / non-adjacent blocks by a real number of times an arbitrary fixed value, such as a real number of times the number of pixels of the current block, a real number of times the number of pixels of adjacent / non-adjacent blocks, etc., and use the accumulated histogram as an occurrence rate histogram.

[0202] As another example, if adjacent / non-adjacent neighboring blocks are blocks having two or more intra prediction modes, such as SGPM, TIMD, and DIMD, the image decoding device may use a value accumulated as an amplitude value of each intra prediction mode by a real number multiple of an arbitrary fixed value, such as a real number multiple of the number of pixels of the current block or a real number multiple of the number of pixels of adjacent / non-adjacent blocks.

[0203] From the occurrence rate histogram generated based on the intra prediction modes of adjacent / non-adjacent blocks, the image decoding device can determine the intra prediction mode with the largest amplitude value as the virtual intra prediction mode of the current block.

[0204] As another example, a predetermined number of intra prediction modes in an arbitrary order or in ascending order of amplitude from the incidence histogram can be used as candidates for the virtual intra prediction mode of the current block. The video encoding device performs encoding, such as prediction / transformation, on the candidate modes (i.e., the intra prediction mode information of the current block and the virtual intra prediction mode). Thereafter, the video encoding device can determine the candidate mode that is optimal in terms of rate distortion or the candidate mode that has the smallest cost value for a predetermined cost function as the virtual intra prediction mode of the current block. The video encoding device can signal related information (e.g., whether to use the virtual intra prediction mode, transform set index / transform kernel index / virtual intra prediction mode, etc.) to the video decoding device.

[0205] As another example, a video decoding device selects a predetermined number of intra prediction modes in an arbitrary order or in ascending order of amplitude from an incidence rate histogram. Thereafter, the video decoding device may determine, among the selected intra prediction modes, the intra prediction mode with the smallest mode number or the intra prediction mode with the largest mode number as the virtual intra prediction mode of the current block.

[0206] When applying methods for deriving a virtual intra prediction mode using an occurrence rate histogram of adjacent / non-adjacent samples, if all amplitudes of the occurrence rate histogram are below a specific value, the image decoding device can determine the virtual intra prediction mode of the current block as a Planar mode, a DC mode, or an arbitrary predetermined mode.

[0207] As another embodiment according to the present disclosure, the designation of an arbitrary intra prediction mode is described.

[0208] In the present embodiment, the video decoding device specifies an arbitrary intra prediction mode as a virtual intra prediction mode for determining a separable / non-separable transform set / kernel. As the prediction accuracy of MIP increases, the residual after performing the prediction may exhibit randomness. That is, the directionality of the current block is mostly removed, so a specific directionality may not appear. Therefore, an arbitrary intra prediction mode is used for determining a separable / non-separable transform set / kernel, and information related to the arbitrary intra prediction mode may be agreed upon in advance between the video encoding device and the video decoding device. For example, when the intra prediction mode is replaced with MIP, the video decoding device may use the Planar mode to determine a separable / non-separable transform set / kernel.

[0209] Hereinafter, with reference to FIGS. 11 and 12, a method for predicting / transforming a current block based on matrix-based intra prediction is described.

[0210] FIG. 11 is a flowchart illustrating a method of encoding a current block performed by an image encoding device according to one embodiment of the present disclosure.

[0211] When the current block is predicted in an intra prediction mode involving application of Matrix-based Intra Prediction (MIP), the video encoding device may perform the following steps.

[0212] The image encoding device obtains a weight matrix for applying MIP to the current block (S1100).

[0213] As an example, an image encoding device may generate a list containing weight matrix candidates based on the locations of reference pixels, the shape of a group of reference pixels, or the type of weight matrix. The image encoding device may derive a weight matrix from the list. Subsequently, the image encoding device encodes an index indicating the weight matrix.

[0214] As another example, the video encoding device can implicitly determine the weight matrix. The video encoding device can calculate a reference value based on information about surrounding blocks. Here, the reference value can be an average value of reference pixel level values, a median value of reference pixel level values, or directional information of blocks surrounding the current block. The video encoding device can compare the calculated reference value with preset thresholds and determine the weight matrix based on the comparison result.

[0215] The image encoding device applies a weight matrix to the reference pixels of the current block to generate a prediction block of the current block (S1102).

[0216] The video encoding device derives a virtual intra prediction mode based on adjacent / non-adjacent samples of the current block (S1104).

[0217] As an example, an image encoding device can calculate gradients by applying an edge detection filter to adjacent / non-adjacent samples, and generate a gradient histogram using the gradients. The image encoding device can select a gradient with a maximum amplitude from the gradient histogram, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode. Here, the mapping is a process of matching or approximating the direction of the gradient to, for example, one of the directional intra prediction modes illustrated in FIG. 3A.

[0218] As another example, a video encoding device can obtain a flag indicating whether to use a virtual intra prediction mode from a higher level. If the flag is checked and the flag is true, indicating that the virtual intra prediction mode is to be used, the video encoding device can determine the virtual intra prediction mode as follows.

[0219] An image encoding device can calculate gradients by applying an edge detection filter to adjacent / non-adjacent samples, and generate a gradient histogram using the gradients. The image encoding device can select a preset number of gradients from the gradient histogram, map directions of the selected gradients to intra prediction modes, and generate candidate modes. The image encoding device can determine a virtual intra prediction mode based on the candidate modes in terms of rate distortion optimization or optimization of a preset cost function. Thereafter, the image encoding device can encode the determined virtual intra prediction mode or encode information indicating the determined virtual intra prediction mode.

[0220] As another example, when the intra prediction modes of adjacent / non-adjacent blocks are DIMD mode or MIMD mode, the virtual intra prediction mode can be derived as follows. The video encoding device can generate a cumulative histogram by combining the HoG or MHoG of the adjacent / non-adjacent blocks. The video encoding device can select a gradient with the maximum amplitude from the cumulative histogram, map the direction of the selected gradient to the intra prediction mode, and determine the mapped intra prediction mode as the virtual intra prediction mode.

[0221] As another example, with respect to adjacent / non-adjacent blocks, the video encoding device can generate a histogram by accumulating the frequency of each intra prediction mode. The video encoding device can select a gradient with the maximum amplitude from the histogram, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode.

[0222] As another example, with respect to adjacent / non-adjacent blocks, the video encoding device can generate a histogram using an amplitude value of each intra prediction mode, which is accumulated by a real number of times the number of pixels of the adjacent / non-adjacent blocks. The video encoding device can select a gradient with the maximum amplitude from the histogram, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode.

[0223] As another example, when adjacent / non-adjacent blocks have two or more intra prediction modes, a virtual intra prediction mode can be derived as follows. An image encoding device can generate a histogram using an amplitude value of each intra prediction mode in relation to adjacent / non-adjacent blocks, a value accumulated by a real number multiple of the number of pixels of the adjacent / non-adjacent blocks. The image encoding device can select a gradient with a maximum amplitude from the histogram, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode.

[0224] The video encoding device selects a separable / non-separable transformation kernel of the current block based on the virtual intra prediction mode (S1106).

[0225] The video encoding device subtracts the prediction block from the current block to generate a residual block of the current block (S1108).

[0226] The video encoding device applies the separable / non-separable transformation kernel of the current block to the residual block to generate transformation coefficients (S1110).

[0227] Thereafter, the image encoding device can quantize the transform coefficients to generate quantized transform coefficients and encode the quantized transform coefficients.

[0228] FIG. 12 is a flowchart illustrating a method for restoring a current block performed by an image decoding device according to one embodiment of the present disclosure.

[0229] If the current block is predicted in an intra prediction mode corresponding to the application of MIP, the image decoding device can perform the following steps.

[0230] The video decoding device obtains a weight matrix for applying MIP to the current block (S1200).

[0231] As an example, a video decoding device can decode an index indicating a weight matrix from a bitstream. The video decoding device can generate a list including weight matrix candidates based on the positions of reference pixels, the shape of a group of reference pixels, or the type of the weight matrix. The video decoding device can derive a weight matrix from the list using the index.

[0232] As another example, the image decoding device can implicitly determine the weight matrix. The image decoding device can calculate a reference value based on information about surrounding blocks. Here, the reference value can be an average value of reference pixel level values, a median value of reference pixel level values, or directional information of blocks surrounding the current block. The image decoding device can compare the calculated reference value with preset thresholds and determine a weight matrix based on the comparison result.

[0233] The image decoding device applies a weight matrix to reference pixels of the current block to generate a prediction block of the current block (S1202).

[0234] The video decoding device derives a virtual intra prediction mode based on adjacent / non-adjacent samples of the current block (S1204).

[0235] As an example, an image decoding device may calculate gradients by applying an edge detection filter to adjacent / non-adjacent samples, and generate a gradient histogram using the gradients. The image decoding device may select a gradient with a maximum amplitude from the gradient histogram, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode. Here, the mapping is a process of matching or approximating the direction of the gradient to, for example, one of the directional intra prediction modes illustrated in FIG. 3A.

[0236] As another example, a video decoding device can decode a flag indicating whether a virtual intra prediction mode is used from a bitstream. If the flag is checked and the flag is true, and thus the virtual intra prediction mode is determined to be used, the video decoding device can decode the virtual intra prediction mode from the bitstream or decode information indicating the virtual intra prediction mode.

[0237] As another example, when the intra prediction modes of adjacent / non-adjacent blocks are DIMD mode or MIMD mode, the virtual intra prediction mode can be derived as follows. The image decoding device can generate a cumulative histogram by combining the HoG or MHoG of the adjacent / non-adjacent blocks. The image decoding device can select a gradient with the maximum amplitude from the cumulative histogram, map the direction of the selected gradient to the intra prediction mode, and determine the mapped intra prediction mode as the virtual intra prediction mode.

[0238] As another example, with respect to adjacent / non-adjacent blocks, the image decoding device can generate a histogram by accumulating the frequency of each intra prediction mode. The image decoding device can select a gradient with the maximum amplitude from the histogram, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode.

[0239] As another example, with respect to adjacent / non-adjacent blocks, the image decoding device can generate a histogram using an amplitude value of each intra prediction mode, which is accumulated as many times as a real number of pixels of the adjacent / non-adjacent blocks. The image decoding device can select a gradient with the maximum amplitude from the histogram, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode.

[0240] As another example, when adjacent / non-adjacent blocks have two or more intra prediction modes, a virtual intra prediction mode can be derived as follows. An image decoding device can generate a histogram using an amplitude value of each intra prediction mode in relation to adjacent / non-adjacent blocks, which is accumulated by a real number multiple of the number of pixels of the adjacent / non-adjacent blocks. The image decoding device can select a gradient with a maximum amplitude from the histogram, map the direction of the selected gradient to an intra prediction mode, and determine the mapped intra prediction mode as a virtual intra prediction mode.

[0241] The video decoding device selects a separable / non-separable inverse transform kernel of the current block based on the virtual intra prediction mode (S1206).

[0242] The video decoding device restores the transform coefficients of the current block from the bitstream (S1208). For example, the video decoding device can decode quantized transform coefficients from the bitstream and inversely quantize the quantized transform coefficients to generate transform coefficients.

[0243] The image decoding device applies the separable / non-separable inverse transform kernel of the current block to the transformation coefficients to generate a residual block of the current block (S1210).

[0244] Afterwards, the video decoding device can generate a restoration block of the current block by adding the prediction block and the residual block.

[0245] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0246] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.

[0247] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0248] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0249]

[0250]

[0251] CROSS-REFERENCE TO RELATED APPLICATION

[0252] This patent application claims priority to Korean patent application No. 10-2024-0041903, filed in Korea on March 27, 2024, and Korean patent application No. 10-2025-0030905, filed in Korea on March 10, 2025, the entire contents of which are incorporated herein by reference.

Claims

1. In a method for restoring a current block performed by an image decryption device, A step of obtaining a weight matrix for applying matrix-based intra prediction (MIP) to the current block; A step of applying the weight matrix to the reference pixels of the current block to generate a prediction block of the current block; A step of deriving a virtual intra prediction mode based on adjacent / non-adjacent samples of the current block; A step of selecting a separable / non-separable inverse transform kernel of the current block based on the virtual intra prediction mode; A step of restoring the transform coefficients of the current block from the bitstream; and A step of generating a residual block of the current block by applying the separable / non-separable inverse transform kernel of the current block to the transformation coefficients. A method comprising:

2. In paragraph 1, The step of obtaining the above weight matrix is: A step of decrypting an index from the above bitstream; A step of generating a list including weight matrix candidates based on the positions of the reference pixels, the shape of the group of the reference pixels, or the type of the weight matrix; and A step of deriving the weight matrix from the list using the above index. A method comprising:

3. In paragraph 1, The step of obtaining the above weight matrix is: A step of calculating a reference value based on the reference pixel level value of the current block; and A step of determining the weight matrix based on the result of comparing the above reference value with preset threshold values. A method comprising:

4. In paragraph 1, The step of deriving the above virtual intra prediction mode is: A step of calculating gradients by applying a boundary detection filter to the adjacent / non-adjacent samples; A step of generating a gradient histogram using the above gradients; and A step of selecting a gradient having the maximum amplitude from the gradient histogram, mapping the direction of the selected gradient to an intra prediction mode, and determining the mapped intra prediction mode as the virtual intra prediction mode. A method comprising:

5. In paragraph 1, The step of deriving the above virtual intra prediction mode is: A step of decoding a flag indicating whether to use the virtual intra prediction mode from the bitstream; and Including the step of checking the above flag, If it is decided to use the above virtual intra prediction mode, A step of decoding the virtual intra prediction mode from the bitstream. A method further comprising:

6. In paragraph 1, When the intra prediction mode of the adjacent / non-adjacent blocks is a decoder-side intra mode derivation mode or a merged intra mode derivation mode, the step of deriving the virtual intra prediction mode is, A step of generating a cumulative histogram by combining the histogram of gradients or the merged histogram of gradients of the adjacent / non-adjacent blocks; and A step of selecting a gradient having the maximum amplitude from the above cumulative histogram, mapping the direction of the selected gradient to an intra prediction mode, and determining the mapped intra prediction mode as the virtual intra prediction mode. A method comprising:

7. In paragraph 1, The step of deriving the above virtual intra prediction mode is: In relation to the above adjacent / non-adjacent blocks, a step of accumulating the frequency of each intra prediction mode to generate a histogram; and A step of selecting a gradient having the maximum amplitude from the histogram, mapping the direction of the selected gradient to an intra prediction mode, and determining the mapped intra prediction mode as the virtual intra prediction mode. A method comprising:

8. In paragraph 1, The step of deriving the above virtual intra prediction mode is: In relation to the adjacent / non-adjacent blocks, a step of generating a histogram using a value accumulated as a real number times the number of pixels of the adjacent / non-adjacent blocks as an amplitude value of each intra prediction mode; and A step of selecting a gradient having the maximum amplitude from the histogram, mapping the direction of the selected gradient to an intra prediction mode, and determining the mapped intra prediction mode as the virtual intra prediction mode. A method comprising:

9. In paragraph 1, If the above adjacent / non-adjacent blocks have two or more intra prediction modes, the step of deriving the virtual intra prediction mode is: In relation to the adjacent / non-adjacent blocks, a step of generating a histogram using a value accumulated as a real number times the number of pixels of the adjacent / non-adjacent blocks as an amplitude value of each intra prediction mode; and A step of selecting a gradient having the maximum amplitude from the histogram, mapping the direction of the selected gradient to an intra prediction mode, and determining the mapped intra prediction mode as the virtual intra prediction mode. A method comprising:

10. In a method for encoding a current block performed by a video encoding device, A step of obtaining a weight matrix for applying matrix-based intra prediction (MIP) to the current block; A step of applying the weight matrix to the reference pixels of the current block to generate a prediction block of the current block; A step of deriving a virtual intra prediction mode based on adjacent / non-adjacent samples of the current block; A step of selecting a separable / non-separable transformation kernel of the current block based on the virtual intra prediction mode; A step of generating a residual block of the current block by subtracting the predicted block from the current block; and A step of generating transformation coefficients by applying the separation / non-separation transformation kernel of the current block to the residual block. A method comprising:

11. In paragraph 10, The step of obtaining the above weight matrix is: A step of generating a list including weight matrix candidates based on the positions of the reference pixels, the shape of a group of the reference pixels, or the type of the weight matrix; A step of deriving the weight matrix from the above list; and A step of encoding an index indicating the above weight matrix A method comprising:

12. In paragraph 10, The step of deriving the above virtual intra prediction mode is: A step of calculating gradients by applying a boundary detection filter to the adjacent / non-adjacent samples; A step of generating a gradient histogram using the above gradients; and A step of selecting a gradient having the maximum amplitude from the gradient histogram, mapping the direction of the selected gradient to an intra prediction mode, and determining the mapped intra prediction mode as the virtual intra prediction mode. A method comprising:

13. In paragraph 10, The step of deriving the above virtual intra prediction mode is: A step of obtaining a flag indicating whether to use the virtual intra prediction mode from a higher level; and Including the step of checking the above flag, If it is decided to use the above virtual intra prediction mode, A step of calculating gradients by applying a boundary detection filter to the adjacent / non-adjacent samples; A step of generating a gradient histogram using the above gradients; A step of selecting a predetermined number of gradients from the gradient histogram, mapping the directions of the selected gradients to intra prediction modes, and generating candidate modes; A step of determining the virtual intra prediction mode based on the candidate modes in terms of rate distortion optimization or optimization of a preset cost function; and A step of encoding the above virtual intra prediction mode A method further comprising:

14. In paragraph 10, The step of deriving the above virtual intra prediction mode is: In relation to the above adjacent / non-adjacent blocks, a step of accumulating the frequency of each intra prediction mode to generate a histogram; and A step of selecting a gradient having the maximum amplitude from the histogram, mapping the direction of the selected gradient to an intra prediction mode, and determining the mapped intra prediction mode as the virtual intra prediction mode. A method comprising:

15. In a method for providing video data to a video decoding device, A step of encoding the above video data into a bitstream; and Including a step of transmitting the bitstream to the image decoding device, The step of encoding the above video data is: A step of obtaining a weight matrix for applying matrix-based intra prediction (MIP) to the current block; A step of applying the weight matrix to the reference pixels of the current block to generate a prediction block of the current block; A step of deriving a virtual intra prediction mode based on adjacent / non-adjacent samples of the current block; A step of selecting a separable / non-separable transformation kernel of the current block based on the virtual intra prediction mode; A step of generating a residual block of the current block by subtracting the predicted block from the current block; and A step of generating transformation coefficients by applying the separation / non-separation transformation kernel of the current block to the residual block. A method comprising:

Images