Method for processing luma block and chroma block in neural network-based filter

Abstract

The present embodiment discloses a method for processing a luma block and a chroma block in a neural network-based filter. In the present embodiment, an image decoding device decodes parameters of an in-loop filter, wherein the in-loop filter is trained in advance on the basis of deep learning. The image decoding device obtains a reconstructed block of the current block, wherein the reconstructed block includes a luma block and a chroma block, and the luma block has the size of W × H. The image decoding device generates transform luma blocks in which spatial resolution has been decomposed by applying transform to the luma block, and generates stacked blocks on the basis of the transform luma blocks and the chroma block. The image decoding device performs filtering by inputting the stacked blocks to the in-loop filter, wherein different convolution operations are applied to the transform luma blocks and the chroma block.

Description

A method for processing luma and chroma blocks in neural network-based filters.

[0001] The present disclosure relates to a method for processing luma blocks and chroma blocks in a neural network-based filter.

[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.

[0003] Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.

[0004] Therefore, when storing or transmitting video data, the encoder compresses the video data and stores or transmits it, and the decoder receives the compressed video data, decompresses it, and plays it back. These video compression technologies include H.264 / AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves encoding efficiency by about 30% compared to HEVC.

[0005] However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.

[0006] Recently, deep learning-based image processing technologies are being applied to existing encoding element technologies. By applying deep learning-based image processing to existing encoding techniques, such as inter-prediction, intra-prediction, in-loop filters, and transformation, encoding efficiency can be improved. Representative application examples include inter-prediction based on virtual reference frames generated using deep learning models and in-loop filters based on image restoration models.

[0007] Meanwhile, when processing input blocks of a neural network-based filter, the original luma block information can be maintained by increasing the number of channels in the luma block instead of reducing the spatial resolution of the luma block by the size of the chroma block. However, the process of increasing the number of luma channels may not utilize the correlation between the boundaries of the segmented luma channels. Therefore, to enhance video encoding efficiency and improve video quality, a method for effectively processing input blocks when utilizing a neural network-based filter needs to be considered.

[0008] The present disclosure aims to provide a video coding method and device that, when processing an input block of a neural network-based filter, converts a luma block into frequency components and reduces the spatial resolution of the converted luma block to the size of a chroma block while increasing the number of luma channels.

[0009] According to an embodiment of the present disclosure, a method for applying an in-loop filter to a current block, performed by a video decoding device, comprises the steps of: decoding parameters of the in-loop filter from a bitstream, wherein the in-loop filter is pre-trained based on deep learning; obtaining a reconstruction block of the current block, wherein the reconstruction block includes a luma block and a chroma block, and the luma block has a size of W×H, where W and H represent a width and a height of the luma block, respectively; applying a transformation to the luma block to generate transformed luma blocks whose spatial resolution is decomposed; generating stacked blocks based on the transformed luma blocks and the chroma blocks; and inputting the stacked blocks to the in-loop filter to perform filtering, wherein the step of performing the filtering applies different convolution operations to the transformed luma blocks and the chroma blocks.

[0010] According to another embodiment of the present disclosure, a method for applying an in-loop filter to a current block, performed by a video encoding device, comprises the steps of: obtaining parameters of the in-loop filter, wherein the in-loop filter is pre-trained based on deep learning; obtaining a reconstruction block of the current block, wherein the reconstruction block includes a luma block and a chroma block, and the luma block has a size of W×H, where W and H represent a width and a height of the luma block, respectively; applying a transformation to the luma block to generate transformed luma blocks whose spatial resolution is decomposed; generating stacked blocks based on the transformed luma blocks and the chroma blocks; inputting the stacked blocks to the in-loop filter to perform filtering; and encoding the parameters of the in-loop filter, wherein the step of performing the filtering applies different convolution operations to the transformed luma blocks and the chroma blocks.

[0011] According to another embodiment of the present disclosure, a method for providing video data to a video decoding device comprises: encoding the video data into a bitstream; and transmitting the bitstream to the video decoding device, wherein the encoding the video data comprises: obtaining parameters of the in-loop filter, wherein the in-loop filter is pre-trained based on deep learning; obtaining a reconstruction block of the current block, wherein the reconstruction block includes a luma block and a chroma block, and the luma block has a size of W×H, where W and H represent a width and a height of the luma block, respectively; applying a transformation to the luma block to generate transformed luma blocks whose spatial resolution is decomposed; generating stacked blocks based on the transformed luma blocks and the chroma block; inputting the stacked blocks to the in-loop filter to perform filtering; And a step of encoding parameters of a loop filter, wherein the step of performing the filtering applies different convolution operations to the transformed luma blocks and the chroma blocks.

[0012] As described above, according to the present embodiment, when processing an input block of a neural network-based filter, a video coding method and device are provided that converts a luma block into frequency components and reduces the spatial resolution of the converted luma block to the size of a chroma block while increasing the number of luma channels, thereby making it possible to improve video encoding efficiency and enhance video quality.

[0013] In addition, according to the present embodiment, by providing a video coding method and device for efficiently processing an input block in a neural network-based filter, it is possible to reduce the burden on the network based on bit rate reduction in various contents such as UHD (Ultra High Definition) video, game broadcasting, 360-degree video streaming, VR / AR (Virtual Reality / Augmented Reality) video, online lectures, etc., and to reduce energy consumption for a video playback device.

[0014] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.

[0015] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.

[0016] FIGS. 3A and 3B are diagrams illustrating multiple intra prediction modes, including wide-angle intra prediction modes.

[0017] Figure 4 is an example diagram of the surrounding blocks of the current block.

[0018] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.

[0019] Figure 6 is an example diagram showing a multilayer neural network as a deep learning-based neural network.

[0020] Figure 7 is an example diagram showing the operation of a convolution layer.

[0021] Figure 8 is an example diagram showing a SISR (Single Image Super Resolution) network.

[0022] Figure 9 is an example diagram showing a residual block used in SISR.

[0023] Figure 10 is an example diagram showing a fixed coefficient in-loop filter based on a CNN (Convolutional Neural Network).

[0024] Figure 11 is an example diagram conceptually representing a LOP (Low-complexity Operating Point) in-loop filter.

[0025] FIG. 12 is an exemplary diagram showing an input portion of a neural network-based filter according to one embodiment of the present disclosure.

[0026] FIG. 13 is an exemplary diagram showing an input portion of a neural network-based filter according to one embodiment of the present disclosure.

[0027] FIG. 14 is an exemplary diagram showing the disassembly of a checkerboard according to one embodiment of the present disclosure.

[0028] FIG. 15 is an exemplary diagram illustrating a group convolution operation according to one embodiment of the present disclosure.

[0029] FIG. 16 is a flowchart illustrating a method for controlling neural network-based video tools performed by a video encoding device according to one embodiment of the present disclosure.

[0030] FIG. 17 is a flowchart illustrating a method for controlling neural network-based video tools performed by a video decoding device according to one embodiment of the present disclosure.

[0031] Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.

[0032] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to the illustration in FIG. 1.

[0033] The video encoding device may be configured to include a picture segmentation unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0034] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0035] A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). And, information commonly referenced by one or more SPS is encoded in a Video Parameter Set (VPS). In addition, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.

[0036] The picture segmentation unit (110) determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as the syntax of SPS or PPS and transmitted to the image decoding device.

[0037] The picture segmentation unit (110) divides each picture constituting an image into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a CU, which is a basic unit of encoding.

[0038] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).

[0039] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.

[0040] As illustrated in FIG. 2, a CTU may first be split into a QT structure. The quadtree splitting may be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further split into one or more of the BT structure or the TT structure. In the BT structure and / or the TT structure, multiple splitting directions may exist. For example, there may be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As illustrated in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0041] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the CU split flag (split_cu_flag) value indicates that the node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is a basic unit of encoding. If the CU split flag (split_cu_flag) value indicates that the node is split, the video encoding device starts encoding from the first flag in the above-described manner.

[0042] As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.

[0043] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.

[0044] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0045] In general, each current block within a picture can be predictively coded. Prediction of the current block can typically be performed using either intra-prediction (using data from the picture containing the current block) or inter-prediction (using data from a picture coded before the picture containing the current block). Inter-prediction encompasses both unidirectional and bidirectional prediction.

[0046] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3a, the multiple intra prediction modes may include two non-directional modes including the Planar mode and the DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently depending on each prediction mode.

[0047] For efficient directional prediction for a rectangular current block, directional modes (intra prediction modes 67 to 80 and -1 to -14) indicated by dotted arrows in Fig. 3b may be additionally used. These may be referred to as "wide-angle intra-prediction modes." In Fig. 3b, the arrows point to corresponding reference samples used for prediction, and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide-angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra-prediction modes, some wide-angle intra-prediction modes available for the current block may be determined based on the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (intra prediction modes 67 to 80) having an angle less than 45 degrees are available when the current block is a rectangular shape whose height is smaller than its width, and wide-angle intra prediction modes (intra prediction modes -1 to -14) having an angle greater than -135 degrees are available when the current block is a rectangular shape whose width is larger than its height.

[0048] The intra prediction unit (122) can determine an intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate bit-rate distortion values ​​using rate-distortion analysis for multiple tested intra prediction modes and select an intra prediction mode with the best bit-rate distortion characteristics among the tested modes.

[0049] The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0050] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma component and the chroma component. The motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0051] The inter prediction unit (124) may perform interpolation on a reference picture or a reference block to improve prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision or resolution of the motion vector can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.

[0052] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the entropy encoding unit (155). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.

[0053] Various methods can be used to minimize the number of bits required to encode motion information.

[0054] For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the image decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.

[0055] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

[0056] As the surrounding blocks for deriving merge candidates, all or part of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the block at the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.

[0057] The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, the merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the video decoding device.

[0058] Merge Skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high encoding efficiency can be achieved for low-motion images, still images, and screen content images.

[0059] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0060] Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.

[0061] In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block in the current picture as shown in FIG. 4 may be used. In addition, a block located in a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block co-located with the current block in the reference picture or blocks adjacent to the block in the co-located block may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.

[0062] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.

[0063] The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information for identifying the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.

[0064] Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.

[0065] The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.

[0066] The transformation unit (140) transforms residual signals in a residual block having pixel values ​​in a spatial domain into transform coefficients in a frequency domain. The transformation unit (140) may transform the residual signals in the residual block using the entire size of the residual block as a transformation unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transformation units to perform the transformation. Alternatively, the residual signals may be transformed using only the transformation domain sub-block as a transformation unit by dividing the sub-blocks into two sub-blocks, that is, a transformation domain and a non-transform domain. Here, the transformation domain sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only a sub-block has been converted, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. In addition, the size of the conversion area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0067] Meanwhile, the transformation unit (140) can individually perform transformations on the residual block in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation function pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation function pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0068] The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.

[0069] The rearrangement unit (150) can perform rearrangement of coefficient values ​​for quantized residual values.

[0070] The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region by using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, the scanning method to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan.

[0071] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0072] In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about the prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about the intra prediction mode) or inter prediction information (information about the encoding mode of motion information (merge mode or AMVP mode), a merge index in the case of the merge mode, and a reference picture index and a differential motion vector in the case of the AMVP mode) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about quantization parameters and information about a quantization matrix.

[0073] The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0074] An adder (170) adds the restored residual block and the predicted block generated by the prediction unit (120) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting the next block.

[0075] The loop filter unit (180) performs filtering on restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filter unit (180) may include all or part of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186) as an in-loop filter.

[0076] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter (184) and the ALF (186) perform additional filtering on the deblocking-filtered image. The SAO filter (184) and the ALF (186) are filters used to compensate for the differences between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in units of CTUs. In contrast, the ALF (186) performs block-based filtering, and compensates for distortion by applying different filters by distinguishing the edges and degrees of variation of the corresponding block. Information on filter coefficients to be used in the ALF can be encoded and signaled to an image decoding device.

[0077] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0078] The video encoding device can store the bitstream of encoded video data on a non-transitory storage medium or transmit it to the video decoding device using a communication network.

[0079] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.

[0080] The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

[0081] Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0082] The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information, information on residual signals, etc. required to restore the current block.

[0083] The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.

[0084] For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (mtt_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Accordingly, each node below the leaf nodes of QT are recursively split into the BT or TT structure.

[0085] As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.

[0086] As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.

[0087] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded by using the division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.

[0088] Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about residual signals.

[0089] The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.

[0090] The inverse quantization unit (520) inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.

[0091] The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.

[0092] In addition, when the inverse transform unit (530) inversely transforms only a portion of a transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only a sub-block of the transform block has been transformed, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block, and inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to restore residual signals, and fills “0” values ​​with residual signals for areas that have not been inversely transformed, thereby generating a final residual block for the current block.

[0093] In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions using MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.

[0094] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.

[0095] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0096] The inter prediction unit (544) uses the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.

[0097] An adder (550) adds the residual block output from the inverse transform unit (530) and the prediction block output from the inter prediction unit (544) or the intra prediction unit (542) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting a block to be decoded later.

[0098] The loop filter unit (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as in-loop filters. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) and the ALF (566) perform additional filtering on restored blocks after deblocking filtering to compensate for differences between restored pixels and original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about filter coefficients decoded from the non-stream.

[0099] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0100] The present embodiment relates to encoding and decoding of images (videos) as described above. More specifically, a video coding method and device are provided, which, when processing an input block of a neural network-based filter, converts a luma block into frequency components, reduces the spatial resolution of the converted luma block to the size of a chroma block, and increases the number of luma channels.

[0101] The following embodiments may be performed by a video encoding apparatus. Additionally, the following embodiments may be performed by a video decoding apparatus.

[0102] The video encoding device can generate signaling information related to the present embodiment in terms of rate distortion optimization in encoding the current block. The video encoding device can encode the signaling information using the entropy encoding unit (155) and then transmit it to the video decoding device. The video decoding device can decode the signaling information related to the decoding of the current block from the bitstream using the entropy decoding unit (510).

[0103] In the following description, the term "target block" may be used interchangeably with the current block or coding unit (CU). Alternatively, the term "target block" may also refer to a portion of a coding unit.

[0104] Also, a value of a flag being true indicates that the flag is set to 1. Also, a value of a flag being false indicates that the flag is set to 0.

[0105] The decoder side includes all or part of the inverse quantization unit (160), the inverse transform unit (165), the prediction unit (120), the adder (170), the loop filter unit (180), and the memory (190) in the video encoding device illustrated in FIG. 1. Alternatively, the decoder side includes all or part of the inverse quantization unit (520), the inverse transform unit (530), the prediction unit (540), the adder (550), the loop filter unit (560), and the memory (570) in the video decoding device illustrated in FIG. 5. With respect to a series of decoding processes, the decoder side of the video encoding device and the decoder side of the video decoding device perform the same operation. The video encoding device determines information related to the operation of the decoder side and signals the determined information to the video decoding device. The video decoding device can decode the signaled information and operate the decoder side based on the decoded information.

[0106] I-1. Multilayer Perceptron (MLP)

[0107] A multilayer perceptron (MLP) is composed of multiple neurons and edges connecting the neurons, as shown in the example in Figure 6. In addition to an input layer and an output layer, an MLP may include one or more hidden layers. Each hidden layer includes one or more hidden units (or hidden nodes). Additionally, different weights may be assigned to each edge. In addition, an activation function may be used in the process of propagating output values ​​from one layer to the next. Representative activation functions include the sigmoid function, the tangent hyperbolic function, or the Relu (Rectified Linear Unit) function.

[0108] As illustrated in the example of Fig. 6, a two-layer MLP with a forward network structure passes an input vector x through an intermediate hidden unit to provide an output vector y. In the aforementioned forward network structure, any node value located in the output layer can be expressed as in mathematical equation 1.

[0109]

[0110] In mathematical expression 1, class is a weight matrix, corresponding to the edges connecting the input vectors and hidden units, and the edges connecting the hidden units and the output vectors, respectively. w ji (1) and w kj (2) are each class Indicates the component of y k represents the components of the output vector y. x0 and z0 represent the bias units of the input layer and the hidden layer, respectively. Therefore, D is the dimension of the input vector x, M represents the number of hidden units, and K is the dimension of the output vector y. In addition, h and σ are the activation functions applied to the hidden layer and the output layer, respectively. Meanwhile, the activation function is not necessarily applied to the output of the hidden layer and the output layer. If the activation function is not used, mathematical expression 1 becomes class It includes an operation corresponding to the weight matrix multiplication between the two.

[0111] The process of calculating weights based on training data and labels is called training. Training can typically be performed using the stochastic gradient descent (SGD) algorithm, which utilizes backpropagation. The process of calculating outputs in a feed-forward manner using the weights calculated through training is called inference or testing.

[0112] Hereinafter, multilayer neural networks, MLPs, or forward-facing networks can be used interchangeably.

[0113] I-2. CNN(Convolutional Neural Network)

[0114] CNN, a neural network comprised of multiple convolutional layers and pooling layers, is a deep learning technique known to be ideal for image processing. Convolutional layers extract feature maps (also known as "features" or "characteristics") using multiple kernels or filters. The kernel coefficients that make up the filters are parameters determined during the learning process.

[0115] Among the convolutional layers of CNN, the front layer closer to the input extracts feature maps that respond to simple, low-level image features such as lines, points, or surfaces, while the back layer closer to the output extracts feature maps that respond to higher-level features such as textures and object parts.

[0116] FIG. 7 is an exemplary diagram showing the operation of a convolutional layer according to one embodiment of the present disclosure.

[0117] A convolutional layer generates a feature map from an input image using a convolution operation. The example in Fig. 7 illustrates a kernel (or filter) with a kernel size of 3×3. The kernel size is also referred to as the kernel size or filter size. The kernel has kernel parameters, also called weights. The kernel illustrated in Fig. 7 has a total of nine kernel parameters. The kernel parameters are initially set to arbitrary values, and their values ​​can be updated based on learning.

[0118] The convolution layer performs convolution operations using blocks of the same size as the kernel size in the input image. At this time, the blocks of the same size as the kernel size in the input image are referred to as windows.

[0119] When filtering an input image in raster scan order, the window movement size is called the stride. In the example of Fig. 7, the stride is 1. If the stride is set to 2, the convolution operation is performed by spacing the window by 2 samples, and as a result, the width and height of the feature map become half the width and height of the input image.

[0120] As mentioned above, a single convolutional layer can include multiple filters. The number of filters or kernels is called a channel. In other words, the number of channels is equal to the number of filters. Furthermore, the number of filters determines the dimensionality of the feature map.

[0121] Padding refers to a method of expanding input data by filling the area around it with a specific value before performing a convolution operation. Padding is primarily used to adjust the spatial size of the output data. The padding value can be determined by hyperparameters, but zero-padding is commonly used. Without padding, the spatial size of the output data decreases with each convolutional layer, potentially causing boundary information to disappear. Therefore, padding is used to prevent this problem. Specifically, padding can be used to match the spatial sizes of the output data from the convolutional layer to the input data.

[0122] The deconvolution layer performs the opposite operation to the convolution layer. It generates the desired data image from the input feature map as output.

[0123] The pooling layer performs pooling, a process of subsampling the feature map generated by the convolutional layer. The pooling layer uses a 2×2 window to select samples so that the output is half the width and height of the input. In other words, the pooling layer is used to reduce the size of the input image or input feature map by condensing a 2×2 region into a single sample.

[0124] The opposite concept of a pooling layer is defined as an unpooling layer. Unpooling layers, in contrast to pooling layers, function to expand the dimensionality and are primarily used after deconvolution layers.

[0125] The convolutional encoder-decoder architecture is a network structure composed of pairs of convolutional layers and deconvolutional layers. The convolutional encoder consists of a convolutional layer and a pooling layer, and outputs a feature map (or feature vector) from the input image. The final output vector of the convolutional encoder is also referred to as a latent vector. The convolutional decoder consists of a deconvolutional layer and an unpooling layer, and generates an output image from the feature map or latent vector.

[0126] The inputs and outputs of a convolutional encoder-decoder can be configured in various ways depending on the application and network purpose. For example, the inputs and outputs can be optical flow maps, saliency maps, image frames, etc.

[0127] Figure 8 is an example diagram showing an SISR network.

[0128] One example of CNN application is Single Image Super Resolution (SISR). The SISR network generates a high-resolution image from a low-resolution input image. As illustrated in Figure 8, the SISR network may include multiple convolutional layers. Each convolutional layer includes an activation function, such as the Rectified Linear Unit (ReLU). The parameters of the SISR network can be trained so that the resulting SR (Super Resolution) image approximates the Ground Truth (GT).

[0129] SR methods using CNN can improve SR performance by increasing the depth (e.g., increasing the number of convolutional layers). To overcome the overfitting problem that may occur in learning due to the increase in depth, a residual block that can perform skip connection and residual learning can be used in the SISR network. The residual block, as illustrated in Fig. 9, is a block that stores input features x l In addition to the path that applies the convolution operation, it includes a skip path. In addition, the residual block outputs x l+1 When generating, a path or skip path for applying convolution operations can also be selected based on learning efficiency. In the example of Fig. 9, the residual block includes a BN (Batch Normalization) layer.

[0130] For example, Enhanced Deep Residual Networks (EDSR) improves network performance by continuously connecting residual blocks to increase their depth. Another example is Accurate Image Super-Resolution Using Very Deep Convolutional Networks (VDSR), a CNN model based on the Visual Geometry Group (VGG) network that uses residual learning, a method that adds residual frames to the final output. VDSR adds the residual signal to the very end of the network, thereby adding it to the input signal.

[0131] As another example, CNN can be used as an in-loop filter in an image encoding device or an image decoding device. In this case, the deep learning-based in-loop filter can be applied to any position within the existing loop filter unit (180, 560) consisting of a deblocking filter, an SAO filter, and an ALF.

[0132] Among the deep learning-based in-loop filters are fixed-coefficient in-loop filters. The fixed-coefficient deep learning in-loop filter uses the same CNN kernel parameters stored on the image encoding and decoding devices.

[0133] Figure 10 is an example diagram showing a CNN-based fixed coefficient in-loop filter.

[0134] A fixed-coefficient in-loop filter may be a noise-removing CNN model including multiple Dense Residual Units (DRUs). The input block (or input frame) passes through a normalized QP map and then is passed to the subsequent stage. The normalized QP map is used to reduce inference errors when quantization noises of different intensities are mixed during the learning and inference processes. The kernel parameters constituting the Dense Residual Units (DRUs) and convolutional layers may be stored identically and used in an image encoding device and an image decoding device. In the example of Fig. 10, each DRU may include all or part of a convolutional layer, a ReLU layer, and a depth-wise separable convolutional (DSC) layer.

[0135] Fixed-coefficient deep learning in-loop filters have the disadvantage of providing general performance for a variety of video frames, which increases the number of layers in the CNN and thus increases the computation time.

[0136] As another example, there is a variable coefficient in-loop filter as a deep learning-based in-loop filter. A video encoding device trains a variable coefficient in-loop filter based on a target video frame and transmits filter parameters according to the training to a video decoding device. The variable coefficient in-loop filter may be a noise-removing CNN model including convolutional layers. As an example, based on an input block (or input frame) generated by packing Y, U, and V blocks, the variable coefficient in-loop filter may output a block from which noise has been removed. Since it is trained based on a target video frame, the kernel parameters can be adaptively configured to the target video. Since a parameter refresh is performed for each I-frame, the kernel parameters can be transmitted to the video decoding device for each I-frame, considering the size of the transmitted parameters.

[0137] The variable coefficient in-loop filter has the disadvantage of requiring double-loop coding because it is adaptively configured for the target video frame. Therefore, considering delay, the variable coefficient in-loop filter has limited applicability to random access scenarios.

[0138] The following embodiments are described with a focus on a video decoding device, but can be implemented in the same or similar manner in a video encoding device. Alternatively, the following embodiments are described with a focus on the decoder side of a video decoding device, but can be implemented in the same or similar manner in a decoder side of a video encoding device.

[0139] Hereinafter, a luma block may include a luma component (or a luma channel), and a chroma block may include a first chroma component (or a first chroma channel) and a second chroma component (or a second chroma channel). For example, (luma component, first chroma component, second chroma component) may be expressed as YUV, YCoCg.

[0140] Hereinafter, the neural network-based filter may be an SR (Super resolution) filter, an in-loop filter, etc. of NNVC (Neural Network-based Video Coding), a branch of the next-generation codec. In NNVC, the in-loop filter refers to an LOP (Low-complexity Operating Point) in-loop filter, a HOP (High-complexity Operating Point) in-loop filter, or a VLOP (Very Low-complexity Operating Point) in-loop filter. In addition, the SR filter refers to an LOP SR filter or a HOP SR filter.

[0141] Hereinafter, a luma block may include a luma component (or a luma channel), and a chroma block may include a first chroma component (or a first chroma channel) and a second chroma component (or a second chroma channel). For example, (luma component, first chroma component, second chroma component) may be expressed as YUV, YCoCg.

[0142] On the input side of a neural network-based filter, blocks or frames can be used as input. A frame can be a combination of blocks.

[0143] The input block of a neural network-based filter includes a (restored) luma block and a chroma block. As described above, the neural network-based filter can be implemented using an MLP, CNN, or the like.

[0144] The color format (i.e., sampling format) can have formats of 4:2;0, 4:2:2, and 4:4;4, depending on the size ratio between the luma block and the chroma block. Here, the 4:2;0 format means that the horizontal / vertical ratio between the luma block and the chroma block is both 2, the 4:2;2 format means that the horizontal / vertical ratio between the luma block and the chroma block is both 2, and the 4:4;4 format means that the horizontal / vertical ratio between the luma block and the chroma block is both 1.

[0145] The following examples are described with a focus on a low-complexity operating point (LOP) in-loop filter, but can be applied equally or similarly to other neural network-based filters.

[0146] Figure 11 is an example diagram conceptually representing a LOP in-loop filter.

[0147] The LOP in-loop filter of the NNVC may include an input unit (1110), a pre-processing unit (1120), a series of backbone blocks (1130), and a post-processing unit (1140). The LOP in-loop filter includes a skip path. Therefore, the main path of the LOP in-loop filter (i.e., including the pre-processing unit (1120), the backbone block (1130), and the post-processing unit (1140)) generates a residual signal of the input components.

[0148] An image encoding device including an LOP in-loop filter may additionally include a training unit (not shown) for training the LOP in-loop filter in addition to the above-described components, or may be implemented in a form that is linked to an external training unit.

[0149] The example of Fig. 12 may correspond to an input portion (1110) of the LOP in-loop filter. As an example, for a video sequence having a 4:2:0 color format, the input blocks (luma blocks and chroma blocks) of the LOP in-loop filter may be processed as in Fig. 12. The video decoding device increases the resolution of the chroma blocks (e.g., U and V blocks) to match the spatial resolution of the luma block (e.g., Y block). In Fig. 12, the Rec block stacks the reconstructed luma block and two chroma channels. The video decoding device may input the stacked channels to the LOP in-loop filter to extract features. In the example of Fig. 12, the width and height of the luma block are represented by W and H, respectively.

[0150] In FIG. 12, d1 is the number of output channels of the convolution layer in the input unit (1110), which may vary depending on the type of neural network-based filter (e.g., SR filter or in-loop filter), complexity (e.g., LOP or HOP), etc.

[0151] The input unit (1110) can obtain a prediction block as an additional input block in addition to the luma / chroma restoration block. In addition, the input unit (1110) can obtain quantization parameters, boundary strength (BS), and / or block shape as additional inputs. Here, the block shape represents one of the IPB (Intra, Bi-predictive, Predictive) blocks.

[0152] The preprocessing unit (1120) is located at the front end of the backbone blocks and combines the input components as described above using a convolutional layer and an activation function. The preprocessing unit (1120) can be implemented as a single path without distinguishing between luma and chroma to combine the input components.

[0153] The backbone block (1130) includes convolutional layers, activation functions, and sparse convolutional layers. The backbone block (1130) can be implemented with two paths to process luma / chroma components. For example, in the example of FIG. 11, N Y is the number of backbone blocks for processing the luma component, and N UV is the number of backbone blocks for processing chroma components. The backbone block (1130) processes preprocessed input components and transmits the processed results to the postprocessing unit (1140).

[0154] The post-processing unit (1140) is located after the backbone blocks and processes the output of the backbone blocks using convolutional layers, sparse convolutional layers, and activation functions to ultimately generate residual signals of luma / chroma components. The post-processing unit (1140) may be implemented with two paths to process the luma / chroma components. The post-processing unit (1140) may apply pixel shuffling to the luma component.

[0155] By adding the input block transmitted along the skip path and the output of the post-processing unit (1140), the LOP in-loop filter can generate an output block.

[0156] The configuration of the LOP in-loop filter as described above can be similarly applied to other neural network-based filters such as the VLOP in-loop filter, the HOP in-loop filter, the LOP SR filter, and the HOP SR filter.

[0157] As mentioned above, neural network-based filters can utilize prediction blocks (including luma prediction blocks and chroma prediction blocks) as input blocks in addition to reconstruction blocks. As another example, neural network-based filters can utilize quantization parameters, block sizes, block shapes, etc., in addition to reconstruction blocks and / or prediction blocks, as input blocks.

[0158] As described above, for a video sequence with a 4:2:0 color format, the luma block and chroma block of the neural network-based filter of NNVC can be processed as shown in FIG. 12. The video decoding device increases the resolution of the chroma block (e.g., U and V blocks) to match the spatial resolution of the luma block (e.g., Y block). In FIG. 12, the Rec block stacks the luma block and two chroma channels. The video decoding device can input the stacked channels to a neural network-based filter to extract features. The above-described method has a disadvantage in that the computational complexity of the neural network-based filter increases as the spatial resolution of the chroma block increases.

[0159] The example in Fig. 12 may correspond to the input unit of a neural network-based filter of NNVC. In Fig. 12, d1 represents the number of output channels of a convolutional layer in the input unit, which may vary depending on the type of filter (e.g., SR filter or in-loop filter), complexity (e.g., VLOP, LOP, or HOP), etc.

[0160] As another example, for a video sequence with a 4:2:0 color format, a video decoding device can divide a luma block of size W×H into four blocks of size W / 2×H / 2. The video decoding device can then stack four luma channels and two chroma channels to form six channels, and then input the formed channels to a neural network-based filter. The above-described method does not increase the computational complexity of the neural network-based filter, but may not properly utilize the correlation between the boundaries of the divided luma channels.

[0161] II. Embodiments according to the present disclosure

[0162] In order to address the problem that occurs in the process of processing input blocks of a neural-based filter as described above, the image decoding device according to the present disclosure generates transform coefficients by performing a 2×2 DCT (discrete cosine transform) transform on a luma block of size W×H. As shown in the example of FIG. 13, the image decoding device can group the transform coefficients of the luma block into four frequency components to generate four blocks of size W / 2×H / 2. Here, the four frequency component groups represent LL, LH, HL, and HH components according to the 2×2 DCT transform, where L represents a low-frequency component in the width / height direction, and H represents a high-frequency component in the width / height direction. At this time, in order to maintain the resolution of the chroma block, the image decoding device can sequentially apply upsampling and downsampling to the chroma block. Alternatively, the image decoding device can use the chroma block as is. The video decoding device can stack four luma blocks and two chroma blocks to form six blocks of size W / 2×H / 2, and input the formed blocks to a neural network-based filter. At this time, the luma block can be processed in the frequency domain, and the chroma block can be processed in the pixel domain.

[0163] As an example, an image decoding device can divide the spatial resolution of a luma block into W / 2×H / 2×4 to generate four blocks, and stack the generated luma blocks and chroma blocks. When processing the stacked blocks in a neural network-based filter, the luma block and chroma block can be computed separately, or the pixel block and frequency component block can be computed separately, thereby improving computational complexity and performance. The processing of the spatial resolution of the luma block can be applied to the SR filter and in-loop filter of the NNVC as described above.

[0164] As another example, instead of performing a 2×2 DCT transform, an image decoding device can apply a 2×2 wavelet transform to a W×H luma block to generate four blocks of size W / 2×H / 2. For example, the Haar (wavelet) transform is used. The Haar transform can analyze a pixel area at multiple resolutions or scales and decompose frequency components together while maintaining spatial domain information of the divided blocks.

[0165] As another example, instead of using a 2×2 transform function, an image decoding device can generate four blocks of size W / 2×H / 2 from a W×H luma block using a 2×2 checkerboard decomposition. For example, as in the example of FIG. 14, pixels at odd indices horizontally and vertically can be configured as a first channel, and pixels at odd indices horizontally and even indices vertically can be configured as a second channel. Additionally, pixels at even indices horizontally and odd indices vertically can be configured as a third channel, and pixels at even indices horizontally and vertically can be configured as a fourth channel.

[0166] As another example, instead of using a 2×2 transform function, an image decoding device can generate four blocks of size W / 2×H / 2 from a W×H luma block using a 2×2 convolution kernel. In this case, the convolution kernel can be trained together with other parameters of a neural network-based filter.

[0167] As another example, instead of using a 2×2 transform function, the image decoding device can generate 16 blocks of size W / 4×H / 4 from a W×H luma block using a 4×4 transform function. In addition, the image decoding device can generate 8 blocks of size W / 4×H / 4 (i.e., 4 for each chroma component) from a W / 2×H / 2 chroma block using a 2×2 transform function. The image decoding device can generate the input of a neural network-based filter by stacking 24 blocks in the channel direction. Generalizing the above, an N×M transform function can be used. In this case, in addition to the above-described DCT transform and wavelet transform, various transform methods such as DST (discrete sine transform) transform, non-separable transform, etc. can be used. By generating input blocks grouped by frequency components based on the method according to the present disclosure, the image decoding device can maintain / reflect the overall characteristics and similarity of the image.

[0168] As another example, instead of using a 2×2 checkerboard decomposition, a video decoder can use a 4×4 checkerboard decomposition to generate 16 blocks of size W / 4×H / 4 from a W×H luma block. Additionally, the video decoder can use a 2×2 checkerboard decomposition to generate 8 blocks of size W / 4×H / 4 (i.e., 4 for each chroma component) from a W / 2×H / 2 chroma block.

[0169] The generalized N×M transform can be applied to various color formats of video. For example, for a video sequence with a 4:4:4 color format, a video decoding device can divide the spatial resolution as follows.

[0170] As an example, the video decoding device can apply a 2×2 DCT transform to a W×H luma block to generate four luma blocks of size W / 2×H / 2. Additionally, the video decoding device can apply a 2×2 DCT transform to a W×H chroma block to generate eight chroma blocks of size W / 2×H / 2 (i.e., four for each chroma component).

[0171] As another example, the video decoding device can apply a 2×2 wavelet transform to a W×H luma block to generate four luma blocks of size W / 2×H / 2. Additionally, the video decoding device can apply a 2×2 wavelet transform to a W×H chroma block to generate eight chroma blocks of size W / 2×H / 2 (i.e., four for each chroma component).

[0172] As another example, a video decoding device may segment a luma block using checkerboard decomposition. Furthermore, the video decoding device may segment a chroma block using the same checkerboard decomposition.

[0173] As another example, instead of applying the same transform to the luma / chroma blocks, different transforms can be applied. For example, a video decoding device can apply a 2×2 DCT transform to a W×H luma block to generate four luma blocks of size W / 2×H / 2. In this case, the video decoding device can apply a 2×2 checkerboard decomposition to a W×H chroma block to generate eight chroma blocks of size W / 2×H / 2 (i.e., four for each chroma component).

[0174] Meanwhile, for a video sequence with a 4:2:2 color format, the video decoding device can divide the spatial resolution as follows:

[0175] As an example, the video decoding device can apply a 2×2 DCT transform to a W×H luma block to generate four luma blocks of size W / 2×H / 2. Additionally, the video decoding device can apply a 1×2 DCT transform to a W / 2×H chroma block to generate four chroma blocks of size W / 2×H / 2 (i.e., two for each chroma component).

[0176] As another example, the video decoding device can apply a 2×2 wavelet transform to a W×H luma block to generate four luma blocks of size W / 2×H / 2. Additionally, the video decoding device can apply a 1×2 wavelet transform to a W / 2×H chroma block to generate four chroma blocks of size W / 2×H / 2 (i.e., two for each chroma component).

[0177] As another example, a video decoding device may segment a luma block using checkerboard decomposition. Furthermore, the video decoding device may segment a chroma block using checkerboard decomposition.

[0178] Meanwhile, segmentation of the luma block as described above may be applied to the luma reconstruction block and / or the luma prediction block. For example, for inputs such as quantization parameters, BS, and block shapes, DCT, wavelet transform, or checkerboard decomposition may not be applied for segmentation of the luma component.

[0179] In FIG. 12 or FIG. 13, the image decoding device processes the input block, i.e., the stacked luma block and chroma block, in the channel direction based on a convolution operation. That is, the convolution operation processes the luma block and chroma block in the channel direction regardless of the components of the input block. At this time, if filtering is performed in the channel direction in the convolution operation, there is a disadvantage in that signals in physically different areas (e.g., pixel area and frequency area) are mixed and operated. To address the above-mentioned problem, in the present disclosure, the convolution operation may be performed as follows.

[0180] As an example, for an input in which four luma blocks of size W / 2×H / 2 and two chroma blocks of size W / 2×H / 2 are stacked, an image decoding device can perform a group convolution operation so that components are not mixed between channels. As shown in FIG. 15, the channel depth of the convolution of group 1 is set to 4 (C1=4) corresponding to the number of channels of the luma block, and the channel depth of the convolution of group 2 is set to 2 (C2=2) corresponding to the number of channels of the chroma block. In FIG. 15, M represents the size of the convolution kernel (or filter) and can be any positive integer value.

[0181] Alternatively, instead of group convolution, the video decoding device can re-divide the input luma and chroma blocks within a filter at the rear of the input and apply the divided blocks to different convolution operations. The video decoding device can then generate the final output of the neural network-based filter by stacking or combining the computed results.

[0182] As another example, for an input that stacks four luma blocks of size W / 2×H / 2 and two chroma blocks of size W / 2×H / 2, the video decoding device can transform the chroma blocks using the same method so that the components are not mixed between the channels. For example, the video decoding device can analyze the frequency components of the chroma blocks using a 2×2 DCT transform function or a wavelet transform function as the transform function. The video decoding device can generate eight chroma blocks of size W / 4×H / 4 (i.e., four for each chroma component), and upsample the components by a factor of 2 again to generate eight chroma blocks of size W / 2×H / 2 (i.e., four for each chroma component).

[0183] As another example, when a luma block includes values ​​of pixel components and frequency components, the above can be similarly applied. The image decoding device first downsamples a luma block of size W×H to generate a luma block of size W / 2×H / 2 including pixel components. At the same time, a 2×2 DCT transform function or a wavelet transform function is applied to the luma block of size W×H to group frequency components, thereby generating four luma blocks of size W / 2×H / 2. The image decoding device can stack the luma block of size W / 2×H / 2 including pixel components and the four luma blocks of size W / 2×H / 2 including frequency components and use them as luma inputs of a neural network-based filter. At this time, one convolution operation can be applied to the luma block of size W / 2×H / 2×5, or, as described above, a group convolution operation having channel direction sizes of 1 and 4 can be used.

[0184] Below, an example of applying the above to an in-loop filter or SR filter of NNVC is described.

[0185] As shown in Fig. 13, for a video sequence with a 4:2:0 color format, the input of the NNVC in-loop filter stacks a luma block and an upsampled chroma block, i.e., three channels. The input of the LOP in-loop filter inputs the stacked three channels to a convolutional layer to extract 12 channels (d1 = 12) as features of the video block. The input of the VLOP in-loop filter extracts four channels (d1 = 4) as features of the video block in a similar manner. In addition, the input of the HOP in-loop filter extracts 192 channels (d1 = 192) as features of the video block in a similar manner.

[0186] Also, as shown in Fig. 13, the input of the NNVC SR filter stacks a luma block and an upsampled chroma block, i.e., three channels. The input of the LOP SR filter inputs the three stacked channels to a convolutional layer to extract 32 channels (d1 = 32) as features of the video block. In addition, the input of the HOP SR filter extracts 192 channels (d1 = 192) as features of the video block in a similar manner.

[0187] Below, the input section of the in-loop filter, the input section of the SR filter, and the input section can be used interchangeably.

[0188] For example, instead of stacking a luma block and an upsampled chroma block, i.e., three channels, as described above, the input unit applies a 2×2 DCT or wavelet transform to the luma block to generate four luma blocks of size W / 2×H / 2. The input unit stacks four luma blocks of size W / 2×H / 2 and two chroma blocks of size W / 2×H / 2 to generate a block of size W / 2×H / 2×6. The input unit can input the stacked blocks to a convolution layer to perform a convolution operation. At this time, a kernel operation of (M×M×6) can be used. Alternatively, as described above, in order to prevent signals of physically different areas from being mixed, the input unit can perform a group convolution operation using kernels of (M×M×4) and (M×M×2).

[0189] As another example, the input unit applies checkerboard decomposition to a luma block to generate four luma blocks of size W / 2×H / 2. The input unit stacks four luma blocks of size W / 2×H / 2 and two chroma blocks of size W / 2×H / 2 to generate a block of size W / 2×H / 2×6. The input unit can input the stacked blocks to a convolution layer to perform a convolution operation.

[0190] As another example, the input unit applies a 2×2 DCT or wavelet transform to a chroma block to generate eight chroma blocks of size W / 4×H / 4 (i.e., four for each chroma component). The input unit applies upsampling to the chroma block of size W / 4×H / 4 to generate eight chroma blocks of size W / 2×H / 2 (i.e., four for each chroma component). The input unit applies a 2×2 DCT or wavelet transform to a luma block to generate four luma blocks of size W / 2×H / 2. The input unit stacks four luma blocks of size W / 2×H / 2 and eight chroma blocks of size W / 2×H / 2 to generate a block of size W / 2×H / 2×12. The input unit can input the stacked blocks to a convolution layer to perform a convolution operation.

[0191] As mentioned above, depending on the type and complexity of the filter (e.g., VLOP, LOP or HOP), the number of channels generated by the convolutional layer may vary.

[0192] As an example, based on the result of the convolution operation generated by the input unit, a neural network-based in-loop filter can perform filtering to remove artifacts present in the input block.

[0193] As an example, based on the result of the convolution operation generated by the input unit, a neural network-based SR filter can increase the resolution of the input block.

[0194] Below, using the cities of FIGS. 16 and 17, a method of applying a neural network-based filter to the current block is described.

[0195] FIG. 16 is a flowchart illustrating a method of applying a neural network-based filter to a current block, performed by an image encoding device, according to one embodiment of the present disclosure.

[0196] The image encoding device obtains parameters of a neural network-based filter (S1600).

[0197] As mentioned above, neural network-based filters include SR filters, in-loop filters, etc. Neural network-based filters can be pre-trained using deep learning.

[0198] The video encoding device obtains a restoration block of the current block (S1602). Here, the restoration block includes a luma block and a chroma block. The luma block has a size of W×H, where W and H represent the width and height of the luma block, respectively.

[0199] The size of a chroma block can be determined based on the color format of the restoration block. If the color format is 4:2:0, the chroma block has a size of W / 2×H / 2. If the color format is 4:2:2, the chroma block has a size of W / 2×H. If the color format is 4:4:4, the chroma block has a size of W×H.

[0200] The video encoding device applies a transformation to a luma block to generate transformed luma blocks with decomposed spatial resolution (S1604).

[0201] As an example, an image encoding device may apply a 2×2 frequency transform to a luma block to generate transform coefficients, and group the transform coefficients according to frequency components to generate transformed luma blocks having a size of W / 2×H / 2. At this time, the frequency transform may be DCT, wavelet transform, DST, non-separable transform, etc.

[0202] As another example, a video encoding device can apply a 2×2 checkerboard decomposition to a luma block to generate transformed luma blocks having a size of W / 2×H / 2. The 2×2 checkerboard decomposition decomposes the pixels of the luma block into a 2×2 region according to the horizontal and vertical positions of each pixel.

[0203] When the color format of the restoration block is 4:4:4, the video encoding device can apply a 2×2 frequency transform to the chroma block to generate transformed chroma blocks having a size of W / 2×H / 2.

[0204] As another example, when the color format of the restoration block is 4:4:4, the video encoding device can apply different transformations to the chroma blocks to generate transformed chroma blocks. For example, the video encoding device can apply a 2×2 DCT transformation to a W×H luma block to generate transformed luma blocks having a size of W / 2×H / 2. In this case, the video encoding device can apply a 2×2 checkerboard decomposition to the W×H chroma block to generate transformed chroma blocks having a size of W / 2×H / 2.

[0205] When the color format of the restoration block is 4:2:2, the video encoding device can apply a 1×2 frequency transform to the chroma block to generate transformed chroma blocks having a size of W / 2×H / 2.

[0206] As another example, a video encoding device can apply a 4×4 frequency transform to a luma block to generate transformed luma blocks having a size of W / 4×H / 4. When the color format of the restoration block is 4:2:0, the video encoding device can apply a 2×2 frequency transform to a chroma block to generate transformed chroma blocks having a size of W / 4×H / 4.

[0207] The video encoding device generates stacked blocks based on the transformed luma blocks and chroma blocks (S1606).

[0208] When the color format of the restoration block is 4:2:0, the video encoding device can generate stacked blocks using transformed luma blocks having a size of W / 2×H / 2 and chroma blocks having a size of W / 2×H / 2.

[0209] As another example, when the color format of the restoration block is 4:2:0, the video encoding device can generate stacked blocks using transformed luma blocks having a size of W / 4×H / 4 and transformed chroma blocks having a size of W / 4×H / 4.

[0210] When the color format of the restoration block is 4:2:2, the video encoding device can generate stacked blocks using transformed luma blocks having a size of W / 2×H / 2 and transformed chroma blocks having a size of W / 2×H / 2.

[0211] When the color format of the restoration block is 4:4:4, the video encoding device can generate stacked blocks using transformed luma blocks having a size of W / 2×H / 2 and transformed chroma blocks having a size of W / 2×H / 2.

[0212] The image encoding device inputs the stacked blocks into a neural network-based filter to perform filtering (S1608).

[0213] The video encoding device may apply a single convolution operation to the transformed luma blocks and chroma blocks. Alternatively, the video encoding device may apply different convolution operations to the transformed luma blocks and chroma blocks.

[0214] As an example, the image encoding device can perform filtering to remove artifacts present in the restored block using a neural network-based in-loop filter.

[0215] As an example, an image encoding device can increase the resolution of a restoration block using a neural network-based SR filter.

[0216] The image encoding device encodes parameters of a neural network-based filter (S1610).

[0217] The video encoding device can transmit a bitstream including encoded parameters to the video decoding device.

[0218] FIG. 17 is a flowchart illustrating a method of applying a neural network-based filter to a current block, performed by an image decoding device according to one embodiment of the present disclosure.

[0219] The video decoding device decodes parameters of a neural network-based filter from a bitstream (S1700).

[0220] As mentioned above, neural network-based filters include SR filters, in-loop filters, etc. Neural network-based filters can be pre-trained using deep learning.

[0221] The video decoding device obtains a restoration block of the current block (S1702). Here, the restoration block includes a luma block and a chroma block. The luma block has a size of W×H, where W and H represent the width and height of the luma block, respectively.

[0222] The video decoding device applies a transformation to a luma block to generate transformed luma blocks with decomposed spatial resolution (S1704).

[0223] As an example, an image decoding device may apply a 2×2 frequency transform to a luma block to generate transform coefficients, and group the transform coefficients according to frequency components to generate transformed luma blocks having a size of W / 2×H / 2. At this time, the frequency transform may be DCT, wavelet transform, DST, non-separable transform, etc.

[0224] As another example, an image decoding device can apply a 2×2 checkerboard decomposition to a luma block to generate transformed luma blocks having a size of W / 2×H / 2. The 2×2 checkerboard decomposition decomposes the pixels of the luma block into a 2×2 region according to the horizontal and vertical positions of each pixel.

[0225] When the color format of the restoration block is 4:4:4, the video decoding device can apply a 2×2 frequency transform to the chroma block to generate transformed chroma blocks having a size of W / 2×H / 2.

[0226] As another example, when the color format of the restoration block is 4:4:4, the video decoding device can apply different transformations to the chroma blocks to generate transformed chroma blocks. For example, the video decoding device can apply a 2×2 DCT transformation to a W×H luma block to generate transformed luma blocks having a size of W / 2×H / 2. In this case, the video decoding device can apply a 2×2 checkerboard decomposition to the W×H chroma block to generate transformed chroma blocks having a size of W / 2×H / 2.

[0227] When the color format of the restoration block is 4:2:2, the video decoding device can apply a 1×2 frequency transform to the chroma block to generate transformed chroma blocks having a size of W / 2×H / 2.

[0228] As another example, the video decoding device can apply a 4×4 frequency transform to a luma block to generate transformed luma blocks having a size of W / 4×H / 4. When the color format of the restoration block is 4:2:0, the video decoding device can apply a 2×2 frequency transform to a chroma block to generate transformed chroma blocks having a size of W / 4×H / 4.

[0229] The video decoding device generates stacked blocks based on the transformed luma blocks and chroma blocks (S1706).

[0230] When the color format of the restoration block is 4:2:0, the image decoding device can generate stacked blocks using transformed luma blocks having a size of W / 2×H / 2 and chroma blocks having a size of W / 2×H / 2.

[0231] As another example, when the color format of the restoration block is 4:2:0, the video decoding device can generate stacked blocks using transformed luma blocks having a size of W / 4×H / 4 and transformed chroma blocks having a size of W / 4×H / 4.

[0232] When the color format of the restoration block is 4:2:2, the image decoding device can generate stacked blocks using transformed luma blocks having a size of W / 2×H / 2 and transformed chroma blocks having a size of W / 2×H / 2.

[0233] When the color format of the restoration block is 4:4:4, the image decoding device can generate stacked blocks using transformed luma blocks having a size of W / 2×H / 2 and transformed chroma blocks having a size of W / 2×H / 2.

[0234] The image decoding device inputs the stacked blocks into a neural network-based filter to perform filtering (S1708).

[0235] The video decoding device may apply a single convolution operation to the transformed luma blocks and chroma blocks. Alternatively, the video decoding device may apply different convolution operations to the transformed luma blocks and chroma blocks.

[0236] As an example, the image decoding device can perform filtering to remove artifacts present in the restored block using a neural network-based in-loop filter.

[0237] As an example, an image decoding device can increase the resolution of a restored block using a neural network-based SR filter.

[0238] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0239] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.

[0240] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0241] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0242]

[0243]

[0244] CROSS-REFERENCE TO RELATED APPLICATION

[0245] This patent application claims priority to Korean patent application No. 10-2024-0031037, filed in Korea on March 5, 2024, and Korean patent application No. 10-2025-0023628, filed in Korea on February 24, 2025, the entire contents of which are incorporated herein by reference.

Claims

1. In a method for applying an in-loop filter to a current block, performed by a video decoding device, A step of decoding parameters of the in-loop filter from a bitstream, wherein the in-loop filter is pre-trained based on deep learning; A step of obtaining a restoration block of the current block, wherein the restoration block includes a luma block and a chroma block, the luma block has a size of W×H, and W and H represent the width and height of the luma block, respectively; A step of applying a transformation to the above luma block to generate transformed luma blocks with decomposed spatial resolution; generating stacked blocks based on the above-described converted luma blocks and the above-described chroma blocks; and A step of inputting the above-mentioned stacked blocks into the above-mentioned in-loop filter to perform filtering. Including, but not limited to, The steps for performing the above filtering are: A method for applying different convolution operations to the above-described transformed luma blocks and the above-described chroma blocks.

2. In paragraph 1, The step of generating the above-mentioned converted luma blocks is: A step of generating transformation coefficients by applying a 2×2 frequency transformation to the above luma block; and A step of grouping the above transformation coefficients according to frequency components to generate transformed luma blocks having a size of W / 2×H / 2. A method comprising:

3. In paragraph 2, The above frequency conversion is, A method that is a DCT (Discrete Cosine Transform), a wavelet transform, a DST (Discrete Sine Transform), or a non-separable transform.

4. In paragraph 2, A method further comprising the step of applying a 2×2 frequency transform to the chroma block to generate transformed chroma blocks having a size of W / 2×H / 2 when the color format of the restoration block is 4:4:

4.

5. In paragraph 1, A method further comprising the step of generating transformed chroma blocks by applying a transformation different from the transformation to the chroma block when the color format of the restoration block is 4:4:

4.

6. In paragraph 2, A method further comprising the step of applying a 1×2 frequency transform to the chroma block to generate transformed chroma blocks having a size of W / 2×H / 2 when the color format of the restoration block is 4:2:

2.

7. In paragraph 1, The step of generating the above-mentioned converted luma blocks is: Apply a 2×2 checkerboard decomposition to the above luma block to generate transformed luma blocks having a size of W / 2×H / 2, The above 2x2 checkerboard decomposition is: A method for decomposing pixels of the above luma block into a 2×2 area according to the horizontal and vertical positions of each pixel.

8. In paragraph 1, The step of generating the above-mentioned converted luma blocks is: A method for generating transformed luma blocks having a size of W / 4×H / 4 by applying a 4×4 frequency transform to the above luma block.

9. In paragraph 8, A method further comprising the step of applying a 2×2 frequency transform to the chroma block to generate transformed chroma blocks having a size of W / 4×H / 4 when the color format of the restoration block is 4:2:

0.

10. A method for applying an in-loop filter to a current block, performed by a video encoding device, A step of obtaining parameters of the above in-loop filter, wherein the in-loop filter is pre-trained based on deep learning; A step of obtaining a restoration block of the current block, wherein the restoration block includes a luma block and a chroma block, the luma block has a size of W×H, and W and H represent the width and height of the luma block, respectively; A step of applying a transformation to the above luma block to generate transformed luma blocks with decomposed spatial resolution; A step of generating stacked blocks based on the above-described converted luma blocks and the above-described chroma blocks; A step of inputting the stacked blocks into the in-loop filter to perform filtering; and A step of encoding the parameters of the above in-loop filter Including, but not limited to, The steps for performing the above filtering are: A method for applying different convolution operations to the above-described converted luma blocks and the above-described chroma blocks.

11. In paragraph 10, The step of generating the above-mentioned converted luma blocks is: A step of generating transformation coefficients by applying a 2×2 frequency transformation to the above luma block; and A step of grouping the above transformation coefficients according to frequency components to generate transformed luma blocks having a size of W / 2×H / 2. A method comprising:

12. In paragraph 11, The above frequency conversion is, A method that is a DCT (Discrete Cosine Transform), a wavelet transform, a DST (Discrete Sine Transform), or a non-separable transform.

13. In paragraph 11, A method further comprising the step of applying a 2×2 frequency transform to the chroma block to generate transformed chroma blocks having a size of W / 2×H / 2 when the color format of the restoration block is 4:4:

4.

14. In paragraph 10, The step of generating the above-mentioned converted luma blocks is: Apply a 2×2 checkerboard decomposition to the above luma block to generate transformed luma blocks having a size of W / 2×H / 2, The above 2x2 checkerboard decomposition is: A method for decomposing pixels of the above luma block into a 2×2 area according to the horizontal and vertical positions of each pixel.

15. In a method for providing video data to a video decoding device, A step of encoding the above video data into a bitstream; and A step of transmitting the above bitstream to the image decoding device Including, The step of encoding the above video data is: A step of obtaining parameters of the above in-loop filter, wherein the in-loop filter is pre-trained based on deep learning; A step of obtaining a restoration block of the current block, wherein the restoration block includes a luma block and a chroma block, the luma block has a size of W×H, and W and H represent the width and height of the luma block, respectively; A step of applying a transformation to the above luma block to generate transformed luma blocks with decomposed spatial resolution; A step of generating stacked blocks based on the above-described converted luma blocks and the above-described chroma blocks; A step of inputting the stacked blocks into the in-loop filter to perform filtering; and A step of encoding the parameters of the above in-loop filter Including, but not limited to, The steps for performing the above filtering are: A method for applying different convolution operations to the above-described converted luma blocks and the above-described chroma blocks.

Images