Method for compressing additional channel other than YUV channel

Abstract

The present embodiment provides a method for compressing an additional channel other than a YUV channel. In the present embodiment, an image decoding device acquires a bitstream including a luma channel and a chroma channel of an image. The image decoding device reconstructs the luma channel and the chroma channel from the bitstream. The image decoding device identifies whether an additional channel is included in the bitstream. When the additional channel is included in the bitstream, the image decoding device reconstructs the additional channel from the bitstream.

Description

Method for compressing additional channels other than YUV channels

[0001] The present disclosure relates to a video encoding / decoding method, an apparatus, and a recording medium for storing a bitstream, and more specifically, to a method for encoding or decoding existing three channels and an additional channel in the same video compression process using a single bitstream.

[0002] The following description merely provides background information related to the present invention and does not constitute prior art.

[0003] Because video data contains a large amount of data compared to audio or still image data, storing or transmitting it as is without compression processing requires significant hardware resources, including memory.

[0004] Therefore, typically when storing or transmitting video data, the encoder compresses the video data for storage or transmission, and the decoder receives the compressed video data, decompresses it, and plays it. Such video compression technologies include H.264 / AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves coding efficiency by more than 30% compared to HEVC.

[0005] However, as video size, resolution, and frame rates are gradually increasing, and consequently the amount of data that needs to be encoded is also increasing, there is a need for a new compression technology that offers better encoding efficiency and higher image quality improvement effects than existing compression technologies. In particular, existing 2D video compression standards were developed to compress three-channel video such as YUV and RGB. However, as the use of various video maps for video processing has recently increased, there is a need for a method to compress a number of channels greater than three.

[0006] The present disclosure aims to provide a video encoding / decoding method and apparatus for processing three existing channels and an additional channel in the same video compression process using a single bitstream, and a recording medium for storing a bitstream generated by said video encoding method / apparatus.

[0007] According to an embodiment of the present disclosure, a method for restoring an image, performed by an image decoding device, comprises the steps of: acquiring a bitstream including a luminance channel and a chroma channel of the image, wherein the chroma channel includes a first chroma channel and a second chroma channel; restoring the luminance channel and the chroma channel from the bitstream; and determining whether the bitstream includes an additional channel, and further comprising the step of restoring the additional channel from the bitstream if the bitstream includes the additional channel.

[0008] According to another embodiment of the present disclosure, a method for encoding an image, performed by an image encoding device, comprises the steps of: acquiring a luminance channel and a chroma channel of the image, wherein the chroma channel includes a first chroma channel and a second chroma channel; encoding the luminance channel and the chroma channel; and determining whether the image includes an additional channel, and further comprising the step of encoding the additional channel if the image includes the additional channel.

[0009] According to another embodiment of the present disclosure, a method for providing video data to an image decoder comprises: a step of encoding the video data into a bitstream; and a step of transmitting the bitstream to the image decoder, wherein the step of encoding the video data comprises: a step of acquiring a luminance channel and a chroma channel of an image, wherein the chroma channel includes a first chroma channel and a second chroma channel; a step of encoding the luminance channel and the chroma channel; and a step of checking whether the image includes an additional channel, and further comprising a step of encoding the additional channel if the image includes the additional channel.

[0010] As described above, by providing a video encoding / decoding method, an apparatus, and a recording medium storing a bitstream generated by the video encoding method / apparatus according to the present embodiment, it is possible to improve video encoding efficiency and video quality.

[0011] In addition, by providing a video encoding / decoding method according to the present embodiment, it is possible to reduce the burden on the network based on bit rate reduction in various content such as UHD (Ultra High Definition) video, game broadcasting, 360-degree video streaming, VR / AR (Virtual Reality / Augmented Reality) video, online lectures, etc., and to reduce energy consumption for video playback devices.

[0012] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the technologies of the present disclosure.

[0013] Figure 2 is a diagram illustrating a method for dividing blocks using a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.

[0014] FIGS. 3a and 3b are diagrams showing a plurality of intra prediction modes including wide-angle intra prediction modes.

[0015] Figure 4 is an example diagram of the surrounding blocks of the current block.

[0016] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the technologies of the present disclosure.

[0017] Figures 6a to 6c are example diagrams showing a tree structure.

[0018] Figure 7 is an example diagram showing a method of signaling a tree structure.

[0019] Figure 8 is an example diagram showing a single tree structure.

[0020] FIG. 9 is an exemplary diagram showing the size of an additional channel according to one embodiment of the present disclosure.

[0021] FIG. 10 is a flowchart illustrating a method for restoring an image according to one embodiment of the present disclosure.

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the exemplary drawings. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing these embodiments, if it is determined that a detailed description of related known components or functions could obscure the essence of these embodiments, such detailed description is omitted.

[0023] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the technologies of the present disclosure. Hereinafter, the image encoding device and its sub-components will be described with reference to FIG. 1.

[0024] The video encoding device may be configured to include a picture splitting unit (110), a prediction unit (120), a subtractor (130), a conversion unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse conversion unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0025] Each component of the video encoding device may be implemented in hardware or software, or as a combination of hardware and software. Additionally, the function of each component may be implemented in software, and a microprocessor may be implemented to execute the software function corresponding to each component.

[0026] A single image (video) consists of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). And each CTU is divided into one or more Coding Units (CUs) by a tree structure. Information applicable to each CU is encoded as the syntax of the CU, and information applicable to all CUs included in a single CTU is encoded as the syntax of the CTU. Additionally, information applicable to all blocks within a single slice is encoded as the syntax of the slice header, and information applicable to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). Also, information commonly referenced by one or more SPSs is encoded in a Video Parameter Set (VPS). Additionally, information commonly applicable to a single tile or tile group may be encoded as the syntax of a tile or tile group header. The syntax included in the SPS, PPS, slice header, and tile or tile group header may be referred to as high-level syntax.

[0027] The picture splitting unit (110) determines the size of the CTU. Information regarding the size of the CTU (CTU size) is encoded as a syntax of SPS or PPS and transmitted to an image decoding device.

[0028] The picture division unit (110) divides each picture constituting the image into multiple CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. The leaf nodes in the tree structure become the CUs, which are the basic units of encoding.

[0029] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of equal size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that combines two or more of these QT, BT, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT combined may be referred to as an MTT (Multiple-Type Tree).

[0030] Figure 2 is a diagram illustrating a method for dividing blocks using a QTBTTT structure.

[0031] As illustrated in FIG. 2, the CTU can first be split into a QT structure. Quadtree splitting can be repeated until the size of the splitting block reaches the minimum block size of the leaf node allowed in QT (MinQTSize). A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder (155) and signaled to the image decoder. If the leaf node of the QT is not larger than the maximum block size of the root node allowed in BT (MaxBTSize), it can be further split into one or more of the BT structure or TT structure. In the BT structure and / or TT structure, multiple splitting directions may exist. For example, there may be two directions in which the block of the corresponding node is split horizontally and vertically. As shown in Figure 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether the nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or the splitting type (binary or ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0032] Alternatively, prior to encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of the lower layer, the CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the value of the CU split flag (split_cu_flag) indicates that it is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a coding unit (CU), which is the basic unit of encoding. If the value of the CU split flag (split_cu_flag) indicates that it is split, the video encoding device starts encoding from the first flag in the manner described above.

[0033] When QTBT is used as another example of a tree structure, there may be two types: a type that divides the block of the corresponding node horizontally into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that divides it vertically (i.e., symmetric vertical splitting). A splitting flag (split_flag) indicating whether each node of the BT structure is split into a block of a lower layer and splitting type information indicating the type of splitting are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may also be an additional type that divides the block of the corresponding node into two blocks of an asymmetric shape. The asymmetric shape may include a shape that divides the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that divides the block of the corresponding node diagonally.

[0034] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from a CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of QTBTTT) is referred to as the 'current block'. Depending on the adoption of QTBTTT partitioning, the shape of the current block may be not only square but also rectangular.

[0035] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0036] Generally, current blocks within a picture can each be predictively coded. Typically, the prediction of a current block can be performed using an intra-prediction technique (using data from the picture containing the current block) or an inter-prediction technique (using data from a picture coded prior to the picture containing the current block). Inter-prediction includes both unidirectional and bidirectional prediction.

[0037] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture containing the current block. Multiple intra prediction modes exist depending on the prediction direction. For example, as shown in FIG. 3a, multiple intra prediction modes may include two non-directional modes, including Planar mode and DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently for each prediction mode.

[0038] For efficient directional prediction for a rectangular current block, directional modes (intra-prediction modes 67 through 80 and -1 through -14) illustrated by dashed arrows in FIG. 3b may be additionally used. These may be referred to as "wide angle intra-prediction modes." In FIG. 3b, the arrows indicate corresponding reference samples used for prediction and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. Among the wide angle intra-prediction modes, some wide angle intra-prediction modes available for the current block may be determined by the ratio of the width to the height of the rectangular current block. For example, wide-angle intra-prediction modes with an angle less than 45 degrees (intra-prediction modes 67 to 80) are available when the current block is a rectangular shape with a height less than the width, and wide-angle intra-prediction modes with an angle greater than -135 degrees (intra-prediction modes -1 to -14) are available when the current block is a rectangular shape with a width greater than the height.

[0039] The intra prediction unit (122) can determine the intra prediction mode to use for encoding the current block. In some examples, the intra prediction unit (122) may encode the current block using several intra prediction modes and select an appropriate intra prediction mode to use from the tested modes. For example, the intra prediction unit (122) may calculate the rate-distortion values ​​using a rate-distortion analysis of several tested intra prediction modes and select the intra prediction mode having the best rate-distortion features among the tested modes.

[0040] The intra prediction unit (122) selects one intra prediction mode among a plurality of intra prediction modes and predicts the current block using a calculation formula and surrounding pixels (reference pixels) determined according to the selected intra prediction mode. Information regarding the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0041] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for the block most similar to the current block within a reference picture that is encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, it generates a motion vector (MV) corresponding to the displacement between the current block in the current picture and the prediction block in the reference picture. Generally, motion estimation is performed on the lumina component, and the motion vector calculated based on the lumina component is used for both the lumina component and the chroma component. Motion information including information about the reference picture used to predict the current block and information about the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0042] The inter prediction unit (124) may perform interpolation on a reference picture or reference block to increase the accuracy of the prediction. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including those two integer samples. When the process of searching for the block most similar to the current block is performed for the interpolated reference picture, the motion vector can be expressed with precision in fractional units rather than precision in integer sample units. The precision or resolution of the motion vector can be set differently for each unit of the target area to be encoded, such as slice, tile, CTU, CU, etc. When such Adaptive Motion Vector Resolution (AMVR) is applied, information regarding the motion vector resolution to be applied to each target area must be signaled for each target area. For example, if the target area is a CU, information regarding the motion vector resolution applied to each CU is signaled. The information regarding the motion vector resolution may be information indicating the precision of the difference motion vector described later.

[0043] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors representing the block location most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from the reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and generates a first reference block and a second reference block by searching for a block similar to the current block within each reference picture. Then, it generates a prediction block for the current block by averaging or weighting the first reference block and the second reference block. Then, it transmits motion information containing information about the two reference pictures used to predict the current block and information about the two motion vectors to the entropy encoding unit (155). Here, reference picture list 0 consists of restored pictures that are prior to the current picture in the display order, and reference picture list 1 may consist of restored pictures that are prior to the current picture in the display order. However, this is not necessarily limited to this, and restored pictures prior to the current picture in the display order may be additionally included in reference picture list 0, and conversely, restored pictures prior to the current picture may be additionally included in reference picture list 1.

[0044] Various methods can be used to minimize the amount of bits required to encode motion information.

[0045] For example, if the reference picture and motion vector of the current block are identical to the reference picture and motion vector of a neighboring block, the motion information of the current block can be transmitted to an image decoder by encoding information that can identify the neighboring block. This method is called 'merge mode'.

[0046] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

[0047] As for the surrounding blocks for deriving merge candidates, as shown in FIG. 4, all or part of the left block (A0), bottom-left block (A1), top block (B0), top-right block (B1), and top-left block (B2) adjacent to the current block within the current picture may be used. Additionally, a block located within a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture where the current block is located may be used as a merge candidate. For example, a block located at the same position as the current block within the reference picture (co-located block) or a block adjacent to that same position may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.

[0048] The inter prediction unit (124) constructs a merge list containing a predetermined number of merge candidates using these surrounding blocks. Among the merge candidates included in the merge list, it selects a merge candidate to be used as movement information for the current block and generates merge index information to identify the selected candidate. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0049] Merge skip mode is a special case of merge mode; after quantization, when all transform coefficients for entropy coding are close to zero, only neighbor block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high coding efficiency can be achieved in images with minimal motion, still images, and screen content images.

[0050] Hereinafter, merge mode and merge skip mode will be collectively referred to as merge / skip mode.

[0051] Another method for encoding motion information is the AMVP (Advanced Motion Vector Prediction) mode.

[0052] In AMVP mode, the inter-prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using the surrounding blocks of the current block. As surrounding blocks used to derive predicted motion vector candidates, all or part of the left block (A0), bottom-left block (A1), top block (B0), top-right block (B1), and top-left block (B2) adjacent to the current block within the current picture shown in FIG. 4 may be used. Additionally, blocks located within a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture where the current block is located may be used as surrounding blocks to derive predicted motion vector candidates. For example, blocks located at the same position as the current block within the reference picture (co-located blocks) or blocks adjacent to the blocks at the same position may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.

[0053] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of the surrounding blocks and determines a predicted motion vector for the current block's motion vector using the predicted motion vector candidates. Then, it calculates a difference motion vector by subtracting the predicted motion vector from the current block's motion vector.

[0054] Predicted motion vectors can be obtained by applying a predefined function (e.g., median, mean operation, etc.) to the predicted motion vector candidates. In this case, the image decoder is also aware of the predefined function. Furthermore, since the surrounding blocks used to derive the predicted motion vector candidates have already been encoded and decoded, the image decoder is also aware of the motion vectors of those surrounding blocks. Therefore, the image decoder does not need to encode information to identify the predicted motion vector candidates. Consequently, in this case, information regarding the difference motion vector and the reference picture used to predict the current block is encoded.

[0055] Meanwhile, the predicted motion vector may be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the difference motion vector and information about the reference picture used to predict the current block.

[0056] The subtractor (130) generates a residual block by subtracting the prediction block generated by the intra prediction unit (122) or the inter prediction unit (124) from the current block.

[0057] The conversion unit (140) converts residual signals within a residual block having pixel values ​​in a spatial domain into conversion coefficients in the frequency domain. The conversion unit (140) can convert the residual signals within the residual block using the entire size of the residual block as the conversion unit, or it can divide the residual block into multiple sub-blocks and use the sub-blocks as the conversion unit to perform the conversion. Alternatively, it can divide the residual signals into two sub-blocks, a conversion area and a non-conversion area, and use only the conversion area sub-block as the conversion unit to convert the residual signals. Here, the conversion area sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only the sub-block has been converted, direction (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. Additionally, the size of the converted area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0058] Meanwhile, the transformation unit (140) can perform transformations on the residual block individually in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices may be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one pair of transformation functions with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) regarding the selected pair of transformation functions among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoder.

[0059] The quantization unit (145) quantizes the transformation coefficients output from the transformation unit (140) using quantization parameters and outputs the quantized transformation coefficients to the entropy encoding unit (155). The quantization unit (145) may quantize the associated residual block directly without transformation for any block or frame. The quantization unit (145) may apply different quantization coefficients (scaling values) depending on the position of the transformation coefficients within the transformation block. The quantization matrix applied to the quantized transformation coefficients arranged in two dimensions can be encoded and signaled to an image decoder.

[0060] The reordering unit (150) can perform reordering of coefficient values ​​for quantized residual values.

[0061] The reordering unit (150) can convert a two-dimensional coefficient array into a one-dimensional coefficient sequence using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from DC coefficients to coefficients in the high-frequency range using a zig-zag scan or a diagonal scan. Depending on the size of the conversion unit and the intra-prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction and a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of a zig-zag scan. That is, depending on the size of the conversion unit and the intra-prediction mode, the scanning method to be used among a zig-zag scan, a diagonal scan, a vertical scan, and a horizontal scan may be determined.

[0062] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transformation coefficients output from the reordering unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0063] Additionally, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the video decoder can divide the block in the same way as the video encoding unit. Additionally, the entropy encoding unit (155) encodes information regarding a prediction type indicating whether the current block is encoded by intra prediction or by inter prediction, and encodes intra prediction information (i.e., information regarding the intra prediction mode) or inter prediction information (information regarding the encoding mode of motion information (merge mode or AMVP mode), the merge index in the case of merge mode, and the reference picture index and difference motion vector in the case of AMVP mode) according to the prediction type. Additionally, the entropy encoding unit (155) encodes information related to quantization, i.e., information regarding quantization parameters and information regarding the quantization matrix.

[0064] The inverse quantization unit (160) inversely quantizes the quantized transformation coefficients output from the quantization unit (145) to generate transformation coefficients. The inverse transformation unit (165) converts the transformation coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0065] The adder (170) restores the current block by adding the restored residual block and the prediction block generated by the prediction unit (120). The pixels within the restored current block are used as reference pixels when intra-predicting the next block in sequence.

[0066] The loop filter section (180) performs filtering on the restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. caused by block-based prediction and transformation / quantization. The loop filter section (180) may include all or part of a deblocking filter (182), a SAO (Sample Adaptive Offset) filter (184), and an ALF (Adaptive Loop Filter, 186) as an in-loop filter.

[0067] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-unit encoding / decoding, and the SAO filter (184) and ALF (186) perform additional filtering on the deblocking filtered image. The SAO filter (184) and ALF (186) are filters used to compensate for the difference between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in CTU units. In contrast, the ALF (186) performs block-unit filtering, and compensates for distortion by applying different filters by distinguishing the degree of edge and change of the corresponding block. Information regarding the filter coefficients to be used in the ALF can be encoded and signaled to an image decoder.

[0068] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in memory (190). Once all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0069] The video encoding device can store the bitstream of encoded video data on a non-transient recording medium or transmit it to a video decoding device using a communication network.

[0070] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the technologies of the present disclosure. Hereinafter, the image decoding device and its sub-components will be described with reference to FIG. 5.

[0071] The image decoding device may be configured to include an entropy decoding unit (510), a reordering unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

[0072] Similar to the image encoding device of FIG. 1, each component of the image decoding device may be implemented in hardware or software, or in combination of hardware and software. Additionally, the function of each component may be implemented in software, and a microprocessor may be implemented to execute the function of the software corresponding to each component.

[0073] The entropy decoding unit (510) determines the current block to be decoded by decoding the bitstream generated by the video encoding device and extracting information related to block division, and extracts prediction information, information on residual signals, etc., necessary to restore the current block.

[0074] The entropy decoding unit (510) extracts information about the CTU size from the SPS (Sequence Parameter Set) or PPS (Picture Parameter Set) to determine the size of the CTU and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and divides the CTU using the tree structure by extracting division information for the CTU.

[0075] For example, when splitting a CTU using a QTBTTT structure, first, a first flag (QT_split_flag) related to QT splitting is extracted to split each node into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, a second flag (mtt_split_flag) related to MTT splitting and splitting direction (vertical / horizontal) and / or splitting type (binary / ternary) information are extracted to split the corresponding leaf nodes into an MTT structure. Accordingly, each node below the leaf nodes of QT is recursively split into a BT or TT structure.

[0076] As another example, when splitting a CTU using the QTBTTT structure, a CU splitting flag (split_cu_flag) indicating whether to split the CU is first extracted, and if the block is split, a first flag (QT_split_flag) is extracted. During the splitting process, each node may undergo zero or more iterative MTT splittings after zero or more iterative QT splittings. For example, the CTU may undergo MTT splitting immediately, or conversely, only multiple QT splittings may occur.

[0077] As another example, when splitting a CTU using a QTBT structure, a first flag (QT_split_flag) related to the splitting of QT is extracted to split each node into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and split direction information are extracted.

[0078] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded using the division of the tree structure, it extracts information regarding the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-predicted, the entropy decoding unit (510) extracts syntax elements for the intra-predicted information (intra-predicted mode) of the current block. If the prediction type information indicates inter-predicted, the entropy decoding unit (510) extracts syntax elements for the inter-predicted information, namely information indicating the motion vector and the reference picture that the motion vector refers to.

[0079] Additionally, the entropy decoding unit (510) extracts information regarding quantization-related information and information regarding residual signals, as well as information regarding the quantized transformation coefficients of the current block.

[0080] The reordering unit (515) can change the sequence of one-dimensional quantized transformation coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning order performed by the image encoding device.

[0081] The inverse quantization unit (520) inversely quantizes the quantized transformation coefficients and inversely quantizes the quantized transformation coefficients using quantization parameters. The inverse quantization unit (520) may apply different quantization coefficients (scaling values) to the quantized transformation coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transformation coefficients.

[0082] The inverse transformation unit (530) generates a residual block for the current block by inversely transforming the inversely quantized transformation coefficients from the frequency domain to the spatial domain and restoring the residual signals.

[0083] Additionally, when the inverse transformation unit (530) inversely transforms only a part of the transformation block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only the sub-block of the transformation block has been transformed, information on the directionality (vertical / horizontal) of the sub-block (cu_sbt_horizontal_flag) and / or information on the position of the sub-block (cu_sbt_pos_flag), restores residual signals by inversely transforming the transformation coefficients of the corresponding sub-block from the frequency domain to the spatial domain, and creates a final residual block for the current block by filling the areas that have not been inversely transformed with "0" values ​​of residual signals.

[0084] Additionally, when MTS is applied, the inverse transformation unit (530) determines a transformation function or transformation matrix to be applied in the horizontal and vertical directions, respectively, using MTS information (mts_idx) signaled from the video encoding device, and performs an inverse transformation on the transformation coefficients within the transformation block in the horizontal and vertical directions using the determined transformation function.

[0085] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is an intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is an inter prediction.

[0086] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0087] The inter prediction unit (544) determines the motion vector of the current block and the reference picture that the motion vector refers to using the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510), and predicts the current block using the motion vector and the reference picture.

[0088] The adder (550) restores the current block by adding the residual block output from the inverse transformation unit (530) and the prediction block output from the inter prediction unit (544) or the intra prediction unit (542). The pixels within the restored current block are used as reference pixels when intra-predicting the block to be decoded later.

[0089] The loop filter section (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as an in-loop filter. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-unit decoding. The SAO filter (564) and the ALF (566) perform additional filtering on the restored blocks after deblocking filtering to compensate for the difference between the restored pixels and the original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about the filter coefficients decoded from the bitstream.

[0090] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture to inter-predict blocks within the picture to be encoded later.

[0091] The present embodiment relates to the encoding and decoding of an image (video) as described above. More specifically, the present invention provides an image encoding / decoding method and apparatus for processing three existing channels and an additional channel in the same video compression process using a single bitstream, and a recording medium for storing a bitstream generated by the image encoding method / apparatus.

[0092] The following embodiments may be performed by a video encoding apparatus. Additionally, the following embodiments may be performed by a video decoding apparatus.

[0093] The video encoding device can generate signaling information related to the present embodiment in terms of rate distortion optimization during the encoding of the current block. The video encoding device can encode the signaling information using the entropy encoding unit (155) and then transmit it to the video decoder. The video decoder can decode the signaling information related to the decoding of the current block from the bitstream using the entropy decoder (510).

[0094] In the following description, the term 'target block' may be used interchangeably with 'current block' or 'Coding Unit (CU).' Alternatively, 'target block' may refer to a specific area of ​​a Coding Unit.

[0095] Also, a value of one flag being true indicates that the flag is set to 1. Also, a value of one flag being false indicates that the flag is set to 0.

[0096] The decoder-side includes all or part of an inverse quantizer (160), an inverse transform (165), a prediction unit (120), an adder (170), a loop filter (180), and a memory (190) in the image encoding device illustrated in FIG. 1. Alternatively, the decoder-side includes all or part of an inverse quantizer (520), an inverse transform (530), a prediction unit (540), an adder (550), a loop filter (560), and a memory (570) in the image decoding device illustrated in FIG. 5. In relation to a series of decoding processes, the decoder-side of the image encoding device and the decoder-side of the image decoding device perform the same operation. The image encoding device determines information related to the operation of the decoder-side and signals the determined information to the image decoding device. The image decoding device can decode the signaled information and operate the decoder-side based on the decoded information.

[0097] I. VVC Channel-Related Technology

[0098] I-1. VVC Color Space and Chroma Sampling

[0099] VVC improves compression efficiency by separating the RGB components of an image into channels containing brightness information and channels containing color information for encoding. Brightness information is called the luminance component, and color information excluding brightness is called the chroma component. YCbCr is a representative color space representation method that separates and uses luminance and chroma information. Y represents the luminance component, and the two components Cb and Cr are included to form the chroma component. An image array containing only the Y component is represented as the Y channel or luminance channel. An image array containing Cb and Cr is represented as the Cb and Cr channels or chroma channel, respectively.

[0100] Compared to applying equal sampling to the R, G, and B components of the original image information, 4:4:4 YCbCr applies equal sampling to the Y component and the Cb and Cr components. As a result, the channel sizes of Y, Cb, and Cr all become identical. In the case of 4:2:2 YCbCr, the two chroma components are subsampled to half the resolution of the luminance component. That is, the horizontal chroma resolution becomes half the horizontal resolution of the luminance component. In the case of 4:2:0 YCbCr, the two chroma components are subsampled horizontally and vertically, respectively, to half the resolution of the luminance component. That is, the horizontal and vertical chroma resolutions become half the horizontal and vertical resolutions of the luminance component. Consequently, in 4:2:2 YCbCr, the channel sizes of Cb and Cr are half the size of the Y channel. Furthermore, in 4:2:0 YCbCr, the channel sizes of Cb and Cr are one-quarter the size of the Y channel. Chroma subsampling is based on the cognitive process that human vision is less sensitive to changes in color than to changes in brightness. Compared to representing original image information in 4:4:4 RGB, when an image is represented in 4:2:0 YCbCr, only half of the information from the entire image is required. Since this reduction in visual information is relatively imperceptible to humans, significant storage capacity can be saved.

[0101] Below, YCbCr can be used compatiblely with YUV.

[0102] Table 1 shows the sampling method for the chroma component of VVC and related representations. Hereinafter, the sampling method for the chroma component will be referred to as the chroma format.

[0103]

[0104] In Table 1, the case where only Y is included represents a monochrome image. SubWidthC and SubHeightC are variables that determine the ratio of width and height between the luminance and chroma components, and can be determined based on the chroma format based on chroma_format_idc and separate_colour_plane_flag. For example, if chroma_format_idc is signaled as 1, SubWidthC and SubHeightC are determined as 2, and the video stream represents a bitstream of compressed 4:2:0 YUV image. If chroma_format_idc is signaled as 3, the video stream represents a bitstream of compressed 4:4:4 YUV image. In this case, the following cases may apply depending on separate_colour_plane_flag.

[0105] When separate_colour_plane_flag is 0, each of the two chroma arrays has the same height and width as the lumina array. The VVC decoding process is performed once.

[0106] When separate_colour_plane_flag is 1, the three color planes, Y, Cb, and Cr, are separated into monochrome images and processed individually. In other words, the three color planes are compressed individually using VVC monochrome coding syntax. In VVC, each plane is distinguished using an ID called colour_plane_id, which has a value of 0, 1, or 2. 0, 1, and 2 correspond to Y, Cb, and Cr, respectively.

[0107] chroma_format_idc and separate_colour_plane_flag can be signaled using higher-level syntax such as VVC's VPS (video parameter set), SPS (sequence parameter set), or PPS (picture parameter set), slice header, etc. The chroma format can be determined using at least one of chroma_format_idc and separate_colour_plane_flag.

[0108] In VVC, the ChromaArrayType variable, which indicates the type of the chroma component array, is determined by chroma_format_idc. If separate_colour_plane_flag is 0, ChromaArrayType can be set to be the same as chroma_format_idc. If separate_colour_plane_flag is 1, ChromaArrayType is set to 0, representing monochrome. ChromaArrayType indicates whether the component currently being compressed is a luminance component or a chroma component, and additionally, it can be used for block splitting, predictive coding, and transformation of the chroma component.

[0109] The number of bits required to represent each sample in the luminance and chroma arrays of a video sequence ranges from 8 to 16. The number of bits used in the luminance array may differ from the number of bits used in the chroma array. This information can be conveyed using higher-level syntax such as SPS.

[0110] I-2. The Partition Tree of VVC

[0111] The image segmentation process of VVC generates various coding units (CUs) based on the recursive segmentation of coding tree units (CTUs). The image is first divided into CTUs of uniform size. The CTU becomes the top node in the segmentation tree structure for dividing into CUs. Therefore, the maximum size of a CU is equal to the size of a CTU. The image is first divided into 128×128 CTUs, and each CTU can be continuously divided into CUs. Within a CTU, CUs can be configured in various sizes and shapes, ranging from a minimum of 4×4 to the maximum allowable CU size.

[0112] In HEVC, CU sizes up to 64×64 are allowed, but in VVC, CUs of size 128×128 may be allowed. Compared to HEVC, VVC can use CUs with a wider range of allowed sizes (e.g., 128×128 CUs). Additionally, compared to HEVC, VVC can use CUs of various partition shapes.

[0113] The size of the CTU is determined for the luminance and chroma components as follows. CtbSizeY, a variable representing the CTU size of the luminance component, is determined based on parameters passed using higher-level syntax such as VPS, SPS, and PPS. Using CtbSizeY and SubWidthC and SubHeightC from Table 1, CtbWidthC and CtbHeightC, variables representing the CTU size of the chroma component, can be determined. If chroma_format_idc is 0 or separate_colour_plane_flag is 1, CtbWidthC and CtbHeightC are 0. In other cases, CtbWidthC and CtbHeightC are determined as in Equation 1.

[0114]

[0115] As mentioned above, VVC uses BT (Binary Tree), TT (Ternary Tree), and QT (Quad Tree) as tree structures for CU partitioning. The BT and TT structures are collectively referred to as MTT (multi-type tree) structures.

[0116] QT is a tree structure that has four child nodes under one node, as in HEVC. When the QT structure is applied to the current block, as exemplified in FIG. 6a, the current block is divided into four child blocks of the same size, and each child block corresponds to a child node. BT is a tree structure that generates two child nodes from one node, and as exemplified in FIG. 6b, the parent block can be divided into two child blocks of the same size in a horizontal or vertical direction. TT is a tree structure that generates three child nodes from one node in a horizontal or vertical direction. As exemplified in FIG. 6c, child blocks can be generated by dividing the width or height of the parent block in a length ratio of 1:2:1 depending on the division direction.

[0117] CUs can have various sizes and shapes ranging from a minimum of 4×4 to a maximum of 128×128. QTs are recursively partitioned within the allowable size. If a partitioned block satisfies the conditions of the MTT configuration (block size, depth, etc.), BT or TT partitioning can be applied to the partitioned block. If BT or TT partitioning is applied to the current node, QT partitioning can no longer be applied to the child nodes.

[0118] In the example of Fig. 7, the root node represents the CTU. The root node is recursively partitioned by QT. When the minimum size block is reached through continuous partitioning that can no longer be partitioned by QT, the block is partitioned into smaller blocks using BT and TT. For example, if the partitioned block becomes smaller than the size of MinQTSize signaled to the SPS, QT is no longer used.

[0119] As previously mentioned, blocks to which BT and TT have been applied are no longer divided according to QT. When the depth of the block's MTT reaches MaxMttDepth, the block is no longer divided. When the width or height of the block becomes equal to MinCbsize, the block is no longer divided.

[0120] In the example of Fig. 7, the flags related to the splitting of the CTU are as follows. split_cu_flag indicates whether to further split the corresponding node, i.e., the corresponding block. split_qt_flag indicates whether to use QT or MTT to split the corresponding block. mtt_split_cu_vertical_flag indicates whether the splitting direction for the MTT splitting of the corresponding block is horizontal or vertical. Additionally, mtt_split_cu_binary_flag indicates whether the MTT splitting of the corresponding block is BT or TT.

[0121] In the case of YUV images, the image consists of a luminance component Y and two chroma components U and V. In VVC, when the characteristics of the luminance component and the chroma component are different, different tree structures may be used so that the images of the luminance component and the chroma component can be segmented differently.

[0122] The single tree structure utilizes the same tree structure in the lumens and chroma components. The single tree structure is used in the I, P, or B slices. The single tree structure can be implemented as shown in the example of Fig. 8. In the example of Fig. 8, the depth of the root node is set to 0.

[0123] The dual tree structure utilizes different tree structures for block splitting in the luminance and chroma components. The dual tree structure can be used only for I slices. The video encoder can specify one of the dual tree or single tree methods by signaling a flag indicating whether to use the dual tree to the video decoder. When the dual tree is used, the size of the smallest luminance block is 4×4, and the size of the chroma block is 4×4 or 8×2.

[0124] I-3. Multispectral Imaging and Image Maps for Video Processing

[0125] Multispectral imaging is an image acquired at a wider range of wavelengths by subdividing the visible spectrum band (R, G, B) region, and can be composed of the visible and near-infrared regions. Satellites utilize multispectral sensors with four to seven spectral bands to monitor geological conditions and weather. As the number of spectral bands gradually increases, hyperspectral technology is increasingly being utilized in defense and machine vision.

[0126] Videos collected in environments such as autonomous driving and the Internet of Things are primarily used for services that recognize, analyze, and utilize objects or events, and the usage of such services is continuously increasing. When the video acquisition device and the analysis device are separated, the transmitting side encodes and transmits the original video using a video codec, while the receiving side utilizes the decoded video for various image processing tasks. However, if the original image quality is degraded due to noise generated during the video compression process, the aforementioned operations may not proceed smoothly. Therefore, it is necessary to transmit auxiliary information along with the video to ensure that various image processing tasks can be performed smoothly at the receiving side. This auxiliary information is distinguished from the existing 3-channel YUV information and can be additionally restored using a separate decoding process. For example, the following information may be transmitted.

[0127] In the case of 3D video compression, a depth map representing depth information is compressed and transmitted in addition to the YUV video signal.

[0128] In MPEG7 CDVS (compact descriptors for visual search) or MPEG VCM (video coding for machine), image feature maps extracted from images for specific vision tasks are transmitted. These feature maps can be used for video recognition, segmentation, etc.

[0129] An alpha map is information that indicates the ratio applied when compositing computer graphics and live-action footage. During compositing, alpha maps can be used to blend the boundaries between the two images naturally, so that the composite image appears as a single unit.

[0130] The following embodiments are described with reference to an image decoding device, but may be implemented identically or similarly in an image encoding device. Alternatively, the following embodiments are described with reference to the decoder side of the image decoding device, but may be implemented identically or similarly in the decoder side of an image encoding device.

[0131] In this specification, the term 'and / or' includes a combination of a plurality of described items or any of a plurality of described items.

[0132] II. Embodiments according to the present disclosure

[0133] The present disclosure shows a method for restoring three existing channels and an additional channel in the same video decoding process using a single bitstream. A video decoder checks whether the bitstream contains an additional channel and restores the additional channel based on the check result. The video decoder restores the additional channel after the decoding of at least one of the Y, U, and V channels is completed. Alternatively, the video decoder may restore the additional channel individually regardless of the decoding process of the Y, U, and V channels. According to the above process, the encoding efficiency of a video having a large number of channels can be increased.

[0134] The following describes a method for indicating and checking whether additional channels in a bitstream are compressed.

[0135] The video encoding device signals the video decoder with a channel index indicating the number of additional channels or a flag indicating the existence of additional channels, thereby conveying that a separate channel has been compressed in addition to the existing three channels. The video decoder can explicitly obtain the index of the additional channel or whether it exists by parsing the channel index indicating the additional channel from the bitstream. The aforementioned information can be transmitted using higher-level syntax such as VPS, SPS, PPS, etc.

[0136] The channel index or flag can be set based on an index or flag indicating the compression method of the existing YUV channel.

[0137] New channels added in addition to the existing YUV channels are defined as A, B, C, D, ... There may be one or more additional channels.

[0138] For example, additional channels can be mapped to channel indices. For instance, if the existing YUV channels are represented by channel indices 0, 1, and 2, the additional channels start at index 3. If the Y channel and UV channel are represented by channel indices 0 and 1, the additional channels start at index 2. Alternatively, if only the Y channel exists and is represented by index 0, new channels other than the UV channel start at index 1. For instance, as shown in Table 2, the number and existence of additional channels are indicated using indices that were not previously used for the YUV channels.

[0139]

[0140] In Table 2, a new channel starts from channel index 3. If the channel index in Table 2 ends at N-1, it indicates that N-3 additional channels have been compressed. Additionally, the channels corresponding to each index can be identified from Table 2.

[0141] As another example, additional channels may start with channel index 0. The video encoding device may transmit a flag indicating the existence of additional channels, and if the flag is true, transmit an index indicating the number of additional channels. The additional channel index starts from 0. For example, if the flag is true, Table 3 shows how to display the additional channel index.

[0142]

[0143] If the additional channel index ends in N-1, it indicates that N additional channels have been compressed. Also, the channels corresponding to each index can be identified from Table 3.

[0144] In addition to explicitly conveying the number and existence of additional channels as in Table 2 or Table 3, as another example, the video decoder may derive the number and existence of additional channels using one or more combinations of indices and flags. For example, as in Table 4, additional channels may be indicated by defining a video format according to the channel index.

[0145]

[0146] In Table 4, channel index 4 implicitly indicates that there is one additional channel by defining 4:2:0 sampling for YUV and monochrome for the additional channel. Channel index 5 implicitly indicates that there are three additional channels by defining 4:2:0 sampling for YUV and 4:2:0 sampling for the additional channel.

[0147] Below, we describe a method to extend VVC's chroma_format_idc to indicate and check whether additional channels are compressed.

[0148] VVC uses chroma_format_idc to indicate the sampling mode of the video. If the sampling mode is monochrome and chroma_format_idc is 0, it can be inferred that only the Y channel is encoded. Additionally, if chroma_format_idc is not 0 and has a value of 1, 2, or 3, it can be inferred that both YUV channels are encoded.

[0149] As an example, additional channels can be indicated using VVC's chroma_format_idc. Since 0, 1, 2, and 3 are already in use as chroma_format_idc, values ​​after index 4 can be used to indicate additional channels A, B, and C. Similar to how VVC's chroma_format_idc indicates channels, as shown in Table 5, values ​​after index 4 represent the sampling of A, B, and C, and the existence of ABC is implicitly derived from the corresponding index.

[0150]

[0151] The values ​​from indices 4 through 7 define the sampling of A, B, and C for YUV4:2:0, and the presence of A, B, and C is implicitly expressed according to these definitions. For example, index 4 indicates that only channel A exists as an additional channel. Index 5 indicates that only A and B exist as additional channels. Indices 6 and 7 indicate that A, B, and C exist as additional channels. Similarly, indices 8 through 11 define the sampling of A, B, and C for YUV4:2:2, and the presence of A, B, and C is implicitly expressed according to these definitions. Indices 12 through 15 define the sampling of A, B, and C for YUV4:4:4, and the presence of A, B, and C is implicitly expressed according to these definitions.

[0152] In the VVC signaling scheme shown in Table 1, each channel is compressed independently by combining chroma_format_idc and separate_colour_plane_flag. For example, if chroma_format_idc is 3 and separate_colour_plane_flag = 1, the Y, U, and V channels are compressed independently. For additional channels as well, as shown in Table 6, the number and existence of additional channels can be indicated and they can be compressed independently depending on chroma_format_idc and separate_colour_plane_flag.

[0153]

[0154] In Table 6, when chroma_format_idc is 7 and separate_colour_plane_flag=1, the image decoder can parse a channel index indicating additional channel A from the bitstream to explicitly obtain the number and existence of additional channels. The aforementioned information can be transmitted using higher-level syntax such as VPS, SPS, PPS, etc.

[0155] The following describes how to display the sampling format (i.e., video format) of additional channels.

[0156] Values ​​regarding the sampling format of additional channels may be explicitly indicated. The video format of additional channels may be explicitly conveyed based on Table 4 or Table 5. The aforementioned information may be conveyed using higher-level syntax such as VPS, SPS, PPS, etc. For example, in Table 4, channel index 5 explicitly signals the sampling format 4:2:0 / 4:2:0 of ​​YUV / ABC.

[0157] Additionally, the video decoder can derive the video format of an additional channel based on one or more previously encoded combinations of indices and flags. For example, assuming the new channel is channel A, one of the following examples may be used.

[0158] The sampling format of channel A follows the sampling format of channel Y.

[0159] If U and V channels exist, the sampling format of channel A follows the sampling format of channels U and V.

[0160] The sampling format of the A channel is determined by combining chroma_format_idc and separate_colour_plane_flag. For example, in Table 6, if chroma_format_idc = 7 and separate_colour_plane_flag = 0, the sampling format of the A channel follows the sampling format of the U and V channels. Alternatively, the sampling format of the A channel follows the sampling format of the Y channel.

[0161] If chroma_format_idc = 7 and separate_colour_plane_flag = 1, the sampling format of channel A is signaled separately.

[0162] If channels A and B exist as one or more new channels, one of the following examples may be used.

[0163] The sampling formats of channels A and B both follow the sampling format of channel Y.

[0164] If U and V channels exist, the sampling formats of A and B channels both follow the sampling formats of U and V channels.

[0165] The sampling format of the A and B channels is determined by combining chroma_format_idc and separate_colour_plane_flag. For example, in Table 6, if chroma_format_idc = 11 and separate_colour_plane_flag = 0, the sampling format of the A and B channels follows the sampling format of the Y channel and (U, V) channel, respectively.

[0166] If chroma_format_idc = 11 and separate_colour_plane_flag = 1, the sampling formats of channels A and B are signaled separately. Alternatively, the sampling formats of channels A and B both follow the sampling format of the Y channel.

[0167] The derivation of the image array size of additional channels is described below.

[0168] The size of the image array of the additional channel can be derived from the size of the (image array) of the YUV channel. In this case, one or a combination of the following methods may be utilized.

[0169] If the vertical and horizontal dimensions of the Y channel are Height and Width, respectively, the additional channel A also has an image array of the same size as the Y channel. That is, the vertical and horizontal dimensions of channel A are Height and Width.

[0170] Channel A has the same size as the UV channel. The vertical and horizontal dimensions of Channel A can be derived as Height / SubHeightC and Width / SubWidthC using SubHeightC and SubWidthC shown in Table 1.

[0171] Channel A can be an array of images with different sizes within the same video sequence. For example, odd-numbered images may have the same size as the Y channel, while even-numbered images may have the same size as the UV channel.

[0172] FIG. 9 is an exemplary diagram showing the size of an additional channel according to one embodiment of the present disclosure.

[0173] Meanwhile, the combined size of the A channel and the UV channel can be set to be equal to the size of the Y channel. That is, as shown in FIG. 9 (a), when an image is transmitted according to the YUV 4:2:0 sampling format, the A channel can be transmitted by upsampling by 2 times in either the horizontal or vertical direction of the UV channel.

[0174] Additionally, as shown in FIG. 9(b), if channel A has the same size as the UV channel and the sum of the sizes of channels U, V, and A is smaller than the Y channel, the remaining area may be left empty without allocating any channels. In this case, the remaining area may be filled with a value of 0 or 2Bitdepth-1. According to the method described above, channels U, V, and A may be packed to have an image array identical to the Y channel. Here, Bitdepth represents the bit depth of the YUV image.

[0175] Based on the method exemplified in Fig. 9, the array of Y channels can always start at a multiple of an integer defined as Height×Width.

[0176] If channels B and C are included as additional channels in addition to channel A, the size of the B and C image arrays can be derived from the size of the YUV channel and the sampling format of channels A, B, and C. For example, if the width and height of channel A are determined as WidthA and HeightA, the size of channels B and C can be determined by defining SubWidthV and SubHeightV, which represent the size ratio between channel A and channels B and C. SubWidthV and SubHeightV are, for example, 1 or 2. Alternatively, any value may be signaled.

[0177] Alternatively, by separately defining SubWidthA and SubHeightA, which represent the size ratio between the Y channel and the A channel, the sampling format of the A channel can be derived from the Y channel. That is, the height and width of the A channel are determined as Height / SubHeightA and Width / SubWidthA. In this case, the size of the B and C channels can be determined from the sampling format of the A channel and the B and C channels, or the size of the B and C channels can be determined by defining separate SubWidthV and SubHeightV. For example, if the video format of the A, B, and C channels is 4:2:0, the image height and width of the B and C channels are Height / SubHeightA / 2 and Width / SubWidthA / 2. Even in the aforementioned case, the array of the Y channel can always start at a position that is a multiple of an integer defined as Height×Width.

[0178] Below, the dependent and independent restoration of additional channels are described.

[0179] As an example, the video decoder may utilize a flag indicating whether to use the parameters of the YUV channel when restoring additional channels. For example, the separate_video_plane_flag may be signaled to indicate whether to use the parameters of the YUV channel.

[0180] When separate_video_plane_flag is 0, the video decoder derives the parameters of the additional channel from the parameters used in the restoration of the Y, U, and V channels when restoring the additional channel. For example, the CTU size of the additional channel, CtbWidthV and CtbHeightV, are derived as shown in Equation 2 using CtbSizeY, which is the CTU size of the Y channel.

[0181]

[0182] In mathematical formula 2, SubWidthV and SubHeightV represent the width and height ratios between the image arrays of the additional channel and the Y channel.

[0183] If separate_video_plane_flag is 1, the video decoder does not use the parameters used in the reconstruction of the Y, U, and V channels when reconstructing additional channels. The video decoder may treat each channel as a monochrome video and use individually derived parameters. For example, CtbWidthV and CtbHeightV are not derived from CtbSizeY, CtbSizeCb, or CtbSizeCr.

[0184] In addition to methods for controlling the dependency and independence of the decoding process between channels, methods for grouping multiple channels and individually controlling the dependency and independence of the decoding process between groups can be utilized. For example, when restoring YUV channels, the association between YUV channels is utilized; however, when restoring a YUV channel group and an additional channel group, the association between the two groups is not utilized, and each group can be restored individually.

[0185] As another example, the video decoder can determine the independent / dependent tree structure of each channel.

[0186] In VVC, the luminance and chroma components may utilize separate tree structures. Alternatively, following the tree structure of the Y channel, both UV channels may utilize the same tree structure as the Y channel. In other words, a dependency may exist between the Y channel and the UV channel. With respect to the newly added A channel, the image decoder may utilize a tree structure that controls the independence and dependency between channels. In this case, one of the following methods may be used.

[0187] ① All channels except the Y channel can be used as dependent channels. For example, a single tree structure can be used. Just as the same block partitioning structure is used for the luminance and chroma components, the same tree structure is used for additional channels. That is, both the UV channel and the A channel are dependent on the Y channel, and all channels except the Y channel become dependent channels.

[0188] ② Some channels may be used as dependent channels, and some channels may be used as independent channels. For example, a dual tree structure may be used. Alternatively, different CTU partitioning structures may be utilized for the luminance and chroma components. For additional channels, the same tree structure as that for the luminance or chroma components is used. That is, the Y channel and the UV channel become independent channels. On the other hand, the A channel may become a dependent channel of the Y channel or a dependent channel of the UV channel. In this case, separate tree structures, such as dual tree structures or independent partitioning structures related to the Y channel and the UV channel, may be applied to intra prediction or inter prediction. When a separate tree structure is applied to inter prediction, motion information of the luminance region existing at the same location may be used as motion information (e.g., motion vector or block vector) of the chroma block being inter-predicted.

[0189] ③ All channels may be used as independent channels. For example, a triple tree structure may be used. That is, a dual tree structure may be used for the Y and UV channels. Alternatively, different CTU or block partitioning structures may be utilized for the luminance channel, chroma channel, and additional channels. In this case, separate tree structures, such as dual tree structures or independent partitioning structures related to the luminance and chroma channels, may be applied to intra-prediction or inter-prediction. When a separate tree structure is applied to inter-prediction, motion information of a luminance region existing at the same location may be used as motion information (e.g., motion vector or block vector) for the chroma block being inter-predicted.

[0190] The tree structure according to ①, ②, and ③ can be displayed in higher-level syntax such as VPS, SPS, PPS, etc.

[0191] Hereinafter, a method for restoring an image including additional channels using the illustration of FIG. 10 is described. The illustration of FIG. 10 can be performed by an image decoding device, and if necessary, the operation by an image encoding device is additionally described.

[0192] FIG. 10 is a flowchart illustrating a method for restoring an image according to one embodiment of the present disclosure.

[0193] The video encoding device acquires a luminance channel and a chroma channel of an image. Here, the chroma channel includes a first chroma channel and a second chroma channel. The luminance channel, the first chroma channel, and the second chroma channel may represent a Y channel, a Cb channel, and a Cr channel, respectively. The video encoding device encodes the luminance channel and the chroma channel.

[0194] The video encoding device checks whether the video contains additional channels. There may be one or more additional channels. For example, the video encoding device can check whether the video contains additional channels by obtaining a channel index indicating the additional channels. If the video contains additional channels, the video encoding device can encode the additional channels. The video encoding device can generate a bitstream containing the encoded luminance channel, chroma channel, and additional channels, and transmit the generated bitstream to the video decoder.

[0195] The image decoder acquires a bitstream including a luminance channel and a chroma channel of the image (S1000). Here, the chroma channel includes a first chroma channel and a second chroma channel. The luminance channel, the first chroma channel, and the second chroma channel may represent a Y channel, a Cb channel, and a Cr channel, respectively.

[0196] The video decoder recovers the luminance channel and chroma channel from the bitstream (S1002).

[0197] The video decoder checks whether the video contains additional channels (S1004). There may be one or more additional channels.

[0198] As an example, an image decoder can determine whether an image contains additional channels by decoding a channel index that indicates additional channels. The channel index can directly indicate additional channels as shown in Table 2 or Table 3.

[0199] As another example, as shown in Table 4, the channel index indicates the video format of the image, and the video format can define the sampling format associated with the luminance channel and chroma channel, and / or the sampling format associated with additional channels.

[0200] As an example, an image decoder can determine whether an image contains additional channels by decoding a chroma format index that indicates the chroma format of the image. As shown in Table 5, the chroma format index defines the sampling format associated with the luminance channel and the chroma channel, and / or the number of additional channels and the sampling format associated with the additional channels.

[0201] A video decoder can determine whether an additional channel is included in the video by decoding a chroma format index indicating the chroma format of the video and a flag indicating whether the channels are processed separately. As shown in Table 6, the chroma format index and the aforementioned flag define the number of additional channels and whether the additional channels are processed separately. If the aforementioned flag does not indicate separate processing of the additional channels, the sampling format of the additional channels may follow the sampling format of the first chroma channel or the sampling format of the second chroma channel.

[0202] If the video contains additional channels (Yes of S1004), the video decoder restores the additional channels (S1006).

[0203] As an example, the video decoding device can make the image size of an additional channel equal to the image size of a first chroma channel or a second chroma channel.

[0204] When the image size of the luminance channel is greater than the sum of the image size of the first chroma channel, the image size of the second chroma channel, and the image size of the additional channel, the image decoder may keep the remaining area of ​​the image filled with the first chroma channel, the second chroma channel, and the additional channel empty. In this case, the remaining area may be filled with a value of 0 or 2Bitdepth-1.

[0205] As an example, parameters of additional channels can be derived based on the parameters of a luminance channel, a first chroma channel, or a second chroma channel.

[0206] As one example, the chroma channel and additional channel may have a tree structure dependent on the lumina channel. As another example, the lumina channel and the chroma channel may each have a separate tree structure, and the additional channel may have a tree structure dependent on the lumina channel. As yet another example, the lumina channel, the chroma channel, and the additional channel may each have a separate tree structure. If the lumina channel and the chroma channel have separate tree structures, the aforementioned separate tree structures may be applied to intra prediction or inter prediction. If the separate tree structures are applied to inter prediction, the motion information of the lumina region existing at the same location may be used as the motion information (e.g., motion vector or block vector) of the chroma block being inter-predicted.

[0207] If the video does not include additional channels (No. of S1004), the restoration of additional channels may be omitted.

[0208] Although the flowcharts and timing diagrams in this specification describe each process as being executed sequentially, this is merely an illustrative explanation of the technical concept of one embodiment of the present disclosure. In other words, a person skilled in the art to which one embodiment of the present disclosure belongs may modify and adapt the flowcharts and timing diagrams in various ways, such as changing the order described in the flowcharts and timing diagrams or executing one or more of the processes in parallel, without departing from the essential characteristics of one embodiment of the present disclosure; therefore, the flowcharts and timing diagrams are not limited to a chronological order.

[0209] It should be understood that the exemplary embodiments described above may be implemented in many different ways. The functions or methods described in one or more examples may be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "...unit" to particularly emphasize their implementation independence.

[0210] Meanwhile, the various functions or methods described in the present embodiment may be implemented as instructions stored in a non-transient recording medium that can be read and executed by one or more processors. A non-transient recording medium includes, for example, any type of recording device in which data is stored in a form readable by a computer system. For example, a non-transient recording medium includes storage media such as an EPROM (erasable programmable read-only memory), a flash drive, an optical drive, a magnetic hard drive, and a solid-state drive (SSD).

[0211] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment.

[0212]

[0213] CROSS-REFERENCE TO RELATED APPLICATION

[0214] This patent application claims priority to Korean patent application No. 10-2024-0146703 filed on October 24, 2024, the entire contents of which are incorporated into this patent application by reference.

Claims

1. A method for restoring an image performed by an image decoding device, A step of acquiring a bitstream including a luminance channel and a chroma channel of the above image, wherein the chroma channel includes a first chroma channel and a second chroma channel; and Step of restoring the luminance channel and the chroma channel from the bitstream A method including 2. In Paragraph 1, It further includes a step of checking whether the above bitstream includes an additional channel, and A method comprising the step of recovering the additional channel from the bitstream when the additional channel is included in the bitstream.

3. In Paragraph 2, The step of checking whether the additional channel is included in the bitstream above is, A method for decrypting a channel index indicating the above additional channel.

4. In Paragraph 3, The above channel index indicates the video format of the above image, and A method in which the above video format defines a sampling format associated with the above additional channel.

5. In Paragraph 2, The step of checking whether the above additional channel is included is, Decoding a chroma format index indicating the chroma format of the above image, and A method in which the above chroma format index defines the number of additional channels and the sampling format associated with the additional channels.

6. In Paragraph 2, The step of checking whether the above additional channel is included is, Decoding a chroma format index indicating the chroma format of the above image and a flag indicating whether the channels are processed separately, and A method in which the above chroma format index and the above flag define the number of additional channels and whether the additional channels are processed separately.

7. In Paragraph 6, A method in which, if the above flag does not indicate separate processing of the above additional channel, the sampling format of the above additional channel follows the sampling format of the first chroma channel or the sampling format of the second chroma channel.

8. In Paragraph 2, The step of restoring the above additional channel is, The image size of the additional channel is made equal to the image size of the first chroma channel or the image size of the second chroma channel, and A method for keeping the remaining area of ​​the image filled with the first chroma channel, the second chroma channel, and the additional channel empty when the image size of the above-mentioned luma channel is greater than the sum of the image size of the above-mentioned first chroma channel, the image size of the above-mentioned second chroma channel, and the image size of the above-mentioned additional channel.

9. In Paragraph 2, The step of restoring the above additional channel is, A method for deriving parameters of an additional channel based on parameters of the above-mentioned luminance channel, the above-mentioned first chroma channel, or the above-mentioned second chroma channel.

10. In Paragraph 2, A method in which each of the above-mentioned luminance channel and the above-mentioned chroma channel has a separate tree structure, and the above-mentioned additional channel has a tree structure dependent on the above-mentioned luminance channel.

11. In Paragraph 2, A method in which each of the above-mentioned luminance channel, the above-mentioned chroma channel, and the above-mentioned additional channel has a separate tree structure.

12. A method for encoding an image performed by an image encoding device, A step of acquiring a luminance channel and a chroma channel of the above image, wherein the chroma channel includes a first chroma channel and a second chroma channel; and A step of encoding the above-mentioned luminance channel and the above-mentioned chroma channel A method including 13. In Paragraph 12, It further includes a step of checking whether additional channels are included in the above video, and A method comprising the step of encoding the additional channel when the additional channel is included in the above image.

14. In Paragraph 13, The step of checking whether the above additional channel is included in the above image is, A method for obtaining a channel index indicating the additional channel above.

15. In Paragraph 14, The above channel index indicates the video format of the above image, and A method in which the above video format defines a sampling format associated with the above additional channel.

16. A method for providing video data to a video decoder, A step of encoding the above video data into a bitstream; and Step of transmitting the above bitstream to the above video decoder Includes, The step of encoding the above video data is, A step of acquiring a luminance channel and a chroma channel of an image, wherein the chroma channel includes a first chroma channel and a second chroma channel; and A step of encoding the above-mentioned luminance channel and the above-mentioned chroma channel A method including

Images