Video signal decoding, encoding and transmission method and medium for storing video signal

Abstract

A method for decoding, encoding and transmitting a video signal and a medium for storing the video signal. The decoding method comprises: obtaining transform index information indicating a transform kernel of a current block from a video signal; determining the size of a non-zero region for an inverse main transform based on the transform kernel indicated by the transform index information and the size of the current block; and performing the inverse main transform based on coefficients in the non-zero region for the inverse main transform by using the transform kernel indicated by the transform index information, wherein the size of the non-zero region is determined to be 32x32 based on the size of the current block being 32x32 and the transform kernel applied to the current block belonging to a first transform group, and the size of the non-zero region is determined to be 16x16 based on the size of the current block being 32x32 and the transform kernel applied to the current block belonging to a second transform group.

Description

[0001] This application is a divisional application of the invention patent application with the original application number 201980021153.5 (International application number: PCT / KR2019 / 006675, application date: June 3, 2019, invention name: Method and device for processing video signals by using reduced transformation). Technical Field

[0002] The present invention relates to methods and apparatus for processing video signals, and more particularly to a method of designing and applying a reducing transform that can be applied to a main transform. Background Art

[0003] The next generation of video content will have features such as high spatial resolution, high frame rate, and high-dimensional scene presentation. In order to process these contents, it will cause a sharp increase in memory storage, memory access rate, and processing power.

[0004] Therefore, there is a need to design new coding tools for more efficiently processing next generation video content. Specifically, when transforms are applied, there is a need to design more efficient transforms in terms of coding efficiency and complexity. Summary of the invention

[0005] Technical issues

[0006] The object of the present invention is to propose a method for performing a primary transformation on a predefined area according to certain conditions.

[0007] The technical objectives of the present invention are not limited to the above-mentioned technical objectives, and other technical objectives not mentioned above will become apparent to those skilled in the art from the following description.

[0008] Technical Solution

[0009] In one aspect of the present invention, a method for decoding a video signal based on a reduced transform is provided, the method comprising the following steps: checking whether a transform skip is applied to a current block; when the transform skip is not applied to the current block, obtaining a transform index indicating a transform kernel of the current block from the video signal; determining an area where a main transform is applied to the current block based on the transform kernel indicated by the transform index and the size of the current block; and performing an inverse main transform on the area to which the main transform is applied by using the transform kernel indicated by the transform index.

[0010] Preferably, determining the region to which the main transform is applied may include considering coefficients of remaining regions except the region to which the main transform is applied in the current block as zero.

[0011] Preferably, determining the area to which the main transform is applied is performed by the following steps: when the transform kernel indicated by the transform index is a predefined transform and the width and / or height of the current block is greater than a predefined size, determining an area with a width and / or height of a predefined size as the area to which the main transform is applied.

[0012] Preferably, the predefined transform may be any one of a plurality of transform combinations configured by a combination of DST7 and / or DCT8.

[0013] Preferably, the predefined size may be 16.

[0014] Preferably, determining the area to which the main transform is applied is performed by the following steps: when the transform kernel indicated by the transform index belongs to the first transform group, determining the smaller value between the width of the current block and 32 as the width of the area to which the main transform is applied, and determining the smaller value between the height of the current block and 32 as the height of the area to which the main transform is applied, and when the transform kernel indicated by the transform index belongs to the second transform group, determining the smaller value between the width of the current block and 16 as the width of the area to which the main transform is applied, and determining the smaller value between the height of the current block and 16 as the height of the area to which the main transform is applied.

[0015] Preferably, the first transform group may include DCT2, and the second transform group may include a plurality of transform combinations configured by a combination of DST7 and / or DCT8.

[0016] In another aspect of the present invention, a device for decoding a video signal based on a reduced transform is provided, the device comprising: a transform skip checking unit, which checks whether a transform skip is applied to a current block; a transform index obtaining unit, which obtains a transform index indicating a transform kernel of the current block from the video signal when the transform skip is not applied to the current block; and a main inverse transform unit, which determines an area to apply a main transform to the current block based on the transform kernel indicated by the transform index and the size of the current block, and performs an inverse main transform by using the transform kernel indicated by the transform index relative to the area to which the main transform is applied.

[0017] Preferably, the main transform unit may regard coefficients of remaining areas except an area to which the main transform is applied in the current block as zero.

[0018] Preferably, when the transform kernel indicated by the transform index is a predefined transform and the width and / or height of the current block is greater than a predefined size, the main transform unit may determine an area having a width and / or height of a predefined size as an area to which the main transform is applied.

[0019] Preferably, the predefined transform may be any one of a plurality of transform combinations configured by a combination of DST7 and / or DCT8.

[0020] Preferably, the predetermined size may be 16.

[0021] Preferably, the main transform unit may determine the smaller value between the width of the current block and 32 as the width of the area to which the main transform is applied, and determine the smaller value between the height of the current block and 32 as the height of the area to which the main transform is applied when the transform kernel indicated by the transform index belongs to the first transform group, and determine the smaller value between the width of the current block and 16 as the width of the area to which the main transform is applied, and determine the smaller value between the height of the current block and 16 as the height of the area to which the main transform is applied when the transform kernel indicated by the transform index belongs to the second transform group.

[0022] Preferably, the first transform group may include DCT2, and the second transform group may include a plurality of transform combinations configured by a combination of DST7 and / or DCT8.

[0023] Beneficial Effects

[0024] According to an embodiment of the present invention, only the predefined area is transformed according to certain conditions, thereby significantly reducing the complexity.

[0025] Advantages obtainable in the present invention are not limited to the above-mentioned effects, and other unmentioned advantages will be clearly understood by those skilled in the art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic block diagram of an encoder that performs video signal encoding as an embodiment to which the present invention is applied.

[0027] Figure 2 is a schematic block diagram of a decoder that performs video signal decoding as an embodiment to which the present invention is applied.

[0028] Figure 3a , Figure 3b , Figure 3c and Figure 3d The embodiments to which the present invention can be applied are illustrated. Figure 3a is a diagram describing a block partition structure in a quadtree (QT) (hereinafter, referred to as "QT"), Figure 3b is a diagram describing a block partition structure as a binary tree (BT) (hereinafter, referred to as "BT"), Figure 3c is a diagram describing a block partition structure in a ternary tree (TT) (hereinafter, referred to as "TT"), and Figure 3d is a diagram describing a block partition structure in an asymmetric tree (AT) (hereinafter, referred to as "AT").

[0029] Figure 4 1 is a schematic block diagram of a transform unit 120 and a quantization unit 130 , a dequantization unit 140 , and an inverse transform unit 150 in an encoder as an embodiment to which the present invention is applied.

[0030] Figure 5 2 is a schematic block diagram of a dequantization unit 220 and an inverse transform unit 230 in a decoder as an embodiment to which the present invention is applied.

[0031] Figure 6 : is a table showing a transform configuration group to which multiple transform selection (MTS) is applied as an embodiment to which the present invention is applied.

[0032] Figure 7 : is a flowchart showing an encoding process of performing multiple transform selection (MTS) as an embodiment to which the present invention is applied.

[0033] Figure 8 : is a flowchart showing a decoding process of performing multiple transform selection (MTS) as an embodiment to which the present invention is applied.

[0034] Fig. 9 : is a flowchart for describing a process of encoding an MTS flag and an MTS index as an embodiment to which the present invention is applied.

[0035] Fig.10 : is a flowchart for describing a decoding process of applying horizontal transform or vertical transform to a row or column based on an MTS flag and an MTS index as an embodiment to which the present invention is applied.

[0036] Fig.11 This is a flowchart of performing inverse transformation based on transformation-related parameters as an embodiment to which the present invention is applied.

[0037] Fig.12 : is a table showing allocation of a transform set for each intra prediction mode in NSST as an embodiment to which the present invention is applied.

[0038] Fig.13 This is a calculation flowchart of Givens rotation as an embodiment to which the present invention is applied.

[0039] Fig.14 As an embodiment to which the present invention is applied, a one round configuration in a 4×4 NSST composed of a Givens rotation layer and a permutation is illustrated.

[0040] Fig.15 is a block diagram for describing operations of forward reduction transform and inverse reduction transform as an embodiment to which the present invention is applied.

[0041] Fig.16 1 is a diagram illustrating a process of performing backward scanning from the 64th to the 17th according to the backward scanning order as an embodiment to which the present invention is applied.

[0042] Fig.17 Three forward scanning orders of a transform coefficient block (transform block) are exemplified as an embodiment to which the present invention is applied.

[0043] Fig.18 The positions and forward scanning order of significant transform coefficients of each 4×4 block when diagonal scanning is applied and 4×4 RST is applied in the upper left 4×8 block as an embodiment to which the present invention is applied are illustrated.

[0044] Fig.19 A case where significant transform coefficients of two 4×4 blocks are combined into one 4×4 block when diagonal scanning is applied and 4×4 RST is applied in the upper left 4×8 block is illustrated as an embodiment to which the present invention is applied.

[0045] Fig. 20 The flowchart of encoding a video signal based on a reduced quadratic transform as an embodiment of the present invention is shown.

[0046] Fig.21 The flowchart of decoding a video signal based on a reduced quadratic transform is an embodiment to which the present invention is applied.

[0047] Fig. 22 FIG. 1 is a diagram illustrating a method of encoding a video signal by using a reduced transform as an embodiment to which the present invention is applied.

[0048] Fig.23 FIG. 1 is a diagram illustrating a method of decoding a video signal by using a reduced transform as an embodiment to which the present invention is applied.

[0049] Fig.24 FIG. 1 is a diagram illustrating a reduced transformation structure based on a reduction factor as an embodiment to which the present invention is applied.

[0050] Fig.25 : is a diagram illustrating a method of performing decoding by adaptively applying a reduced transform as an embodiment to which the present invention can be applied.

[0051] Fig.26 : is a diagram illustrating a method of performing decoding by adaptively applying a reduced transform as an embodiment to which the present invention can be applied.

[0052] Fig. 27and Fig.28 is a diagram illustrating an example of a forward reduced quadratic transform and an inverse reduced quadratic transform and pseudo codes for deriving them.

[0053] Fig.29 FIG. 1 is a diagram illustrating a method of applying a reduced quadratic transform to a non-square region as an embodiment to which the present invention is applied.

[0054] Fig.30 FIG. 1 is a diagram illustrating a reduction transformation controlled by a reduction factor as an embodiment to which the present invention is applied.

[0055] Fig.31 is a diagram illustrating an inverse transform unit according to an embodiment to which the present invention is applied.

[0056] Fig.32 A video encoding system to which the present invention is applied is exemplified.

[0057] Fig.33 It is a structural diagram of a content streaming system as an embodiment to which the present invention is applied. DETAILED DESCRIPTION

[0058] Hereinafter, the configuration and operation of an embodiment of the present invention will be described with reference to the accompanying drawings, and the configuration and operation of the present invention described by the accompanying drawings will be described as an embodiment, thereby the technical spirit of the present invention and its core composition and its operation are not limited.

[0059] In addition, the terms used in the present invention are selected as common terms widely used now as much as possible, and in specific cases, the terms arbitrarily selected by the applicant will be used for description. In this case, since the meaning is clearly explained in the detailed description of this part, it should not be interpreted only by the name of the term used in the description of the present invention, and it should be understood that the meaning of the term should be interpreted.

[0060] In addition, when selecting general terms or other terms with similar meanings for describing the present invention, the terms used in the present invention may be replaced for more appropriate interpretation. For example, signals, data, samples, pictures, frames, blocks, etc. may be appropriately replaced and interpreted in each encoding process. In addition, separation, decomposition, segmentation, and division may be appropriately replaced and interpreted in each encoding process.

[0061] In this document, multiple transform selection (MTS) may refer to a method of performing a transform using at least two transform types. This may also be expressed as adaptive multiple transform (AMT) or explicit multiple transform (EMT), and similarly, mts_idx may also be expressed as AMT_idx, EMT_idx, tu_mts_idx, AMT_TU_idx, EMT_TU_idx, transform index, or transform combination index, and the present invention is not limited to these expressions.

[0062] Figure 1 is a schematic block diagram of an encoder that performs video signal encoding as an embodiment to which the present invention is applied.

[0063] Reference Figure 1 , the encoder 100 can be configured to include an image division unit 110, a transformation unit 120, a quantization unit 130, a dequantization unit 140, an inverse transformation unit 150, a filtering unit 160, a decoded picture buffer (DPB) 170, an inter-frame prediction unit 180, an intra-frame prediction unit 185 and an entropy encoding unit 190.

[0064] The image division unit 110 divides the input image (or picture or frame) input to the encoder 100 into one or more processing units. For example, the processing unit may be a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0065] However, these terms are only for the convenience of describing the present invention, and the present invention is not limited to the definitions of these terms. In addition, in this specification, for the convenience of description, the term "coding unit" is used as a unit used when encoding or decoding a video signal, but the present invention is not limited thereto and can be appropriately interpreted according to the present invention.

[0066] The encoder 100 subtracts a prediction signal (or a prediction block) output from the inter prediction unit 180 or the intra prediction unit 185 from the input image signal to generate a residual signal (or a residual block), and the generated residual signal is transmitted to the transform unit 120 .

[0067] The transform unit 120 may generate a transform coefficient by applying a transform technique to the residual signal.The transform process may be applied to a quadtree structure square block and a block (square or rectangular) partitioned in a binary tree structure, a ternary tree structure, or an asymmetric tree structure.

[0068] The transform unit 120 may perform transform based on multiple transforms (or transform combinations), and the transform scheme may be referred to as Multiple Transform Selection (MTS). MTS may also be referred to as Adaptive Multiple Transform (AMT) or Enhanced Multiple Transform (EMT).

[0069] MTS (or AMT or EMT) may refer to a transform scheme performed based on a transform (or a transform combination) adaptively selected from a plurality of transforms (or transform combinations).

[0070] Multiple transformations (or combinations of transformations) may be included in the description. Figure 6 In this specification, a transform or transform type may be denoted as, for example, DCT-type 2, DCT-II, DCT2, or DCT-2.

[0071] The transform unit 120 may perform the following implementations.

[0072] The present invention provides a method for designing a RST that can be applied to 4×4 blocks.

[0073] The present invention provides a configuration of an area to which 4×4 RST is applied, a method of arranging transform coefficients generated after applying 4×4 RST, a scanning order of the arranged transform coefficients, a method for sorting and combining transform coefficients generated for each block, and the like.

[0074] The present invention provides a method for encoding a transform index of a specified 4×4 RST.

[0075] The present invention provides a method for conditionally encoding a corresponding transform index by checking whether a non-zero transform coefficient exists in an unacceptable region when applying 4×4 RST.

[0076] The present invention provides a method for conditionally encoding the corresponding transform index after encoding the last non-zero transform coefficient position and then omitting the encoding of the associated residuals of the unaccepted positions.

[0077] The present invention provides a method for applying different transform index coding and residual coding to a luminance block and a chrominance block when a 4×4 RST is applied.

[0078] The detailed implementation thereof will be described in more detail in this specification.

[0079] The quantization unit 130 may quantize the transform coefficient and transmit the quantized transform coefficient to the entropy encoding unit 190 , and the entropy encoding unit 190 may entropy encode the quantized signal and output the entropy-encoded quantized signal as a bitstream.

[0080] Although the transformer 120 and the quantization unit 130 are described as separate functional units, the present invention is not limited thereto and they may be combined into one functional unit. The dequantization unit 140 and the inverse transform unit 150 may also be similarly combined into one functional unit.

[0081] The quantized signal output from the quantization unit 130 may be used to generate a prediction signal. For example, a residual signal is reconstructed by cyclically applying dequantization and inverse transformation to the quantized signal through the dequantization unit 140 and the inverse transformation unit 150. The reconstructed residual signal is added to the prediction signal output from the inter prediction unit 180 or the intra prediction unit 185 to generate a reconstructed signal.

[0082] In addition, due to the quantization error that occurs during this compression process, degradation showing block boundaries may occur. This phenomenon is called blocking artifact, which is one of the key elements for evaluating image quality. In order to reduce the degradation, a filtering process can be performed. Through the filtering process, block degradation is eliminated and the error of the current picture is reduced to enhance the image quality.

[0083] The filtering unit 160 applies filtering to the reconstructed signal and outputs the applied reconstructed signal to the reproduction device, or transmits the output reconstructed signal to the decoded picture buffer 170. The inter-frame prediction unit 170 may use the filtered signal transmitted to the decoded picture buffer 180 as a reference picture. In this way, in the inter-picture prediction mode, the filtered picture is used as a reference picture to enhance image quality and encoding efficiency.

[0084] The decoded picture buffer 170 may store the filtered picture in order to use the filtered picture as a reference picture in the inter prediction unit 180 .

[0085] The inter-frame prediction unit 180 performs temporal prediction and / or spatial prediction to remove temporal redundancy and / or spatial redundancy by referring to the reconstructed picture. Here, since the reference picture used for prediction is a transform signal quantized and inversely quantized in units of blocks when encoding / decoding at the previous time, there may be a blocking effect or a ringing effect.

[0086] Therefore, the inter-frame prediction unit 180 can interpolate signals between pixels in sub-pixel units by applying a low-pass filter to solve the performance degradation caused by discontinuity or quantization of such signals. Here, sub-pixel means a virtual pixel generated by applying an interpolation filter, and integer pixel means an actual pixel present in the reconstructed picture. As an interpolation method, linear interpolation, bilinear interpolation, Wiener filter, etc. can be used.

[0087] An interpolation filter is applied to the reconstructed picture to enhance prediction accuracy. For example, the inter prediction unit 180 applies an interpolation filter to integer pixels to generate interpolated pixels, and may perform prediction by using an interpolation block composed of the interpolated pixels as a prediction block.

[0088] In addition, the intra prediction unit 185 can predict the current block by referring to samples near the block that will undergo current encoding. The intra prediction unit 185 can perform the following process to perform intra prediction. First, reference samples can be prepared, which are required to generate a prediction signal. In addition, a prediction signal can be generated by using the prepared reference samples. Thereafter, the prediction mode is encoded. In this case, the reference samples can be prepared by reference sample padding and / or reference sample filtering. Since the reference samples undergo prediction and reconstruction processes, there may be quantization errors. Therefore, a reference sample filtering process can be performed relative to each prediction mode used for intra prediction in order to reduce this error.

[0089] The prediction signal generated by the inter prediction unit 180 or the intra prediction unit 185 may be used to generate a reconstructed signal or to generate a residual signal.

[0090] Figure 2 is a schematic block diagram of a decoder that performs video signal decoding as an embodiment to which the present invention is applied.

[0091] Reference Figure 2 , the decoder 200 can be configured to include a parsing unit (not illustrated), an entropy decoding unit 210, a dequantization unit 220, an inverse transform unit 230, a filtering unit 240, a decoded picture buffer (DPB) unit 250, an inter-frame prediction unit 260 and an intra-frame prediction unit 265.

[0092] In addition, the reconstructed video signal output by the decoder can be reproduced by the reproduction device.

[0093] The decoder 200 can receive Figure 1 The received signal can be entropy decoded by the entropy decoding unit 210.

[0094] The dequantization unit 220 obtains a transform coefficient from the entropy-decoded signal by using the quantization step information.

[0095] The inverse transform unit 230 performs inverse transform on the transform coefficients to obtain a residual signal.

[0096] Here, the present invention provides a method for configuring a transform combination for each transform configuration group divided by at least one of a prediction mode, a block size, or a block shape, and the inverse transform unit 230 can perform inverse transform based on the transform combination configured by the present invention. In addition, the embodiments described in this specification can be applied.

[0097] The inverse transform unit 230 may perform the following embodiments.

[0098] The present invention provides a method for reconstructing a video signal based on a reduced quadratic transform.

[0099] The inverse transform unit 230 may derive a secondary transform corresponding to the secondary transform index, perform an inverse secondary transform on the transform coefficient block by using the secondary transform, and perform an inverse main transform on the block on which the inverse secondary transform is performed. Here, the secondary transform refers to a reduced secondary transform, and the reduced secondary transform means a transform that inputs N residual data (N×1 residual vectors) to output L (L<N) transform coefficient data (L×1 transform coefficient vectors).

[0100] The present invention is characterized in that the reduced secondary transform is applied to a specific area of ​​the current block, and the specific area is an upper left M×M (M≤N) area in the current block.

[0101] The present invention is characterized in that, when performing inverse secondary transform, 4×4 reduced secondary transform is applied to each 4×4 block divided in the current block.

[0102] The present invention is characterized in that whether a secondary transform index is obtained is determined based on the position of the last non-zero transform coefficient in the transform coefficient block.

[0103] The present invention is characterized in that when the last non-zero transform coefficient is not in a specific area, a secondary transform index is obtained, and the specific area indicates the remaining area except for the position where the non-zero transform coefficient may exist when the transform coefficients are arranged according to the scanning order when a reduced secondary transform is applied.

[0104] The inverse transform unit 230 may derive a transform combination corresponding to the main transform index and perform inverse main transform by using the transform combination. Here, the main transform index corresponds to any one of a plurality of transform combinations consisting of a combination of DST7 and / or DCT8, and the transform combination includes a horizontal transform and a vertical transform. In this case, the horizontal transform and the vertical transform correspond to DST7 or DCT8.

[0105] Although the dequantization unit 220 and the inverse transform unit 230 are described as separate functional units, the present invention is not limited thereto, and they may be combined into one functional unit.

[0106] The obtained residual signal is added to the prediction signal output from the inter prediction unit 260 or the intra prediction unit 265 to generate a reconstructed signal.

[0107] The filtering unit 240 applies filtering to the reconstructed signal and outputs the applied reconstructed signal to the generating device or transmits the output reconstructed signal to the decoded picture buffer unit 250. The inter prediction unit 250 may use the filtered signal transmitted to the decoded picture buffer unit 260 as a reference picture.

[0108] In the present specification, the embodiments described in the corresponding functional units and the transform unit 120 of the encoder 100 may be equally applied to the corresponding functional units and the inverse transform unit 230 of the decoder, respectively.

[0109] Figure 3a , Figure 3b , Figure 3c and Figure 3d shows an embodiment in which the present invention can be applied, Figure 3a is a diagram describing a block partition structure in a quadtree (QT) (hereinafter, referred to as "QT"), Figure 3b is a diagram describing a block partition structure as a binary tree (BT) (hereinafter, referred to as "BT"), Figure 3c is a diagram describing a block partition structure in a ternary tree (TT) (hereinafter, referred to as "TT"), and Figure 3d is a diagram describing a block partition structure in an asymmetric tree (AT) (hereinafter, referred to as "AT").

[0110] In video encoding, a block can be divided based on a quadtree (QT). In addition, a sub-block divided by QT can be further recursively divided using QT. Leaf blocks that are no longer divided by QT can be divided by at least one of a binary tree (BT), a ternary tree (TT), and an asymmetric tree (AT). BT can have two types of division: horizontal BT (2N×N, 2N×N) and vertical BT (N×2N, N×2N). TT can have two types of division: horizontal TT (2N×1 / 2N, 2N×N, 2N×1 / 2N) and vertical TT (1 / 2N×2N, N×2N, 1 / 2N×2N). AT may have four types of divisions: horizontal-upper AT (2N×1 / 2N, 2N×3 / 2N), horizontal-lower AT (2N×3 / 2N, 2N×1 / 2N), vertical-left AT (1 / 2N×2N, 3 / 2N×2N), vertical-right AT (3 / 2N×2N, 1 / 2N×2N). Each of BT, TT, and AT may be further recursively divided by using BT, TT, and AT.

[0111] Figure 3a An example of QT division is illustrated. Block A can be divided into four sub-blocks A0, A1, A2, and A3 according to QT. Sub-block A1 can be further divided into four sub-blocks B0, B1, B2, and B3 according to QT.

[0112] Figure 3b An example of BT partitioning is illustrated. Block B3, which is no longer divided by QT, can be divided into vertical BT (C0, C1) or horizontal BT (D0, D1). Like block C0, each sub-block can be further recursively divided in the form of horizontal BT (E0, E1) or vertical BT (F0, F1).

[0113] Figure 3c An example of TT partitioning is illustrated. Block B3, which is no longer divided by QT, can be divided into vertical TT (C0, C1, C2) or horizontal TT (D0, D1, D2). Like block C1, each sub-block can be further recursively divided in the form of horizontal TT (E0, E1, E2) or vertical TT (F0, F1, F2).

[0114] Figure 3d An example of AT partitioning is illustrated. Block B3, which is no longer divided by QT, can be divided into vertical AT (C0, C1) or horizontal AT (D0, D1). Like block C1, each sub-block can be further recursively divided in the form of horizontal AT (E0, E1) or vertical TT (F0, F1).

[0115] In addition, BT, TT and AT divisions can be used and performed together. For example, a sub-block divided by BT can be divided by TT or AT. In addition, a sub-block divided by TT can be divided by BT or AT. In addition, a sub-block divided by AT can be divided by BT or TT. For example, after horizontal BT division, each sub-block can be divided into vertical BT, or after vertical BT division, each sub-block can be divided into horizontal BT. The two types of division methods are different from each other in the division order, but are the same as each other in the final division shape.

[0116] In addition, when the block is divided, the order of searching the blocks can be defined in various ways. Generally, the search can be performed from left to right and from top to bottom, and the search block can mean the order of deciding whether to further divide each divided sub-block, the encoding order of each sub-block when each sub-block is no longer divided, or the search order when referring to information of another adjacent block in the sub-block.

[0117] Figure 4 and Figure 5 The embodiment to which the present invention is applied is illustrated. Figure 4 is a schematic block diagram of a transform unit 120 and a quantization unit 130, a dequantization unit 140 and an inverse transform unit 150 in an encoder, and Figure 5 is a schematic block diagram of the dequantization unit 220 and the inverse transform unit 230 in the decoder.

[0118] Reference Figure 4 The transform unit 120 and the quantization unit 130 may include a primary transform unit 121, a secondary transform unit 122, and a quantization unit 130. The dequantization unit 140 and the inverse transform unit 150 may include a dequantization unit 140, an inverse secondary transform unit 151, and an inverse primary transform unit 152.

[0119] Reference Figure 5, the dequantization unit 220 and the inverse transform unit 230 may include a dequantization unit 220 , an inverse secondary transform unit 231 , and an inverse main transform unit 232 .

[0120] In the present invention, when performing a transformation, the transformation can be performed through multiple steps. For example, Figure 4 In the example, two steps of primary transform and secondary transform are applied, or more transform steps may be used depending on the algorithm. Here, the primary transform may also be referred to as a core transform.

[0121] The main transform unit 121 may apply a main transform to the residual signal, and here, the main transform may be defined in a table of an encoder and / or a decoder.

[0122] The main transform may employ discrete cosine transform type 2 (hereinafter, referred to as "DCT2").

[0123] Alternatively, only in specific cases, discrete sine transform type 7 (hereinafter, referred to as "DST7") may be adopted. For example, DST7 may be applied to a 4×4 block in an intra prediction mode.

[0124] In addition, the main transform can use a combination of various transforms DST 7, DCT 8, DST 1 and DCT5 of multiple transform selection (MTS). For example, Figure 6 .

[0125] The secondary transform unit 122 may apply a secondary transform to the primary transformed signal, and here, the secondary transform may be defined in a table of an encoder and / or a decoder.

[0126] As an embodiment, the secondary transform may conditionally adopt a non-separable secondary transform (hereinafter, referred to as "NSST"). For example, NSST may be applied only to intra-prediction blocks and may have a transform set suitable for each prediction mode group.

[0127] Here, the prediction mode group can be configured based on symmetry with respect to the prediction direction. For example, since prediction mode 52 and prediction mode 16 are symmetric based on prediction mode 34 (diagonal direction), the same transform set can be applied by forming a group. In this case, when applying a transform for prediction mode 52, since prediction mode 52 has the same transform set as prediction mode 16, the input data is swapped and then applied.

[0128] Furthermore, since there is no directional symmetry in the case of the planar mode and the DC mode, each mode has a different transform set, and the corresponding transform set can be composed of two transforms. With respect to the remaining directional modes, each transform set can be composed of three transforms.

[0129] As another embodiment, the secondary transform may use a combination of various transforms DST 7, DCT 8, DST 1, and DCT 5 of multiple transform selection (MTS). For example, Figure 6 .

[0130] As another implementation, DST 7 may be applied to the secondary transform.

[0131] As another embodiment, the secondary transform may not be applied to the entire primary transform block but may be applied only to the upper left specific area. For example, when the block size is 8×8 or larger, 8×8 NSST is applied, and when the block size is less than 8×8, 4×4 secondary transform may be applied. In this case, the block may be divided into 4×4 blocks, and then a 4×4 secondary transform may be applied to each divided block.

[0132] As another embodiment, even in the case of 4×N / N×4 (N>=16), a 4×4 secondary transform may be applied.

[0133] Will refer to Figures 12 to 15 The secondary transform (eg, NSST), 4×4 secondary transform, and 8×8 secondary transform are described in more detail in other embodiments in the specification.

[0134] The quantization unit 130 may perform quantization on the secondarily transformed signal.

[0135] The dequantization unit 140 and the inverse transform unit 150 perform the above-described processes in reverse, and a redundant description thereof will be omitted.

[0136] Figure 5 is a schematic block diagram of the inverse transform unit 230 and the dequantization unit 220 in the decoder.

[0137] Reference Figure 5 , the dequantization unit 220 and the inverse transform unit 230 may include a dequantization unit 220 , an inverse secondary transform unit 231 , and an inverse main transform unit 232 .

[0138] The dequantization unit 220 obtains a transform coefficient from the entropy-decoded signal by using the quantization step information.

[0139] The inverse secondary transform unit 231 performs an inverse secondary transform on the transform coefficients. Here, the inverse secondary transform is represented by Figure 4 The inverse of the quadratic transform described in .

[0140] As another embodiment, the secondary transform may use a combination of various transforms DST 7, DCT 8, DST 1, and DCT 5 of multiple transform selection (MTS). For example, Figure 6 .

[0141] The inverse main transform unit 232 performs an inverse main transform on the signal (or block) after the inverse secondary transform, and obtains a residual signal. Here, the inverse main transform is represented by Figure 4 The inverse transform of the main transform described in .

[0142] As an embodiment, the main transform may use a combination of various transforms DST 7, DCT 8, DST1 and DCT 5 of the Multiple Transform Selection (MTS). For example, Figure 6 .

[0143] As an embodiment of the present invention, DST 7 may be applied to the main transformation.

[0144] As an embodiment of the present invention, DST 8 may be applied to the main transform.

[0145] The present invention provides a method for configuring a transform combination for each transform configuration group classified by at least one of a prediction mode, a block size, or a block shape, and the inverse main transform unit 232 can perform inverse transform based on the transform combination configured by the present invention. In addition, the embodiments described in this specification can be applied.

[0146] Figure 6 : is a table showing a transform configuration group to which multiple transform selection (MTS) is applied as an embodiment to which the present invention is applied.

[0147] Applying a transform configuration group with Multiple Transform Selection (MTS)

[0148] In this specification, the configuration group G is used to transform i The j-th transform combination candidate of is represented by the pair shown in the following Equation 1.

[0149] [Formula 1]

[0150] (H(G i ,j),V(G i ,j))

[0151] Here, H(G i ,j) represents the jth candidate horizontal transformation, and V(G i ,j) represents the jth candidate vertical transformation. For example, Figure 6 In this case, we can express H(G 3 ,2)=DST7、V(G 3 ,2)=DCT8. According to the context, it is assigned to H(G i ,j) or V(G i ,j) may be a nominal value to distinguish the transformation, as in the above example, or may be an index value indicating the transformation, or may be a two-dimensional (D) matrix for the transformation.

[0152] In addition, in the present specification, 2D matrix values ​​of DCT and DST may be expressed as shown in the following Equations 2 and 3.

[0153] [Formula 2]

[0154] DCT Type 2: DCT Type 8:

[0155] [Formula 3]

[0156] DST Type 7: DST Type 4:

[0157] Here, whether it is DST or DCT is indicated by S or C, the type number is indicated in Roman numerals in the superscript, and the N in the subscript indicates N×N transform. and Such a 2D matrix assumes that the column vectors form the transformation basis.

[0158] Reference Figure 6 , the transform configuration group can be determined based on the prediction mode, and the number of groups can be a total of six groups G0 to G5. In addition, G0 to G4 correspond to the case where intra prediction is applied, and G5 represents a transform combination (or a transform set and a transform combination set) applied to a residual block generated by inter prediction.

[0159] A transform combination may consist of a horizontal transform (or row transform) applied to rows of the corresponding 2D block and a vertical transform (or column transform) applied to columns.

[0160] Here, each of all transform configuration groups may have four transform combination candidates. The four transform combinations may be selected or determined by transform combination indexes 0 to 3, and the four transform combinations may be transmitted from the encoder to the decoder by encoding the transform combination indexes.

[0161] As an implementation method, the residual data (or residual signal) obtained by intra prediction may have different statistical characteristics according to the intra prediction mode. Figure 6 As illustrated in , a transform other than a general cosine transform may be applied to each intra prediction mode.

[0162] Reference Figure 6, illustrating the case where 35 intra prediction modes are used and the case where 67 intra prediction modes are used. Multiple transform combinations can be applied to each transform configuration group divided in each intra prediction mode column. For example, these multiple transform combinations can be composed of four (row direction transform and column direction transform) combinations. As a specific example, DST-7 and DST-5 can be applied in the row (horizontal) direction and column (vertical) direction of group 0, resulting in a total of four combinations available.

[0163] Since the total number of transform kernel combinations can be applied to each intra prediction mode, a transform combination index for selecting one of the transform kernel combinations can be transmitted per transform unit. In this specification, the transform combination index may be referred to as an MTX index and is denoted as mtx_idx.

[0164] In addition, in addition to Figure 6 In addition to the transform kernel represented in , it is also possible that the DCT2 is optimal for both the row direction and the column direction due to the characteristics of the residual signal. Therefore, an MTS flag is defined for each coding unit to adaptively perform the transform. Here, when the MTS flag is 0, DCT-2 can be applied to both the row direction and the column direction, and when the MTS flag is 1, one of the four combinations can be selected or determined by the MTS index.

[0165] As an implementation, when the MTS flag is 1, if the number of non-zero transform coefficients for a transform unit is not greater than the threshold, Figure 6 In the case of a transform kernel of , DST-7 is applied in both the row direction and the column direction. For example, the threshold value may be set to 2, which may be set differently based on the block size or the size of the transform unit. This also applies to other embodiments in the specification.

[0166] As an embodiment, if the number of non-zero transform coefficients is not greater than a threshold, by first parsing the transform coefficient values, the amount of additional information transmission can be reduced by applying DST-7 without parsing the MTS index.

[0167] As an implementation, when the MTS flag is 1, if the number of non-zero transform coefficients is greater than a threshold of one transform unit, the MTS index may be parsed, and the horizontal transform and the vertical transform may be determined based on the MTS index.

[0168] As an implementation, MTS may be applied only when both the width and height of the transform unit are equal to or less than 32.

[0169] As an implementation method, the pre-configuration can be performed through offline training. Figure 6 .

[0170] As an embodiment, the MTS index may be defined as one index that may indicate both horizontal transform and vertical transform. Alternatively, the MTS index may be defined as a horizontal transform index and a vertical transform index, respectively.

[0171] As an implementation, an MTS flag or an MTX index may be defined at at least one level in a sequence, a picture, a slice, a block, a coding unit, a transform unit, or a prediction unit. For example, an MTS flag or an MTX index may be defined at at least one level in a sequence parameter set (SPS), a coding unit, or a transform unit. In addition, as an example, a syntax flag for enabling / disabling MTX may be defined at at least one level in a sequence parameter set (SPS), a picture parameter set (PPS), or a slice header.

[0172] As another embodiment, a transform combination (horizontal transform or vertical transform) corresponding to a transform index may be configured without relying on an MTS flag, a prediction mode, and / or a block shape. For example, the transform combination may be configured by at least one of DCT2, DST7, and / or DCT8. As a specific example, when the transform index is 0, 1, 2, 3, or 4, each transform combination may be (DCT2, DCT2), (DST7, DST7), (DCT8, DST7), (DST7, DCT8), or (DCT8, DCT8).

[0173] Figure 7 : is a flowchart showing an encoding process of performing multiple transform selection (MTS) as an embodiment to which the present invention is applied.

[0174] In the present specification, an embodiment in which transforms are applied to the horizontal direction and the vertical direction separately is basically described, but a transform combination may be configured as an inseparable transform.

[0175] Alternatively, the transform combination can be configured by a mixture of separable transforms and non-separable transforms. In this case, when non-separable transforms are used, row / column transform selection or horizontal / vertical direction selection may not be required, and only when separable transforms are selected can the transform selection be used. Figure 6 transformation combination.

[0176] In addition, the scheme proposed through the present specification can be applied regardless of the main transform or the secondary transform. That is, there is no limitation that the scheme should be applied only to either one of the main transform and the secondary transform and that the scheme can be applied to both the main transform and the secondary transform. Here, the main transform may mean a transform for first transforming the residual block, and the secondary transform may mean a transform for applying the transform to the block generated as a result of the main transform.

[0177] First, the encoder can determine the transform configuration group corresponding to the current block. Here, the transform configuration group can mean Figure 6 The present invention is not limited to this, and the transformation configuration group can be composed of other transformation combinations.

[0178] The encoder may perform transforms on candidate transform combinations available in the transform configuration group (S720).

[0179] As a result of performing the transform, the encoder 100 may determine or select a transform combination having a minimum rate-distortion (RD) cost ( S730 ).

[0180] The encoder 100 may encode a transform combination index corresponding to the selected transform combination (S740).

[0181] Figure 8 : is a flowchart showing a decoding process of performing multiple transform selection (MTS) as an embodiment to which the present invention is applied.

[0182] First, the decoder may determine a transform configuration group of a current block ( S810 ).

[0183] The decoder may parse (or obtain) a transform combination index according to the video signal, and here, the transform combination index may correspond to any one of a plurality of transform combinations in the transform configuration group (S820). For example, the transform configuration group may include discrete sine transform type (DST) 7 and discrete cosine transform type (DST) 8. The transform combination index may be referred to as an MTS index.

[0184] As an implementation, the transform configuration group may be configured based on at least one of a block shape, a block size, or a prediction mode of the current block.

[0185] The decoder may derive a transform combination corresponding to the transform combination index (S830). Here, the transform combination may be composed of a horizontal transform and a vertical transform, and may include at least one of DST-7 or DCT-8.

[0186] In addition, the transformation combination can mean Figure 6 The transformation combination described in the above description is not limited to this. That is, according to other embodiments in this specification, the transformation combination can be configured by other transformation combinations.

[0187] The decoder may perform an inverse transform on the current block based on the transform combination (S840). When the transform combination consists of a row (horizontal) transform and a column (vertical) transform, the column (vertical) transform may be applied after the row (horizontal) transform is first applied. However, the present invention is not limited thereto, and the transform order may be reversed, or when the transform combination consists of an inseparable transform, the inseparable transform may be applied immediately.

[0188] As an embodiment, when the vertical transform or the horizontal transform is DST-7 or DCT-8, an inverse transform of DST-7 or an inverse transform of DCT-8 may be applied to each row.

[0189] As an embodiment, a different transform may be applied to each row and / or each column relative to a vertical transform or a horizontal transform.

[0190] As an implementation manner, the transform combination index may be acquired based on an MTS flag indicating whether MTS is performed. That is, when MTS is performed according to the MTS flag, the transform combination index may be acquired.

[0191] As an embodiment, the decoder may check whether the number of non-zero transform coefficients is greater than a threshold. In this case, a transform may be obtained when the number of non-zero transform coefficients is greater than the threshold.

[0192] As an implementation manner, an MTS flag or an MTX index may be defined at at least one level of a sequence, a picture, a slice, a block, a coding unit, a transform unit, or a prediction unit.

[0193] As an implementation, the inverse transform may be applied only when both the width and height of the transform unit are equal to or less than 32.

[0194] On the other hand, as another implementation, the process of determining the transform configuration group and the process of parsing the transform combination index may be performed simultaneously. Alternatively, step S810 may be pre-configured and omitted in the encoder and / or decoder.

[0195] Fig. 9 : is a flowchart for describing a process of encoding an MTS flag and an MTS index as an embodiment to which the present invention is applied.

[0196] The encoder may determine whether to apply multiple transform selection (MTS) to a current block ( S910 ).

[0197] When multiple transform selection (MTS) is applied, the encoder may encode the MTS flag as 1 (S920).

[0198] In addition, the encoder may determine an MTS index based on at least one of the horizontal transform, vertical transform, and prediction mode of the current block (S930). Here, the MTS index may mean an index indicating any one of a plurality of transform combinations for each intra prediction mode, and the MTS index may be transmitted for each transform unit.

[0199] When the MTS index is determined, the encoder may encode the MTS index (S940).

[0200] On the other hand, when the multiple transform selection (MTS) is not applied, the encoder may encode the MTS flag as 0 (S920).

[0201] Fig.10 : is a flowchart for describing a decoding process of applying horizontal transform or vertical transform to a row or column based on an MTS flag and an MTS index as an embodiment to which the present invention is applied.

[0202] The decoder may parse the MTS flag according to the bitstream (S1010). Here, the MTS flag may indicate whether a multiple transform selection (MTS) is applied to the current block.

[0203] The decoder may determine whether to apply multiple transform selection (MTS) to the current block based on the MTS flag (S1020). For example, it may be checked whether the MTS flag is 1.

[0204] When the MTS flag is 1, the decoder may check whether the number of non-zero transform coefficients is greater than (or equal to or greater than) a threshold value (S1030). For example, the threshold value may be set to 2, which may be set differently based on the block size or the size of the transform unit.

[0205] When the number of non-zero transform coefficients is greater than the threshold, the decoder may parse the MTS index (S1040). Here, the MTS index may mean any one of a plurality of transform combinations for each of the intra prediction mode or the inter prediction mode, and the MTS index may be transmitted for each transform unit. Alternatively, the MTS index may mean an index indicating any one of the transform combinations defined in a preconfigured transform combination table, and here, the preconfigured transform combination table may mean Figure 6 , but the present invention is not limited to this.

[0206] The decoder may derive or determine horizontal transform and vertical transform based on at least one of the MTS index and the prediction mode ( S1050 ).

[0207] Alternatively, the decoder may derive a transform combination corresponding to the MTX index. For example, the decoder may derive or determine a horizontal transform and a vertical transform corresponding to the MTS index.

[0208] When the number of non-zero transform coefficients is not greater than the threshold, the decoder may apply a pre-configured vertical inverse transform (S1060). For example, the vertical inverse transform may be an inverse transform of DST7.

[0209] In addition, the decoder may apply a pre-configured horizontal inverse transform to each row (S1070). For example, the horizontal inverse transform may be an inverse transform of DST7. That is, when the number of non-zero transform coefficients is not greater than a threshold, a transform kernel pre-configured by the encoder or decoder may be used. For example, Figure 6 The transform combinations shown in the table exemplify transform kernels that are not defined but are widely used (e.g., DCT-2, DST-7, or DCT-8).

[0210] In addition, when the MTS flag is 0, the decoder may apply a pre-configured vertical inverse transform to each column (S1080). For example, the vertical inverse transform may be an inverse transform of DCT2.

[0211] In addition, the decoder may apply a pre-configured horizontal inverse transform to each row (S1090). For example, the horizontal inverse transform may be an inverse transform of DCT2. That is, when the MTS flag is 0, a transform kernel pre-configured by the encoder or decoder may be used. For example, the horizontal inverse transform may be an inverse transform of DCT2. Figure 6 The transform combinations illustrated in the table are not defined but widely used transform kernels.

[0212] Fig.11 This is a flowchart of performing inverse transformation based on transformation-related parameters as an embodiment to which the present invention is applied.

[0213] The decoder to which the present invention is applied can obtain sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag (S1110). Here, sps_mts_intra_enabled_flag indicates whether tu_mts_flag exists in the residual coding syntax of the intra coding unit. For example, when sps_mts_intra_enabled_flag=0, tu_mts_flag does not exist in the residual coding syntax of the intra coding unit, and when sps_mts_intra_enabled_flag=0, tu_mts_flag exists in the residual coding syntax of the intra coding unit. In addition, sps_mts_inter_enabled_flag indicates whether tu_mts_flag exists in the residual coding syntax of the inter coding unit. For example, when sps_mts_inter_enabled_flag=0, tu_mts_flag does not exist in the residual coding syntax of the intra coding unit, and when sps_mts_inter_enabled_flag=0, tu_mts_flag exists in the residual coding syntax of the inter coding unit.

[0214] The decoder may obtain tu_mts_flag based on sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag (S1120). For example, when sps_mts_intra_enabled_flag=1 or sps_mts_inter_enabled_flag=1, the decoder may obtain tu_mts_flag. Here, tu_mts_flag indicates whether multiple transform selections (hereinafter referred to as "MTS") are applied to the residual samples of the luma transform block. For example, when tu_mts_flag=0, MTS is not applied to the residual samples of the luma transform block, and when tu_mts_flag=1, MTS is not applied to the residual samples of the luma transform block.

[0215] As another example, at least one of the embodiments of this document may be applied to tu_mts_flag.

[0216] The decoder may obtain mts_idx based on tu_mts_flag (S1130). For example, when tu_mts_flag=1, the decoder may obtain mts_idx. Here, mts_idx indicates which transform kernel is applied to luma residual samples in the horizontal and / or vertical direction of the current transform block.

[0217] For example, at least one of the embodiments of this document may be applied to mts_idx. As a specific example, Figure 6 At least one of the embodiments.

[0218] The decoder may derive a transform kernel corresponding to mts_idx (S1140). For example, the transform kernel corresponding to mts_idx may be defined by dividing it into horizontal transform and vertical transform.

[0219] As another example, different transform kernels may be applied to the horizontal transform and the vertical transform. However, the present invention is not limited thereto, and the same transform kernel may be applied to the horizontal transform and the vertical transform.

[0220] As an implementation, mts_idx may be defined as shown in Table 1 below.

[0221] [Table 1]

[0222] mts_idx[x0][y0] trTypeHor trTypeVer 0 0 0 1 1 1 2 2 1 3 1 2 4 2 2

[0223] In addition, the decoder may perform inverse transform based on the transform kernel ( S1150 ).

[0224] On top Fig.11, the following implementation is mainly described: tu_mts_flag is obtained to determine whether MTS is applied, and mts_idx is obtained according to the tu_mts_flag value obtained subsequently to determine the transform kernel, but the present invention is not limited to this. As an example, the decoder directly parses mts_idx without parsing tu_mts_flag to determine the transform kernel. In this case, the above Table 1 can be used. That is, when the mts_idx value indicates 0, DCT-2 can be applied in the horizontal / vertical direction, and when the mts_idx value indicates a value other than 0, DST-7 and / or DCT-8 can be applied according to the mts_idx value.

[0225] As another embodiment of the present invention, a decoding process for performing a transform process is described.

[0226] The decoder may check the transform size nTbS (S10). Here, the transform size nTbS may be a variable representing a horizontal sample size of a scaled transform coefficient.

[0227] The decoder may check the transformation kernel type trType (S20). Here, the transformation kernel type trType may be a variable representing the transformation kernel type, and various embodiments of the present document may be applied. The transformation kernel type trType may include a horizontal transformation kernel type trTypeHor and a vertical transformation kernel type trTypeVer.

[0228] Referring to Table 1, when the transform kernel type trType is 0, the transform kernel type may represent DCT2, when the transform kernel type trType is 1, the transform kernel type may represent DST7, and when the transform kernel type trType is 2, the transform kernel type may represent DCT8.

[0229] The decoder may perform transform matrix multiplication based on at least one of the transform size nTbS or the transform kernel type (S30).

[0230] As another example, when the transform kernel type is 1 and the transform size is 4, a predetermined transform matrix 1 may be applied when performing the transform matrix multiplication.

[0231] As another example, when the transform kernel type is 1 and the transform size is 8, a predetermined transform matrix 2 may be applied when performing the transform matrix multiplication.

[0232] As another example, when the transform kernel type is 1 and the transform size is 16, a predetermined transform matrix 3 may be applied when performing the transform matrix multiplication.

[0233] As another example, when the transform kernel type is 1 and the transform size is 32, the predefined transform matrix 4 may be applied when performing the transform matrix multiplication.

[0234] Similarly, when the transform kernel type is 2 and the transform size is 4, 8, 16, or 32, predefined transform matrices 5, 6, 7, and 8 may be applied, respectively.

[0235] Here, each of the predefined transformation matrices 1 to 8 may correspond to any of various types of transformation matrices. As an example, Figure 6 A transformation matrix of the type illustrated in .

[0236] The decoder may derive transform samples (or transform coefficients) based on the transform matrix multiplication ( S40 ).

[0237] Each of the above embodiments may be used, but the present invention is not limited thereto and may be used in combination with the above embodiments and other embodiments of this specification.

[0238] Fig.12 This is a table for allocating transform sets for each intra prediction mode in NSST as an embodiment to which the present invention is applied.

[0239] Non-separable quadratic transform (NSST)

[0240] The secondary transform unit may apply secondary transform to the primary transformed signal, and here, the secondary transform may be defined in a table of an encoder and / or a decoder.

[0241] As an embodiment, the secondary transform may conditionally adopt a non-separable secondary transform (hereinafter, referred to as "NSST"). For example, NSST may be applied only to intra-prediction blocks and may have a transform set suitable for each prediction mode group.

[0242] Here, the prediction mode group can be configured based on symmetry with respect to the prediction direction. For example, since prediction mode 52 and prediction mode 16 are symmetrical (diagonally) based on prediction mode 34, the same transform set can be applied by forming a group. In this case, when applying a transform for prediction mode 52, since prediction mode 52 has the same transform set as prediction mode 16, the input data is swapped and then applied.

[0243] In addition, since there is no symmetry of direction in the case of planar mode and DC mode, each mode has a different transformation set, and the corresponding transformation set can be composed of two transformations. With respect to the remaining directional modes, each transformation set can be composed of three transformations. However, the present invention is not limited to this, and each transformation set can be composed of multiple transformations.

[0244] In an embodiment, it is possible to define Fig.12 For example, as shown in Table 2 below, a transform set may be determined according to a predefined table according to an intra prediction mode (or intra prediction mode group). Syntax indicating a specific transform in a transform set determined according to an intra prediction mode may be signaled from an encoder to a decoder.

[0245] [Table 2]

[0246] IntraPredMode Transform Set Index IntraPredMode<0 1 0<=IntraPredMode<=1 0 2<=IntraPredMode<=12 1 13<=IntraPredMode<=23 2 24<=IntraPredMode<=44 3 45<=IntraPredMode<=55 2 56<=IntraPredMode 1

[0247] Referring to Table 2, the predefined transformation set may be assigned to the grouped intra prediction modes (or intra prediction mode groups). Here, the IntraPredMode value may be a mode value transformed in consideration of wide angle intra prediction (WAIP).

[0248] Fig.13 This is a calculation flowchart of Givens rotation as an embodiment to which the present invention is applied.

[0249] As another embodiment, NSST may be applied not to the entire main transform block but only to the upper left 8×8 region. For example, when the block size is 8×8 or larger, 8×8 NSST is applied, and when the block size is less than 8×8, 4×4 NSST is applied, and in this case, the block is divided into 4×4 blocks, and then 4×4 NSST is applied to each of the divided blocks.

[0250] As another embodiment, even in the case of 4×N / N×4 (N>=16), 4×4 NSST can be applied.

[0251] Since both 8×8 NSST and 4×4 NSST follow the transform combination configuration described in this document and are inseparable transforms, 8×8 NSST receives 64 data and outputs 64 data and 4×4 NSST has 16 inputs and 16 outputs.

[0252] Both 8×8 NSST and 4×4 NSST are configured by hierarchical combination of Givens rotations. A matrix corresponding to one Givens rotation is shown in Equation 4, and the matrix product is shown in Equation 5 below.

[0253] [Formula 4]

[0254]

[0255] [Formula 5]

[0256] tm=x m cosθ-x n sinθ

[0257] t n =x m sinθ+x n cosθ

[0258] like Fig.13 As illustrated in , since one Givens rotation rotates two data, in order to process 64 data (for 8×8 NSST) or 16 data (for 4×4 NSST), a total of 32 or 8 Givens rotations are required.

[0259] Therefore, a batch of 32 or 8 Givens rotations is used to form a Givens rotation layer. The output data of one Givens rotation layer is transmitted as the input data of the next Givens rotation layer through the determined arrangement.

[0260] Fig.14 As an embodiment to which the present invention is applied, a one-round configuration in a 4×4 NSST composed of Givens rotation layers and arrangements is illustrated.

[0261] Reference Fig.14 , illustrates the sequential processing of four Givens rotation layers in the case of 4×4 NSST. Fig.14 As illustrated in FIG. 1 , output data of one Givens rotation layer is transmitted as input data of the next Givens rotation layer through the determined arrangement (or shuffling).

[0262] like Fig.14 As illustrated in , the pattern to be permuted is regularly defined, and in the case of 4×4 NSST, four Givens rotation layers and their corresponding permutations are combined to form one round.

[0263] In the case of 8×8 NSST, six Givens rotation layers and corresponding permutations form one round. 4×4 NSST goes through two rounds, and 8×8 NSST goes through four rounds. Different rounds use the same permutation pattern, but the applied Givens rotation angles are different. Therefore, it is necessary to store the angle data of all Givens rotations that constitute each transformation.

[0264] As a final step, the data output by the Givens rotation layer is further permuted once, and the corresponding permutation information is stored for each transformation. In the forward NSST, the corresponding permutation is performed last, while in the inverse NSST, the corresponding inverse permutation is first applied inversely thereto.

[0265] In the case of inverse NSST, the Givens rotation layers and the permutations applied to the forward NSST are performed in the reverse order, and even negative values ​​are taken for rotation for each Givens rotation angle.

[0266] Fig.15is a block diagram for describing operations of forward reduction transform and inverse reduction transform as an embodiment to which the present invention is applied.

[0267] Reduced quadratic transform (RST)

[0268] When an orthogonal matrix representing a transform is assumed to have an N×N form, among N transform basis vectors (R<N), the reduced transform (hereinafter, referred to as "RT") leaves only R transform basis vectors. The matrix for generating the forward RT of the transform coefficient is given by the following equation 6.

[0269] [Formula 6]

[0270]

[0271] Since the matrix for inverse RT becomes the permutation matrix of the forward RT matrix, the application of forward RT and inverse RT is as follows Fig.15 Here, the reduction factor is defined as R / N(R <N)。

[0272] The number of elements of the reduced transform is R×N, which is less than the size of the entire matrix (N×N). In other words, the required matrix is ​​R / N of the entire matrix. In addition, the number of multiplications required is R×N, which is R / N less than the original N×N. When the reduced transform is applied, R coefficients are provided, and as a result, only R coefficient values ​​can be sent instead of N coefficients.

[0273] Assuming that RT is applied to an upper left 8×8 block of a main transformed transform block, the RT may be referred to as an 8×8 reduced secondary transform (8×8 RST).

[0274] When the R value of the above Formula 6 is 16, the forward 8×8 RST has a 16×64 matrix form, and the inverse 8×8 RST has a 64×16 matrix form.

[0275] In addition, Fig.12 The same transform set configuration as illustrated in the example can even be applied to 8×8 RST. Fig.12 The corresponding 8×8RST is applied to the transformation set in .

[0276] As an implementation method, when according to Fig.12 When the intra prediction mode in the transform set is composed of two or three transforms, it can be configured to select one of up to 4 transforms including the case where the secondary transform is not applied. Here, one transform can be regarded as an identity matrix.

[0277] When indexes 0, 1, 2, and 3 are assigned to four transforms, respectively, a syntax element called an NSST index may be signaled for each transform block, thereby specifying the corresponding transform. That is, in the case of NSST, an 8×8 NSST may be specified for an 8×8 top left block by an NSST index, and an 8×8 RST may be specified in an RST configuration. In addition, in this case, index 0 may be assigned to a case where the identity matrix (i.e., a secondary transform) is not applied.

[0278] When the forward 8×8 RST shown in equation 6 is applied, 16 valid transform coefficients are generated. As a result, it can be considered that the 64 input data constituting the 8×8 area are reduced to 16 output data. From the perspective of the two-dimensional area, only one quarter of the area is filled with valid transform coefficients. Therefore, Fig.16 The 4×4 upper left area in may be filled with 16 output data obtained by applying the forward 8×8 RST.

[0279] Fig.16 1 is a diagram illustrating a process of performing backward scanning from the 64th to the 17th according to the backward scanning order as an embodiment to which the present invention is applied.

[0280] Fig.16 The example shows scanning from the 17th coefficient to the 64th coefficient when the forward scanning order starts from 1 (in the forward scanning order). However, Fig.16 The backward scan is illustrated, and it illustrates that the backward scan is performed from the 64th coefficient to the 17th coefficient.

[0281] Reference Fig.16 , the upper left 4×4 region is a region of interest (ROI) assigned with a valid transformation system, while the rest of the region is empty. That is, the value 0 may be assigned to the rest of the region by default.

[0282] If in Fig.16 If there are valid transform coefficients other than 0 in the region other than the ROI region, this means that 8×8 RST is not applied. As a result, in this case, the NSST index encoding corresponding thereto can be omitted.

[0283] On the contrary, if Fig.16 If there are no non-zero transform coefficients outside the ROI region (if 8×8 RST is applied, when 0 is assigned to the region other than the ROI), it is possible that 8×8 RST will be applied, and as a result, the NST index can be encoded.

[0284] As such, since the presence of non-zero transform coefficients must be checked, conditional NSST index encoding may be performed after the residual encoding process.

[0285] The present invention provides a method of designing RST and an associated optimization method that can be applied to a 4×4 block from an RST structure. In addition to 4×4 RST, the embodiments disclosed in this specification can also be applied to 8×8 RST or another type of transform.

[0286] Fig.17 Three forward scanning orders of a transform coefficient block (transform block) are exemplified as an embodiment to which the present invention is applied.

[0287] Implementation 1: RST for 4×4 blocks

[0288] The non-separable transform that can be applied to a 4×4 block is a 16×16 transform. That is, when the data elements constituting the 4×4 block are arranged in row-major or column-major order, the non-separable transform is applied using a 16×1 vector.

[0289] The forward 16×16 transform is composed of 16 row-wise transformed basis vectors, and when the inner product is applied to the 16×1 vector and each transform basis vector, the transform coefficient for the transform basis vector is obtained. The process of obtaining the transform coefficients corresponding to all 16 transform basis vectors is equivalent to multiplying the 16×16 non-separable transform matrix by the input 16×1 vector.

[0290] The transform coefficient obtained by matrix multiplication has a 16×1 vector form, and the statistical characteristics may be different for each transform coefficient. For example, when the 16×1 transform coefficient vector is composed of the 0th element to the 15th element, the variance of the 0th element may be greater than the variance of the 15th element. In other words, as the element is set closer to the front, the corresponding variance value of the element is larger, so that the element can have a larger energy value.

[0291] When the inverse 16×16 non-separable transform is applied from the 16×1 transform coefficients, the original 4×4 block signal can be restored. When the forward 16×16 non-separable transform is an orthogonal transform, the corresponding inverse 16×16 transform can be obtained by transposing the matrix used for the forward 16×16 transform.

[0292] When a 16×1 transform coefficient vector is multiplied by an inverse 16×16 non-separable transform matrix, data in the form of a 16×1 vector can be obtained, and when the obtained data is configured with row priority or column priority applied first, a 4×4 block signal can be restored.

[0293] As described above, the elements constituting the 16×1 transform coefficient vector may have different statistical characteristics.

[0294] If the transform coefficients arranged on the front side (close to the 0th element) have greater energy, a signal very close to the original signal can be restored even if the inverse transform is applied to some of the transform coefficients that appear first without using all the transform coefficients. For example, when the inverse 16×16 inseparable transform is composed of 16 column basis vectors, only L column basis vectors are left to form a 16×L matrix. In addition, when the 16×L matrix and L×1 are multiplied by each other only after L important transform coefficients among the transform coefficients (L×1 vectors), a 16×1 vector with a small error in the original input 16×1 vector data can be restored.

[0295] As a result, since only L coefficients are used for data recovery, when obtaining the transform coefficient, an L×1 transform coefficient vector is obtained instead of a 16×1 transform coefficient vector. That is, when an L×16 transform is configured by selecting L corresponding row vectors in a forward 16×16 non-separable transform matrix and the configured L×16 transform is multiplied by a 16×1 input vector, L important transform coefficients can be obtained.

[0296] The range of L value is 1≤L<16, and generally, L vectors can be selected from 16 transform basis vectors by any method, but from the perspective of encoding and decoding, selecting transform basis vectors that are highly important in terms of signal energy will be beneficial in terms of coding efficiency.

[0297] Implementation 2: Configuration of the application area of ​​4×4 RST and arrangement of transform coefficients

[0298] 4×4 RST may be applied as a secondary transform and may be auxiliary applied to a block to which a main transform such as DCT type 2 is applied. When the size of a block to which the main transform is applied is N×N, the size of the block to which the main transform is applied is generally larger than 4×4. Therefore, when 4×4 RST is applied to an N×N block, there may be the following two methods.

[0299] Embodiment 2-1) 4×4 RST is not applied to all N×N regions, but may be applied only to certain regions. For example, 4×4 RST may be applied only to the upper left M×M region (M≤N).

[0300] Embodiment 2-2) A region to which secondary transform is applied may be divided into 4×4 blocks, and 4×4 RST may be applied to each of the divided blocks.

[0301] As an embodiment, the embodiments 2-1) and 2-2) may be mixed and applied. For example, only the upper left M×M area may be divided into 4×4 blocks, and then 4×4 RST may be applied.

[0302] As an embodiment, the secondary transform may be applied only to the upper left 8×8 region, and when the N×N block is equal to or greater than 8×8, the 8×8RST may be applied, and when the N×N block is less than ×88 (4×4, 8×4, and 4×8), the N×N block may be divided into 4×4 blocks, and then the 4×4RST may be applied to each of the 4×4 blocks, as in Embodiment 2-2. In addition, even in the case of 4×N / N×4 (N>=16), the 4×4NSST may be applied.

[0303] When L (1≤L<16) transform coefficients are generated after applying 4×4 RST, there is a degree of freedom in how to arrange the L transform coefficients. However, since there will be a predetermined order when processing the transform coefficients in the residual encoding step, the encoding performance may vary depending on how the L transform coefficients are arranged in the 2D block.

[0304] For example, in the case of residual coding of HEVC, coding starts from the position farthest from the DC position. Coding performance is enhanced by utilizing the fact that the quantized coefficient value is zero or close to zero as moving away from the DC position.

[0305] Therefore, in terms of encoding performance, it may be advantageous to arrange more important coefficients having high energy for the L transform coefficients so that the L transform coefficients are then encoded in the order of residual encoding.

[0306] Fig.17 Three forward scanning orders are shown in units of 4×4 transform blocks (coefficient groups (CGs)) used in HEVC. Residual coding follows Fig.17 The encoding is performed in the reverse order of the scanning order of (i.e., encoding is performed in the order of 16 to 1).

[0307] Since the intra prediction mode is selected Fig.17 Therefore, even for L transform coefficients, the present invention can be configured to similarly determine the scanning order according to the intra prediction mode.

[0308] Fig.18 As an embodiment to which the present invention is applied, when diagonal scanning is applied and 4×4 RST is applied in an upper left 4×8 block, a forward scanning order of each 4×4 block and positions of significant transform coefficients are illustrated.

[0309] When to follow Fig.17 When the diagonal scan order in is used and the upper left 4×8 block is divided into 4×4 blocks and 4×4 RST is applied to each 4×4 block, if the L value is 8 (i.e., if only 8 transform coefficients out of 16 transform coefficients are left), then it can be as follows Fig.18 Set the transform coefficients in the middle.

[0310] Only half of each 4x4 block may have transform coefficients, and by default a value of 0 may be applied to the positions marked X.

[0311] Therefore, according to Fig.17 The scanning order illustrated in arranges L transform coefficients for each 4×4 block and assumes that 16-L remaining positions of each 4×4 block are filled with 0 to perform residual encoding.

[0312] Fig.19 A case where significant transform coefficients of two 4×4 blocks are combined into one 4×4 block when diagonal scanning is applied and 4×4 RST is applied in the upper left 4×8 block is illustrated as an embodiment to which the present invention is applied.

[0313] Reference Fig.19 , L transform coefficients arranged in two 4×4 blocks can be combined into one. Specifically, when the L value is 8, since the transform coefficients of the two 4×4 blocks completely fill one 4×4 block while being combined, there is no transform coefficient left in the other 4×4 block.

[0314] Therefore, since most residual coding is not required with respect to an empty 4×4 block, the corresponding coded_sub_block_flag may be coded as 0.

[0315] In addition, as an embodiment of the present invention, even various schemes may be applied to how to mix transform coefficients of two 4×4 blocks. The transform coefficients may be combined according to a random order, but the present invention may provide the following method.

[0316] 1) The transform coefficients of two 4×4 blocks are mixed alternately in scanning order. Fig.18 In the case where the transform coefficients of the top block are and and the transform coefficients of the bottom block are and When , the transformation coefficients can be and Alternatively, and In other words, the order can be configured so that

[0317] 2) The transform coefficients for the first 4×4 block may be arranged first, and then the transform coefficients for the second 4×4 block may be arranged. In other words, the transform coefficients may be arranged like Alternatively, the sequence may be as follows: Change in the same way.

[0318] Implementation 3: Method for encoding NSST index of 4×4 RST

[0319] When Fig.18 When 4×4 RST is applied as exemplified in , L+1 to 16th blocks may be filled with 0 values ​​according to the transform coefficient scanning order of each 4×4 block.

[0320] Therefore, when non-zero values ​​are generated at the L+1st to 16th positions even in one of the two 4×4 blocks, it can be known that this is a case where 4×4 RST is not applied.

[0321] When 4×4 RST also has a structure in which one of the transform sets prepared as NSST is selected and applied, a transform index to which the transform is to be applied (in this embodiment, it may be referred to as an NSST index) may be signaled.

[0322] It is assumed that any decoder can know the NSST index through bitstream parsing, and the parsing is performed after residual decoding.

[0323] When residual decoding is performed and it is confirmed that there is at least one non-zero transform coefficient between L+1th to 16th, 4×4 RST is not applied, and thus the NSST index may be configured not to be parsed.

[0324] Therefore, the NSST index is selectively parsed only when it is necessary to reduce the signaling cost.

[0325] If Fig.18 As illustrated in the example, when 4×4 RST is applied to multiple 4×4 blocks in a specific area (for example, the same 4×4 RST can be applied to all of the multiple 4×4 blocks, or different 4×4 RSTs can be applied), the 4×4 RST applied to all 4×4 blocks can be specified by one NSST index. In this case, the same 4×4 RST can be specified, or the 4×4 RST applied to each of all 4×4 blocks can be specified.

[0326] Since 4×4 RST is applied to all 4×4 blocks by one NSST index, it is possible to check whether there are non-zero transform coefficients at the L+1 to 16th positions for all 4×4 blocks during the residual decoding process. As a result of the check, when a non-zero transform coefficient exists at a position (L+1 to 16th positions) that is not accepted even in one 4×4 block, the NSST index may be configured not to be encoded.

[0327] NSST indexes may be signaled separately for luma blocks and chroma blocks, and in the case of chroma blocks, separate NSST indexes may be signaled for Cb and Cr, and one NSST index may be shared.

[0328] When Cb and Cr share a NSST index, a 4×4 RST specified by the same NSST index may be applied. In this case, the 4×4 RST for Cb and Cr may be the same, or the NSST index may be the same, but a separate 4×4 RST may be provided.

[0329] In order to apply conditional signaling to the shared NSST index, it is checked whether there are non-zero transform coefficients at the L+1th to 16th positions of all 4×4 blocks for Cb and Cr, and when there are non-zero transform coefficients, the NSST index can be configured not to be notified by a signal.

[0330] like Fig.19 As illustrated in , even for the case where transform coefficients of two 4×4 blocks are combined, when 4×4 RST is applied, it is checked whether there is a non-zero transform coefficient at a position where there is no valid transform coefficient, and then it can be determined whether to signal NSST.

[0331] For example, Fig.19 As illustrated in (b), when the L value is 8 and there is no valid transform coefficient for a 4×4 block (block marked with X) when 4×4 RST is applied, the coded_sub_block_flag of the block where there is no valid transform coefficient can be checked. In this case, when coded_sub_block_flag is 1, the NSST index can be configured not to be notified by a signal.

[0332] Embodiment 4: Optimization method in case of encoding NSST index before residual encoding

[0333] When encoding is performed on the NSST index before residual encoding, whether 4×4 RST is applied is predetermined, and as a result, residual encoding can be omitted as to whether 0 is assigned to the position of the transform coefficient.

[0334] Here, whether 4×4 RST is applied may be configured to be known by the NSST index. For example, when the NSST index is 0, 4×4 RST is not applied.

[0335] Alternatively, the NSST index may be signaled by a separate syntax element (e.g., NSST flag). For example, if the separate syntax element is called the NSST flag, the NSST flag is first parsed to determine whether 4×4 RST is applied, and if the NSST flag value is 1, residual coding may be omitted for positions where no significant transform coefficients may exist.

[0336] As an embodiment, when residual coding is performed, the last non-zero transform coefficient position on the TU is first encoded. When the NSST index is encoded after the last non-zero transform coefficient position is encoded and it is assumed that 4×4RST is applied to the position of the last non-zero transform coefficient, if the last non-zero transform coefficient position is determined to be a position where a non-zero transform coefficient cannot be generated, 4×4RST can be configured not to be applied to the last non-zero transform coefficient position, and the NSST index is not encoded.

[0337] For example, due to Fig.18 In the case of positions marked with X in the 4×4 RST, no valid transform coefficients are set when 4×4 RST is applied (for example, these positions can be filled with 0 values), so the encoding of the NSST index can be omitted when the last non-zero transform coefficient is in the region marked with X. When the last non-zero transform coefficient is not in the region marked with X, the encoding of the NSST index can be performed.

[0338] As an implementation, when checking whether 4×4 RST is applied by conditionally encoding the NSST index after encoding the last non-zero transform coefficient position, the remaining residual coding part can be processed by the following two schemes.

[0339] 1) In the case where 4×4 RST is not applied, general residual coding is maintained as it is. That is, coding is performed under the assumption that a non-zero transform coefficient may exist at any position from a non-zero transform coefficient position to DC.

[0340] 2) When 4×4 RST is applied, since for a specific location or a specific 4×4 block (e.g. Fig.18 There is no transform coefficient at the X position (which can be filled with 0 by default), so the residual of the corresponding position or block may not be performed.

[0341] For example, upon arrival Fig.18 In the case of a position marked with X in , the encoding of sig_coeff_flag can be omitted. Here, sig_coeff_flag means a flag indicating whether there is a non-zero transform coefficient at the corresponding position.

[0342] When Fig.19 When the transform coefficients of two blocks are combined as exemplified in , for the 4×4 blocks assigned to 0, encoding of coded_sub_block_flag may be omitted and the corresponding value may be derived to 0, and all corresponding 4×4 blocks may be derived to a 0 value without separate encoding.

[0343] In the case where the NSST index is encoded after the non-zero transform coefficient positions are encoded, when the x position P of the last non-zero transform coefficient x and the y position P y Less than T x and T y , NSST index encoding can be omitted and 4×4 RST may not be applied.

[0344] For example, T x =1 and T y =1 means that for the case where there is a non-zero transform coefficient at the DC position, the NSST index encoding is omitted.

[0345] The scheme for determining whether to encode the NSST index by comparing with a threshold value may be applied differently to luma and chroma. x and T y Can be applied to both luma and chroma, and thresholds can be applied to luma but not to chroma. Or vice versa.

[0346] The above two methods, i.e., the first method of omitting NSST index encoding when a non-zero transform coefficient is located in an area where no valid transform coefficient exists and the second method of omitting NSST index encoding when each of the X coordinate and the Y coordinate of the non-zero transform coefficient is less than a predetermined threshold, may be applied simultaneously.

[0347] For example, a threshold value of the position coordinates of the last non-zero transform coefficient may be checked first, and then it may be checked whether the last non-zero transform coefficient is located in a region where no valid transform coefficient exists. Alternatively, the order may be changed.

[0348] The method proposed in Embodiment 4 can even be applied to 8×8 RST. That is, when the last non-zero transform coefficient is located in an area other than the upper left 4×4 in the upper left 8×8 area, the encoding of the NSST index can be omitted, otherwise, NSST index encoding can be performed.

[0349] In addition, when both the X and Y coordinate values ​​for the non-zero transform coefficients are less than a threshold, encoding of the NSST index may be omitted. Alternatively, the two methods may be applied together.

[0350] Implementation 5: Applying different NSST index encoding and residual encoding schemes to luma and chroma when applying RST

[0351] The schemes described in Embodiments 3 and 4 may be applied differently to luma and chroma, respectively. That is, the NSST index encoding and residual encoding schemes for luma and chroma may be applied differently.

[0352] For example, the luma may adopt the scheme of Embodiment 4 and the chroma may adopt the scheme of Embodiment 3. Alternatively, the luma may adopt the conditional NSST index encoding proposed in Embodiment 3 or 4, and the chroma may not adopt the conditional NSST index encoding. Or vice versa.

[0353] Fig. 20 The flowchart of encoding a video signal based on a reduced quadratic transform as an embodiment of the present invention is shown.

[0354] The encoder may determine (or select) a forward secondary transform based on at least one of a block shape, a block size, and / or a prediction mode of the current block (S2010). In this case, the candidates for the forward secondary transform may include Figure 6 and / or Fig.12 At least one of the embodiments.

[0355] The encoder may determine the best forward quadratic transform by rate-distortion optimization. The best forward quadratic transform may correspond to one of a plurality of transform combinations, and the plurality of transform combinations may be defined by transform indexes. For example, for RD optimization, the results of all forward quadratic transforms, quantization, residual coding, etc. may be performed for each candidate comparison. In this case, equations such as cost=ratio+λ·distortion or cost=distortion+λ·ratio may be used, but the present invention is not limited thereto.

[0356] The encoder may signal a secondary transform index corresponding to the best forward secondary transform (S2020). Here, the secondary transform index may adopt other implementations described in this specification.

[0357] For example, the secondary transform index can be Fig.12 Since one transform set is composed of two or three transforms depending on the intra prediction mode, one of up to four transforms can be configured to be selected except for the case where the secondary transform is not applied. When indexes 0, 1, 2, and 3 are assigned to four transforms, respectively, the applied transform can be specified by signaling the secondary transform index of each transform coefficient block. In this case, index 0 can be assigned to the case where the identity matrix (i.e., the secondary transform) is not applied.

[0358] As another embodiment, signaling the secondary transform index may be performed in any of the following steps: 1) before residual encoding; 2) in the middle of residual encoding (after encoding the non-zero transform coefficient position); or 3) after residual encoding. Hereinafter, the embodiments will be described in detail.

[0359] 1) Method for signaling secondary transform index before residual coding

[0360] The encoder may determine the forward secondary transform.

[0361] The encoder may signal a secondary transform index corresponding to the forward secondary transform.

[0362] The encoder may encode the position of the last non-zero transform coefficient.

[0363] The encoder may perform residual encoding on syntax elements except for the position of the last non-zero transform coefficient.

[0364] 2) Method of signaling secondary transform index during residual coding

[0365] The encoder may determine the forward secondary transform.

[0366] The encoder may encode the position of the last non-zero transform coefficient.

[0367] When the non-zero transform coefficient is not located in a specific area, the encoder may encode the secondary transform index corresponding to the forward secondary transform. Here, in the case of applying a reduced secondary transform, the specific area refers to the remaining area except for the position where the non-zero transform coefficient may exist when the transform coefficient is arranged according to the scanning order. However, the present invention is not limited to this.

[0368] The encoder may perform residual encoding on syntax elements except for the position of the last non-zero transform coefficient.

[0369] 3) Method for signaling secondary transform index before residual coding

[0370] The encoder may determine the forward secondary transform.

[0371] The encoder may encode the position of the last non-zero transform coefficient.

[0372] When the non-zero transform coefficient is not located in a specific region, the encoder may perform residual coding on syntax elements other than the position of the last non-zero transform coefficient. Here, in the case of applying a reduced secondary transform, the specific region refers to the remaining region except for the position where the non-zero transform coefficient may exist when the transform coefficients are arranged according to the scanning order. However, the present invention is not limited thereto.

[0373] The encoder may encode a secondary transform index corresponding to the forward secondary transform.

[0374] In addition, the encoder may perform a forward first-order transform on the current block (residual block) (S2030). Here, step S2010 and / or step S2020 may be similarly applied to the forward main transform.

[0375] The encoder may perform a forward secondary transform on the current block by using the optimal forward secondary transform (S2040). For example, the optimal forward secondary transform may be a reduced secondary transform. A reduced secondary transform refers to a transform that inputs N residual data (N×1 residual vectors) and outputs L (L<N) transform coefficient data (L×1 transform coefficient vectors).

[0376] As an embodiment, a reduced secondary transform may be applied to a specific area of ​​the current block. For example, when the current block is N×N, the specific area may refer to the upper left N / 2×N / 2 area. However, the present invention is not limited thereto, and may be configured differently according to at least one of a prediction mode, a block shape, or a block size. For example, when the current block is N×N, the specific area may refer to the upper left M×M area (M≤N).

[0377] Furthermore, the encoder performs quantization on the current block to generate a transform coefficient block (S2050).

[0378] The encoder performs entropy encoding on the transform coefficient block to generate a bitstream.

[0379] Fig.21 The flowchart of decoding a video signal based on a reduced quadratic transform is an embodiment to which the present invention is applied.

[0380] The decoder can obtain a secondary transform index from the bitstream (S2110). Here, the secondary transform index can adopt other implementations described in this specification. For example, the secondary transform index may include Figure 6 and / or Fig.12 At least one of the embodiments.

[0381] As another embodiment, the secondary transform index may be obtained in any of the following steps: 1) before residual encoding; 2) in the middle of residual encoding (after decoding the non-zero transform coefficient position); or 3) after residual encoding.

[0382] The decoder may derive a secondary transform corresponding to the secondary transform index (S2120). In this case, the candidates for the forward secondary transform may include Figure 6 and / or Fig.12 At least one of the embodiments.

[0383] However, steps S2110 and S2120 are implementations, and the present invention is not limited thereto. For example, the decoder may not obtain the secondary transform index, but may derive the secondary transform based on at least one of the block shape, block size and / or prediction mode of the current block.

[0384] In addition, the decoder may obtain a transform coefficient block by entropy decoding the bitstream, and may perform dequantization on the transform coefficient block ( S2130 ).

[0385] The decoder 200 may perform an inverse quadratic transform on the dequantized transform coefficient block (S2140). For example, the inverse quadratic transform may be a reduced quadratic transform. A reduced quadratic transform refers to a transform that inputs N residual data (N×1 residual vectors) and outputs L (L<N) transform coefficient data (L×1 transform coefficient vectors).

[0386] As an embodiment, a reduced secondary transform may be applied to a specific area of ​​the current block. For example, when the current block is N×N, the specific area may refer to the upper left N / 2×N / 2 area. However, the present invention is not limited thereto, and may be configured differently according to at least one of a prediction mode, a block shape, or a block size. For example, when the current block is N×N, the specific area may refer to the upper left M×M area (M≤N) or M×L (M≤N, L≤N).

[0387] In addition, the decoder may perform inverse main transform on the inverse secondary transform result (S2150).

[0388] The decoder generates a residual block through step S2150, and adds the residual block to the prediction block to generate a reconstructed block.

[0389] Implementation 6: Reduced Multiple Transform Selection (MTS)

[0390] The embodiments of the present invention propose a method for increasing complexity by applying the main transform only to a predefined region. When a combination of various transforms (or transform kernels) such as MTS (e.g., DCT2, DST7, DCT8, DST1, DCT5, etc.) is optionally applied to the main transform, the complexity may increase. Specifically, as the size of the coding block (or transform block) increases, various transforms need to be considered, thereby significantly increasing the complexity.

[0391] Therefore, the present invention proposes a method of performing transformation only on a predetermined area according to a specific condition instead of performing (or applying) the transformation on all areas in order to reduce complexity.

[0392] As an implementation method, based on the above reference Figures 15 to 21 In the described reduced transform (RT) method, the encoder / decoder may obtain a transform block having an R×R size by performing a transform on an area having an R×R size (M>=R), rather than obtaining a transform block having an M×M size by performing a transform only on an area having an R×R size (M>=R). As an example, the R×R area may be an upper left R×R area in the current block (coding block or transform block).

[0393] As a result, significant coefficients (non-zero coefficients) may exist only for the R×R region. As an example, in this case, the encoder / decoder may not perform calculations on coefficients existing in regions other than the R×R region, but may regard the values ​​of the coefficients as 0 (zero output).

[0394] In addition, the encoder / decoder may apply the main transform only to a predetermined area, which is determined according to the size of the coding block (or transform block) and / or the type of transform (transform kernel). Table 3 below shows a reduced adaptive multiple transform (RAMT) using a predefined R value according to the size of the transform (or the size of the transform block). In the present invention, a reduced adaptive multiple transform (RAMT) representing a reduced transform adaptively determined according to a block size may be referred to as a reduced multiple transform selection (MTS), a reduced explicit multiple transform, a reduced main transform, etc.

[0395] [Table 3]

[0396] Change size Reduction transformation 1 Reduction Transformation 2 Reduction transformation 3 8×8 4×4 6×6 6×6 16×16 8×8 12×12 8×8 32×32 16×16 16×16 16×16 64×64 32×32 16×16 16×16 128×128 32×32 16×16 16×16

[0397] Referring to Table 3, at least one reduced transform may be defined according to the size of the transform (or the size of the transform block). In an embodiment, which reduced transform to use among the reduced transforms shown in Table 3 may be determined according to the transform (or transform kernel) applied to the current block (coding block or transform block). Although it is assumed that three reduced transforms are used in Table 3, the present invention is not limited thereto, and one or more various reduced transforms may be predefined according to the size of the transform.

[0398] In addition, in an embodiment of the present invention, when the above-mentioned reduced adaptive multiple transform is applied, the reduced transform factor (R) can be independently determined according to the main transform. For example, when the main transform is DCT2, the amount of calculation is relatively simple compared to other main transforms (e.g., a combination of DST7 and / or DCT8), so the reduced transform is not performed for smaller blocks or a relatively large R value is used, thereby minimizing the reduction in encoding performance. Table 4 below shows a reduced adaptive multiple transform (RAMT) using a predefined R value according to the size of the transform (or the size of the transform block) and the transform kernel.

[0399] [Table 4]

[0400] Change size Reduced transform for DCT2 Reduced transforms other than DCT2 8×8 8×8 4×4 16×16 16×16 8×8 32×32 32×32 16×16 64×64 32×32 32×32 128×128 32×32 32×32

[0401] Referring to Table 4, when the transform applied to the main transform is a combination of DCT2 and other transforms (eg, DCT7 and / or DCT8), different reduced transform factors may be used.

[0402] Fig. 22 FIG. 1 is a diagram illustrating a method of encoding a video signal by using a reduced transform as an embodiment to which the present invention is applied.

[0403] Reference Fig. 22 , the encoder first determines whether to apply a transform to the current block (S2201). The encoder can encode a transform skip flag according to the determined result. In this case, encoding the transform skip flag can be included in step S2201.

[0404] When applying a transform to the current block, the encoder determines a transform kernel of a primary transform applied to the current block (S2202). The encoder may encode a transform index indicating the determined transform kernel, and in this case, encoding the transform index may be included in step S2202.

[0405] The encoder determines a region in which a main transform is applied to the current block based on a transform kernel of a first transform applied to the current block and a size of the current block ( S2203 ).

[0406] As an embodiment, the encoder may regard coefficients of the remaining regions in the current block except for the region to which the main transform is applied as zero.

[0407] In addition, as an embodiment, when the transform kernel indicated by the transform index is a predefined transform and the width and / or height of the current block is greater than a predefined size, the encoder may determine a region having a width and / or height of a predefined size as a region to which the main transform is applied.

[0408] For example, the predefined transform may be any one of a plurality of transform combinations configured by a combination of DST7 and / or DCT8, and the predefined size may be 16. Alternatively, the predefined transform may be the remaining transforms except DCT2. In addition, as an example, when the transform kernel indicated by the transform index is DCT2 and the width and / or height of the current block is greater than 32, the encoder may determine a region having a width and / or height of 32 as a region to which the main transform is applied.

[0409] In addition, as an embodiment, when the transform kernel indicated by the transform index belongs to the first transform group, the encoder may determine the smaller value of the width of the current block and the first threshold as the width of the region to which the main transform is applied, and determine the smaller value of the height of the current block and the first threshold as the height of the region to which the main transform is applied. As an example, the first threshold may be 32, but the present invention is not limited thereto, and the first threshold may be 4, 8, or 16 as shown in Table 3 or Table 4 described above.

[0410] In addition, when the transform kernel indicated by the transform index belongs to the second transform group, the encoder may determine the smaller value of the width of the current block and the second threshold as the width of the region to which the main transform is applied, and determine the smaller value of the height of the current block and the second threshold as the height of the region to which the main transform is applied. As an example, the second threshold may be 16, but the present invention is not limited thereto, and the second threshold may be 4, 6, 8, 12, or 32 as shown in Table 3 or Table 4 described above.

[0411] As an embodiment, the first transform group may include DCT2, and the second transform group may include a plurality of transform combinations configured by a combination of DST7 and / or DCT8.

[0412] The encoder performs a forward main transform by using a transform kernel of the main transform applied to the current block on the region to which the main transform is applied (S2204). The encoder may obtain a transform coefficient after the main transform by performing the forward main transform. As an embodiment, the encoder may apply a secondary transform to the transform coefficient after the main transform, and in this case, the above in Figures 4 to 20 The method described in .

[0413] Fig.23 FIG. 1 is a diagram illustrating a method of decoding a video signal by using a reduced transform as an embodiment to which the present invention is applied.

[0414] The decoder checks whether transform skip is applied to the current block (S2301).

[0415] When transform skip is not applied to the current block, the decoder obtains a transform index indicating a transform kernel applied to the current block from a video signal (S2302).

[0416] The decoder determines a region in which a main transform is applied to the current block based on a transform kernel indicated by a transform index and a size (ie, width and / or height) of the current block (S2303).

[0417] As an embodiment, the decoder may regard coefficients of the remaining regions in the current block except for the region to which the main transform is applied as zero.

[0418] In addition, as an embodiment, when the transform kernel indicated by the transform index is a predefined transform and the width and / or height of the current block is greater than a predefined size, the decoder can determine that the area with a width and / or height of the predefined size is the area where the main transform is applied.

[0419] For example, the predefined transform may be any one of a plurality of transform combinations configured by a combination of DST7 and / or DCT8, and the predefined size may be 16. Alternatively, the predefined transform may be the remaining transforms except DCT2. In addition, as an example, when the transform kernel indicated by the transform index is DCT2 and the width and / or height of the current block is greater than 32, the decoder may determine a region having a width and / or height of 32 as a region to which the main transform is applied.

[0420] In addition, as an embodiment, when the transform kernel indicated by the transform index belongs to the first transform group, the decoder may determine the smaller value of the width of the current block and the first threshold as the width of the region to which the main transform is applied, and determine the smaller value of the height of the current block and the first threshold as the height of the region to which the main transform is applied. As an example, the first threshold may be 32, but the present invention is not limited thereto, and the first threshold may be 4, 8, or 16 as shown in Table 3 or Table 4 described above.

[0421] In addition, when the transform kernel indicated by the transform index belongs to the second transform group, the decoder may determine the smaller value of the width of the current block and the second threshold as the width of the region to which the main transform is applied, and determine the smaller value of the height of the current block and the second threshold as the height of the region to which the main transform is applied. As an example, the second threshold may be 16, but the present invention is not limited thereto, and the second threshold may be 4, 6, 8, 12, or 32 as shown in Table 3 or Table 4 described above.

[0422] As an embodiment, the first transform group may include DCT2, and the second transform group may include a plurality of transform combinations configured by a combination of DST7 and / or DCT8.

[0423] The decoder performs an inverse main transform on the region to which the main transform is applied by using the transform kernel indicated by the transform index (S2304). The decoder may obtain the transform coefficients after the main inverse transform by performing the inverse main transform. As an embodiment, the decoder may apply a secondary transform to the dequantized transform coefficients before performing the main transform, and in this case, the above Figures 4 to 20 The method described in .

[0424] According to an embodiment of the present invention, only predefined regions are transformed according to specific conditions, thereby significantly reducing the worst case complexity.

[0425] Implementation 7: Reduction Transformation

[0426] In an embodiment of the present invention, it is described Figures 15 to 23Various implementations of the reduced transform proposed in the present invention are provided to improve the complexity problem of the transform. As described above, the reduced transform proposed in the present invention can be applied regardless of whether it is a primary transform (e.g., DCT, DST) or a secondary transform (e.g., NSST, low frequency non-separable transform (LFNST)).

[0427] Fig.24 FIG. 1 is a diagram illustrating a reduced transformation structure based on a reduction factor as an embodiment to which the present invention is applied.

[0428] Reference Fig.24 , the decoder is mainly described for the convenience of description, but the reduction transformation proposed in the implementation manner can be equally applied to the encoder.

[0429] The decoder 200 may apply an inverse reduction transform to the dequantized transform coefficients. In this case, the decoder may use a predetermined (or predefined) reduction factor (eg, R or R / N) and / or a transform kernel to perform the reduction transform.

[0430] In one embodiment, the transform kernel may be selected based on available information such as the size (e.g., width / height) of the current block (coding block or transform block), intra / inter prediction mode, CIdx, etc. When the current coding block is a luma block, the value of CIdx may be 0. Otherwise (i.e., if the current coding block is a Cb or Cr block), CIdx may have a non-zero value such as 1.

[0431] Fig.25 : is a diagram illustrating a method of performing decoding by adaptively applying a reduced transform as an embodiment to which the present invention can be applied.

[0432] Reference Fig.25 , the decoder is mainly described for the convenience of description, but the method of performing transformation using reduced transformation proposed in the embodiment can be equally applied to the encoder.

[0433] The decoder performs dequantization on the current block (S2501).

[0434] The decoder checks whether a transform is applied to (or used for) the current block (S2502). If a transform is not applied to the current block, the decoder terminates the transform process.

[0435] When a transform is applied to the current block, the decoder parses a transform index indicating a transform kernel applied to the current block according to the video signal (S2503).

[0436] The decoder checks whether the reduced inverse transform condition is met (S2504). If the reduced inverse transform condition is not met, the decoder performs a normal inverse transform on the current block (S2505). If the reduced inverse transform condition is met, the decoder performs a reduced inverse transform on the current block (S2507). In this case, the decoder can select a transform kernel to be applied to the current block based on the transform index parsed in step S2503 (S2506). As an embodiment, the transform kernel can be selected based on available information such as the size (e.g., width / height) of the current block (coding block or transform block), intra / inter prediction mode, CIdx, etc. In addition, when applying a reduced inverse transform to the current block, step S2506 may include selecting a reduction factor.

[0437] In an embodiment, a reduced inverse transform condition may be applied to the above condition 6) (e.g., Table 3 and Table 4). In other words, whether to apply a reduced inverse transform may be determined based on the size of the current block (coding block or transform block) and the transform type (or transform kernel).

[0438] As an example, the reduction transform may be used when the following specific conditions are satisfied. In other words, the reduction transform may be applied to a block having a predetermined size or larger (or larger than a predetermined size) that satisfies the following specific conditions.

[0439] - Width>TH && Height>HT (where TH is a predefined value (e.g., 4) representing a specific threshold)

[0440] or

[0441] -width×height>K&&MIN(width,height)>TH (where K or TH is a predefined value and represents a specific threshold)

[0442] As another example, the reduction transform may be used when the following specific conditions are satisfied. In other words, the reduction transform may be applied to a block having a predetermined size or less (or less than a predetermined size) that satisfies the following specific conditions.

[0443] - Width <= TH && Height <= HT (where TH is a predefined value (e.g., 8) representing a specific threshold)

[0444] or

[0445] - width×height<=K&&MIN(width, height)<=TH (where K or TH is a predefined value and represents a specific threshold)

[0446] As another example, the reducing transform may be applied only to a predetermined group of blocks.

[0447] -Width==TH&&Height==TH

[0448] or

[0449] -Width == Height

[0450] As an embodiment, if the conditions for using the reduced transform are not met, a normal transform may be applied. Specifically, a regular transform may be predefined and available to the encoder / decoder. An example of a normal transform is shown below.

[0451] -DCT2, DCT4, DCT5, DCT7, DCT8

[0452] or

[0453] -DST1, DST4, DST7

[0454] or

[0455] -Inseparable transformation

[0456] or

[0457] -NSST(HyGT)

[0458] or

[0459] -LFNST (Low Frequency Non-separable Transform)

[0460] The above conditions can be interpreted based on logical operators as shown in Table 5 below.

[0461] [Table 5]

[0462]

[0463] In addition, if Fig.25 As illustrated in , the reduced transform condition may depend on a transform index Transform_idx indicating the transform applied to the current block. As an example, Transform_idx may be sent twice from the encoder to the decoder. One may be the horizontal transform index Transform_idx_h, and the other may be the vertical transform index Transform_idx_v.

[0464] Fig.26 : is a diagram illustrating a method of performing decoding by adaptively applying a reduced transform as an embodiment to which the present invention can be applied.

[0465] Reference Fig.26 , the decoder is mainly described for the convenience of description, but the method of performing transformation using reduced transformation proposed in the embodiment can be equally applied to the encoder.

[0466] In an embodiment of the present invention, the above-mentioned reduced transformation can be used to perform a secondary transformation. In this regard, the Fig.25 The method described in was repeated as described in .

[0467] 1) The decoder performs dequantization on the current block and then checks whether NSST is activated in the current block. The decoder can use predefined conditions to determine whether it is necessary to parse the NSST index.

[0468] 2) If NSST is activated, the decoder parses the NSST index and checks whether the reduced inverse quadratic transform is applied.

[0469] 3) The decoder checks whether the reduced inverse transform conditions are met.

[0470] 4) If the reduced inverse transform condition is not met, the decoder performs a normal inverse transform on the current block.

[0471] 5) If the reduced inverse quadratic transform condition is met, the decoder performs a reduced inverse quadratic transform on the current block.

[0472] 6) In this case, the decoder can select the transform kernel to apply to the current block based on the NSST index. As an embodiment, the transform kernel can be selected based on available information such as the size (e.g., width / height) of the current block (coding block or transform block), intra / inter prediction mode, CIdx, etc. In addition, when applying a reduced quadratic inverse transform to the current block, the decoder can select a reduction factor.

[0473] In an embodiment, the reduced quadratic inverse transform condition may adopt the above condition 6) (e.g., Table 3 and Table 4). In other words, whether to apply the reduced inverse transform may be determined based on the size of the current block (coding block or transform block) and the transform type (or transform kernel).

[0474] Embodiment 8: Reduced Transform as a Secondary Transform with Different Block Sizes

[0475] In an embodiment of the present invention, a reduced transform considering various block sizes for secondary transform / secondary inverse transform is proposed. As an example, reduced transforms of different block sizes of 4×4, 8×8 and 16×16 for secondary transform / secondary inverse transform can be defined.

[0476] Fig. 27 and Fig.28 is a diagram illustrating an example of a forward reduced quadratic transform and an inverse reduced quadratic transform and pseudo codes for deriving them.

[0477] Reference Fig. 27 and 28 , when the block to which the secondary transform is applied is an 8×8 block and the reduction coefficient R is 16, the secondary transform is reduced and the inverse secondary transform is reduced. This can be done by using Fig.28 The pseudo code shown in the example is used to derive Fig. 27The reduced quadratic transform and the reduced inverse quadratic transform are illustrated in FIG.

[0478] Embodiment 9: Reduction Transformation as a Quadratic Transformation with Non-Rectangular Shape

[0479] As described above, due to the complexity problem of applying a secondary transform to an inseparable transform, in the related art image compression technology, a secondary transform is applied to the upper left 4×4 or 8×8 area of ​​a coding block (or transform block).

[0480] The embodiments of the present invention propose a method of applying a reduced quadratic transform to various non-square images in addition to a 4×4 or 8×8 square area.

[0481] Fig.29 FIG. 1 is a diagram illustrating a method of applying a reduced quadratic transform to a non-square region as an embodiment to which the present invention is applied.

[0482] Reference Fig.29 In an embodiment, the reduced quadratic transform may be applied to only a portion of the block, such as Fig.29 exemplified in .

[0483] exist Fig.29 In , each square represents a 4×4 region. Therefore, the encoder / decoder can apply a reduced quadratic transform to 10×4 pixels (i.e., a 160 pixel region). In this case, the reduction coefficient R=16 and the entire RST matrix corresponds to a 16×160 matrix, thereby reducing the computational complexity of applying the quadratic transform.

[0484] Implementation 10: Reduction Factor

[0485] Fig.30 FIG. 1 is a diagram illustrating a reduction transformation controlled by a reduction factor as an embodiment to which the present invention is applied.

[0486] Reference Fig.30 , as described above, the reduction transformation according to the embodiment of the present invention can be controlled by the reduction factor, such as Fig.30 exemplified in .

[0487] Specifically, modifying the reduction factor can modify the memory complexity and the number of multiplication operations. As previously discussed in Fig.15 The memory and multiplications can be reduced by modifying the reduction factor R / N mentioned in equation 6. For example, for 8×8 NSST with R=16, the memory and multiplications can be reduced by 1 / 4.

[0488] Implementation 11: Advanced Syntax

[0489] Embodiments of the present invention propose a high-level syntax structure for controlling reduction transformations at a high level.

[0490] In an embodiment, as shown in the example of Table 6 below, whether to accept the reduced transformation through a sequence parameter set (SPS) and information related to the size and / or reduction factor may be transmitted. However, the present invention is not limited thereto, and the syntax may be signaled through a picture parameter set (PPS), a slice header, etc.

[0491] [Table 6]

[0492]

[0493]

[0494] Referring to Table 6, if Reduced_transform_enabled_flag is 1, the reduced transform may be available and applied. In the case where Reduced_transform_enabled_flag is 0, this may indicate that the reduced transform may not be available. If Reduced_transform_enabled_flag does not exist, this value may be estimated to be equal to zero.

[0495] Reduced_transform_factor represents a syntax element that specifies the number of reduced dimensions for a reduced transform.

[0496] min_reduced_transform_size represents a syntax element that specifies the minimum transform size to which the reduced transform will be applied. If min_reduced_transform_size is not present, this value may be estimated to be equal to zero.

[0497] max_reduced_transform_size represents a syntax element that specifies the maximum transform size to which the reduced transform will be applied. If max_reduced_transform_size is not present, this value may be estimated to be equal to zero.

[0498] Reduced_transform_factor represents a syntax element that specifies the number of reduced dimensions for a reduced transform. If Reduced_transform_factor does not exist, this value may be estimated to be equal to zero.

[0499] Implementation 12: Secondary Transform Kernel

[0500] Embodiments of the present invention propose various secondary transform kernels.

[0501] In an embodiment, a 4×4 NSST kernel for DC mode may be defined as shown in Table 7 below.

[0502] [Table 7]

[0503]

[0504] Additionally, in an embodiment, a 4×4 NSST kernel for a planar pattern may be defined as shown in Table 8 below.

[0505] [Table 8]

[0506]

[0507] Additionally, in an embodiment, an 8×4 NSST kernel for DC mode may be defined as shown in Table 9 below.

[0508] [Table 9]

[0509]

[0510]

[0511]

[0512]

[0513]

[0514] Additionally, in an embodiment, an 8×8 NSST kernel for a planar mode may be defined as shown in Table 10 below.

[0515] [Table 10]

[0516]

[0517]

[0518]

[0519]

[0520]

[0521] The transform kernels of Tables 7 to 10 described above may be defined as smaller transform kernels for reduced transforms.

[0522] For example, for 8×8 NSST with DC mode and R=8, memory and multiplication can be reduced by 1 / 2. Therefore, by keeping only the coefficients of the upper half of the transform kernel of Table 7 above (8×16 matrix), the reduced transform kernel can be defined in a smaller size as shown in Table 11 below.

[0523] [Table 11]

[0524]

[0525] In another example, for 8×8 NSST with DC mode and R=16, the memory and multiplication can be reduced by 1 / 4. Therefore, by keeping only the coefficients of the upper 1 / 4 part of the transform kernel of Table 9 above (16×64 matrix), the reduced transform kernel can be defined in a smaller size as shown in Table 12 below.

[0526] [Table 12]

[0527]

[0528]

[0529]

[0530]

[0531]

[0532] In the above example, each transform coefficient is represented by 9 bits (i.e., 1 bit: sign; 8 bits: absolute value from 0 to 255). In an embodiment of the present invention, various precisions may be used to represent the transform coefficients. For example, 8 bits may be used instead of 9 bits to represent each coefficient. In this case, the sign bit does not change, but the range of absolute values ​​may change.

[0533] For convenience of description, the above embodiments of the present invention have been described separately, but the present invention is not limited thereto. That is, the embodiments described in the above embodiments 1 to 12 may be performed independently, and one or more various embodiments may be combined and performed.

[0534] Fig.31 is a diagram illustrating an inverse transform unit according to an embodiment to which the present invention is applied.

[0535] exist Fig.31 , for convenience of description, the inverse transform unit is illustrated as one block, but the inter prediction unit may be implemented in a configuration included in an encoder and / or a decoder.

[0536] Reference Fig.31 , the inverse transform unit implements the above Figures 4 to 30 Specifically, the inverse transform unit may be configured to include a transform skip check unit 3101, a transform index obtaining unit 3102 and a main inverse transform unit 3103.

[0537] The transform skip checking unit 3101 checks whether transform skip is applied to the current block.

[0538] When transform skip is not applied to the current block, the transform index obtaining unit 3102 obtains a transform index indicating a transform kernel applied to the current block according to the video signal.

[0539] The main inverse transform unit 3103 determines a region to apply the main transform to the current block based on the transform kernel indicated by the transform index and the size (ie, width and / or height) of the current block.

[0540] As an embodiment, the main inverse transform unit 3103 may regard coefficients of remaining areas in the current block except for the area to which the main transform is applied as 0.

[0541] In addition, as an embodiment, when the transform kernel indicated by the transform index is a predefined transform and the width and / or height of the current block is greater than the predefined size, the main inverse transform unit 3103 can determine that the area with a width and / or height of the predefined size is the area to which the main transform is applied.

[0542] For example, the predefined transform may be any one of a plurality of transform combinations configured by a combination of DST7 and / or DCT8, and the predefined size may be 16. Alternatively, the predefined transform may be the remaining transforms except DCT2. In addition, as an example, when the transform kernel indicated by the transform index is DCT2 and the width and / or height of the current block is greater than 32, the main inverse transform unit 3103 may determine that the region having a width and / or height of 32 is the region to which the main transform is applied.

[0543] In addition, as an embodiment, when the transform kernel indicated by the transform index belongs to the first transform group, the main inverse transform unit 3103 may determine the smaller value of the width of the current block and the first threshold as the width of the area to which the main transform is applied, and determine the smaller value of the height of the current block and the first threshold as the height of the area to which the main transform is applied. As an example, the first threshold may be 32, but the present invention is not limited thereto, and the first threshold may be 4, 8, or 16 as shown in Table 3 or Table 4 described above.

[0544] In addition, when the transform kernel indicated by the transform index belongs to the second transform group, the decoder may determine the smaller value of the width of the current block and the second threshold as the width of the region to which the main transform is applied, and determine the smaller value of the height of the current block and the second threshold as the height of the region to which the main transform is applied. As an example, the second threshold may be 16, but the present invention is not limited thereto, and the second threshold may be 4, 6, 8, 12, or 32 as shown in Table 3 or Table 4 described above.

[0545] As an embodiment, the first transform group may include DCT2, and the second transform group may include a plurality of transform combinations configured by a combination of DST7 and / or DCT8.

[0546] The main inverse transform unit 3103 performs an inverse main transform on the region to which the main transform is applied by using the transform kernel indicated by the transform index. The main inverse transform unit 3103 may obtain the transform coefficient after the main inverse transform by performing the inverse main transform. As an embodiment, the decoder may apply a secondary transform to the transform coefficient after dequantization before performing the main transform, and in this case, the above Figures 4 to 20 The method described in .

[0547] Fig.32 A video encoding system to which the present invention is applied is exemplified.

[0548] The video encoding system may include a source device and a receiving device. The source device may transmit the encoded video / image information or data to the receiving device in the form of a file or stream transmission via a digital storage medium or a network.

[0549] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0550] The video source can obtain the video / image through the capture, synthesis or generation process of the video / image. The video source may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive including previously captured videos / images, etc. The video / image generation device may include, for example, a computer, a tablet computer, and a smart phone, and may (electronically) generate the video / image. For example, a virtual video / image may be generated by a computer, etc., and in this case, the video / image capture process may be replaced by a process that generates relevant data.

[0551] The encoding device can encode the input video / image. The encoding device can perform a series of processes including prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0552] The transmitter can send the encoded video / image information or data output in the form of a bit stream to the receiver of the receiving device in the form of a file or stream transmission through a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for sending through a broadcast / communication network. The receiver may extract the bit stream and send the extracted bit stream to a decoding device.

[0553] The decoding device performs a series of processes including dequantization, inverse transformation, prediction, etc. corresponding to the operation of the encoding device to decode the video / image.

[0554] The renderer can render the decoded video / image, and the rendered video / image can be displayed through a display.

[0555] Fig.33 It is a structural diagram of a content streaming system as an embodiment to which the present invention is applied.

[0556] Reference Fig.33 The content streaming system to which the present invention is applied may mainly include an encoding server, a streaming server, a network server, a media storage, a user device and a multimedia input device.

[0557] The encoding server compresses the content input from the multimedia input device including a smartphone, a camera, a camcorder, etc. into digital data for generating a bit stream, and transmits the bit stream to the streaming server. As another example, when the multimedia input device including a smartphone, a camera, a camcorder, etc. directly generates a bit stream, the encoding server can be omitted.

[0558] A bitstream can be generated by applying the encoding method or the bitstream generating method of the present invention, and a streaming server can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0559] The streaming server transmits multimedia data to the user device through the network server based on the user's request, and the network server serves as a medium to inform the user of the existing services. When the user requests the desired service from the network server, the network server transmits the requested service to the streaming server, and the streaming server transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, and in this case, the control server is used to control the command / response between the corresponding devices in the content streaming system.

[0560] The streaming server can receive content from a media storage and / or an encoding server. For example, when the streaming server receives content from the encoding server, the streaming server can receive the content in real time. In this case, the streaming server can store the bitstream for a predetermined time to provide a stable streaming service.

[0561] Examples of user devices may include cellular phones, smart phones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, touch screen PCs, tablet PCs, ultrabooks, wearable devices such as smart watches, smart glasses, head mounted displays (HMDs), etc., and the like.

[0562] Each server in the content streaming system may operate as a distributed server, in which case data received by each server may be distributed and processed.

[0563] As described above, the embodiments described in the present invention can be implemented and executed on a processor, a microprocessor, a controller or a chip. For example, the functional units illustrated in each figure can be implemented and executed on a computer, a processor, a microprocessor, a controller or a chip.

[0564] In addition, the decoder and encoder to which the present invention is applied may be included in a multimedia broadcast transmission and reception device, a mobile communication terminal, a home theater video device, a digital theater video device, a surveillance camera, a video chat device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camera, a video on demand (VoD) service provider, a (overhead) OTT video device, an Internet streaming service provider, a three-dimensional (3D) video device, a video phone video device, a transportation tool terminal (e.g., a vehicle terminal, an airplane terminal, a ship terminal, etc.), a medical video device, etc., and may be used to process a video signal or a data signal. For example, an over-the-top (OTT) video device may include a game console, a Blu-ray player, an Internet access TV, a home theater system, a smart phone, a tablet PC, a digital video recorder (DVR), etc.

[0565] In addition, the processing method of the present invention can be generated in the form of a program executed by a computer, and can be stored in a computer-readable recording medium. The multimedia data with a data structure according to the present invention can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices that store computer-readable data. The computer-readable recording medium may include, for example, a blue disc (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable recording medium includes a medium implemented in the form of a carrier wave (e.g., transmission on the Internet). In addition, the bit stream generated by the encoding method can be stored in a computer-readable recording medium, or can be transmitted through a wired / wireless communication network.

[0566] In addition, the embodiments of the present invention may be implemented as a computer program product through a program code, and the program code may be executed on a computer through the embodiments of the present invention. The program code may be stored on a computer readable carrier.

[0567] In the above-mentioned embodiments, the components and features of the present invention are combined in a predetermined form. Unless otherwise explicitly stated, each component or feature should be considered as a selection. Each component or feature can be implemented without being associated with other components or features. In addition, the embodiments of the present invention can be configured by associating some components and / or features. The order of the operations described in the embodiments of the present invention can be changed. Some components or features of any embodiment can be included in another embodiment, or replaced by components and features corresponding to another embodiment. It is obvious that claims that are not explicitly cited in the claims are combined to form an embodiment or included in a new claim by modification after submission.

[0568] The embodiments of the present invention may be implemented by hardware, firmware, software or a combination thereof. In the case of being implemented by hardware, according to the hardware implementation, the exemplary embodiments described herein may be implemented using one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0569] In the case of being implemented by firmware or software, the embodiments of the present invention may be implemented in the form of modules, processes, functions, etc. that perform the above functions or operations. The software code may be stored in a memory and executed by a processor. The memory may be inside or outside the processor and may send data to / receive data from the processor by various known means.

[0570] It is obvious to those skilled in the art that the present invention can be implemented in other specific forms without departing from the essential features of the present invention. Therefore, the detailed description mentioned above should not be interpreted as being restrictive in all aspects, and should be considered as exemplary. The scope of the present invention should be determined by the reasonable interpretation of the attached claims, and all modifications within the equivalent scope of the present invention are included within the scope of the present invention.

[0571] Industrial Applicability

[0572] Above, for illustrative purposes, preferred embodiments of the present invention are disclosed, and hereinafter, those skilled in the art will modify, change, replace or add various other embodiments within the technical spirit and technical scope of the present invention disclosed in the appended claims.

Claims

1. A method for decoding a video signal by a decoding device, the method comprising the following steps: Obtaining transform index information indicating a transform kernel of a current block from the video signal; determining a size of a non-zero region for inverse main transform based on the transform kernel indicated by the transform index information and a size of the current block; as well as performing the inverse main transform based on coefficients in the non-zero region for the inverse main transform by using the transform kernel indicated by the transform index information, The transform kernel indicated by the transform index information includes a pair of horizontal transform kernel and vertical transform kernel, wherein, based on the current block having a size of 32×32 and the transform kernel applied to the current block belonging to the first transform group, the size of the non-zero region is determined to be 32×32, and Wherein, based on the fact that the size of the current block is 32×32 and the transform kernel applied to the current block belongs to the second transform group, the size of the non-zero area is determined to be 16×16.

2. The method according to claim 1, wherein: The values ​​of coefficients of the remaining areas in the current block except the non-zero area are assigned to 0.

3. The method according to claim 1, wherein: The first transform group includes a DCT2 kernel and the second transform group does not include the DCT2 kernel.

4. The method according to claim 1, wherein: The first transform group includes a DCT2 kernel and the second transform group includes at least one of a DST7 kernel or a DCT8 kernel.

5. A method for encoding a video signal performed by an encoding device, the method comprising the following steps: deriving a transform kernel to be applied to the current block; determining a size of a non-zero region for a main transform based on the transform kernel; performing the primary transform on the current block based on the transform kernel, wherein non-zero coefficients are located in the non-zero region; generating transform index information indicating the transform kernel applied to the current block; and encoding the video signal including the transform index information, The transform kernel indicated by the transform index information includes a pair of horizontal transform kernel and vertical transform kernel, wherein, based on the current block having a size of 32×32 and the transform kernel applied to the current block belonging to the first transform group, the size of the non-zero region is determined to be 32×32, and Wherein, based on the fact that the size of the current block is 32×32 and the transform kernel applied to the current block belongs to the second transform group, the size of the non-zero area is determined to be 16×16.

6. The method according to claim 5, wherein: The values ​​of coefficients of the remaining areas in the current block except the non-zero area are assigned to 0.

7. The method according to claim 5, wherein: The first transform group includes a DCT2 kernel and the second transform group does not include the DCT2 kernel.

8. The method according to claim 5, wherein: The first transform group includes a DCT2 kernel and the second transform group includes at least one of a DST7 kernel or a DCT8 kernel. 9 . A computer-readable medium for storing a program, which, when executed by at least one processor, causes the at least one processor to perform the method according to claim 5 .

10. A method for transmitting data of a video signal, the method comprising the following steps: Obtaining a bitstream for the video signal, wherein the bitstream is generated based on the following steps: deriving a transform kernel applied to a current block; determining a size of a non-zero region for a main transform based on the transform kernel, performing the main transform on the current block based on the transform kernel, wherein non-zero coefficients are located in the non-zero region; generating transform index information indicating the transform kernel applied to the current block; and encoding the video signal including the transform index information; and sending said data comprising said bit stream, The transform kernel indicated by the transform index information includes a pair of horizontal transform kernel and vertical transform kernel, wherein, based on the current block having a size of 32×32 and the transform kernel applied to the current block belonging to the first transform group, the size of the non-zero region is determined to be 32×32, and Wherein, based on the fact that the size of the current block is 32×32 and the transform kernel applied to the current block belongs to the second transform group, the size of the non-zero area is determined to be 16×16.

Images