Video signal encoding / decoding method and device therefor

Abstract

To provide an image signal processing method and an apparatus which apply an appropriate transform to a current block.SOLUTION: A method for decoding a video signal according to an embodiment of the present invention may include steps of: determining, among predefined secondary transform sets, a secondary transform set applied to a current block based on intra-prediction modes of the current block; obtaining a first syntax element indicating a secondary transform matrix applied to the current block in the determined secondary transform set; deriving a secondary inverse-transformed block by performing a secondary inverse transform on a left top region of the current block using the secondary transform matrix specified by the first syntax element; and deriving a residual block of the current block by performing a primary inverse transform on the secondary inverse-transformed block using a primary transform matrix of the current block.SELECTED DRAWING: Figure 22

Description

[Technical Field]

[0001] The present invention relates to a method and apparatus for processing a video signal, and more particularly to a method and apparatus for encoding / decoding a video signal by performing a transformation. [Background technology]

[0002] Compression encoding refers to a series of signal processing techniques used to transmit digitized information over communication lines or store it in a suitable format for storage media. Media such as video, images, and audio can be compressed and encoded, and the technology that compresses and encodes video in particular is called video compression.

[0003] Next-generation video content will be characterized by high spatial resolution, high frame rate, and high dimensionality of scene representation, and processing such content will bring about a huge increase in memory storage, memory access rate, and processing power.

[0004] Therefore, it is necessary to design coding tools for more efficiently processing next-generation video content. In particular, video codec standards following the high efficiency video coding (HEVC) standard require more accurate prediction techniques as well as efficient conversion techniques for converting spatial domain video signals into frequency domain. Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE INVENTION It is an object of the present invention to provide an image signal processing method and apparatus that applies a transform appropriate for a current block.

[0006] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below. [Means for solving the problem]

[0007] One aspect of the present invention is a method for decoding a video signal, which may include the steps of: determining a secondary transform set to be applied to the current block from predefined secondary transform sets based on an intra prediction mode of the current block; obtaining a first syntax element indicating a secondary transform matrix to be applied to the current block within the determined secondary transform set; deriving a secondary inverse-transformed block by performing a secondary inverse transform on an upper left corner region of the current block using the secondary transform matrix identified by the first syntax element; and deriving a residual block of the current block by performing a primary inverse transform on the secondary inverse-transformed block using the primary transform matrix of the current block.

[0008] Preferably, each of the predefined quadratic transformation sets may include two quadratic transformation matrices.

[0009] Preferably, the step of deriving the second-order inverse transformed block may further include the step of determining an input length and an output length of the second-order inverse transform based on a width and a height of the current block.

[0010] Preferably, if the height and width of the current block are each 4, the input length of the non-separable transform can be determined as 8 and the output length as 16.

[0011] Preferably, the method may further include the steps of parsing a second syntax element indicating a linear transformation matrix to be applied to a linear transformation of the current block, and determining whether a secondary transformation can be applied to the current block based on the second syntax element.

[0012] Preferably, the step of determining whether a secondary transform can be applied can be performed by determining that a secondary transform can be applied to the current block if the second syntax element indicates a predefined specific transform type.

[0013] Preferably, the predefined specific transform type can be defined as DCT2.

[0014] Another aspect of the present invention is an apparatus for decoding a video signal, comprising: a memory for storing the video signal; and a processor coupled to the memory, wherein the processor determines a secondary transform set to be applied to the current block from among predefined secondary transform sets based on an intra prediction mode of the current block; obtains a first syntax element indicating a secondary transform matrix to be applied to the current block within the determined secondary transform set; derives a secondary inverse-transformed block by performing a secondary inverse transform on an upper left corner region of the current block using the secondary transform matrix specified by the first syntax element; and derives a residual block of the current block by performing a primary inverse transform on the secondary inverse-transformed block using a primary transform matrix of the current block. [Effects of the Invention]

[0015] According to an embodiment of the present invention, it is possible to improve the efficiency of transformation by determining and applying a transformation suitable for a current block.

[0016] Furthermore, according to an embodiment of the present invention, the transformations used in the primary and secondary transforms can be efficiently designed, thereby improving computational complexity and enhancing compression performance.

[0017] Furthermore, according to embodiments of the present invention, the computational complexity can be significantly improved by limiting the transform kernel of the primary transform to which the secondary transform is applied.

[0018] The effects obtained by the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the following description. [Brief explanation of the drawings]

[0019] The accompanying drawings, which are included as part of the detailed description to facilitate understanding of the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical features of the present invention.

[0020] [Figure 1] An example of a video coding system will be shown as an embodiment to which the present invention is applied. [Figure 2] 1 shows a schematic block diagram of an encoding device for encoding a video / image signal as an embodiment to which the present invention is applied. [Figure 3] 1 shows a schematic block diagram of a decoding device that decodes a video signal as an embodiment to which the present invention is applied. [Figure 4] 1 is a structural diagram of a content streaming system as an embodiment to which the present invention is applied; [Figure 5a] 1 is a diagram for explaining a block division structure using a QT (QuadTree, hereinafter referred to as "QT") as an embodiment to which the present invention can be applied. [Figure 5b]1 is a diagram illustrating a block division structure using a Binary Tree (BT, hereinafter referred to as "BT") as an embodiment to which the present invention can be applied. [Figure 5c] 1 is a diagram illustrating a block division structure using a ternary tree (TT, hereinafter referred to as "TT") as an embodiment to which the present invention can be applied. [Figure 5d] 1 is a diagram illustrating a block division structure using an Asymmetric Tree (hereinafter referred to as "AT") as an embodiment to which the present invention can be applied. [Figure 6] 1 shows a schematic block diagram of a transform and quantization unit and an inverse quantization and inverse transform unit in an encoding device as an embodiment to which the present invention is applied. [Figure 7] 1 shows a schematic block diagram of an inverse quantization and inverse transform unit in a decoding device as an embodiment to which the present invention is applied. [Figure 8] 1 is a flowchart showing a process in which AMT (adaptive multiple transform) is performed. [Figure 9] 10 is a flowchart illustrating a decoding process in which AMT is performed. [Figure 10] 10 is a flowchart illustrating an MTS-based inverse transformation process according to an embodiment of the present invention. [Figure 11] FIG. 2 is a block diagram of an apparatus for performing MTS-based decoding according to an embodiment of the present invention. [Figure 12] 1 is an encoding / decoding flowchart in which a secondary transform is applied as an embodiment to which the present invention is applied. [Figure 13] 1 is an encoding / decoding flowchart in which a secondary transform is applied as an embodiment to which the present invention is applied. [Figure 14] As an embodiment to which the present invention is applied, a diagram for explaining Givens rotation is shown. [Figure 15]As an embodiment to which the present invention is applied, a configuration of one round in a 4x4 NSST (non-separable secondary transform) composed of a Givens rotation layer and permutation is shown. [Figure 16] As an embodiment to which the present invention is applied, the operation of a reduced secondary transform (RST) will be shown. [Figure 17] 10 is a diagram illustrating a process of performing a reverse scan from the 64th to the 17th blocks based on a reverse scan order, according to an embodiment of the present invention. [Figure 18] As an embodiment to which the present invention is applied, an example of an encoding flowchart using a single transform indicator (STI) is shown. [Figure 19] As an embodiment to which the present invention is applied, an example of an encoding flowchart using a unified transform indicator (UTI) will be shown. [Figure 20a] Another example of an encoding flowchart using UTI will be shown as an embodiment to which the present invention is applied. [Figure 20b] Another example of an encoding flowchart using UTI will be shown as an embodiment to which the present invention is applied. [Figure 21] As an embodiment to which the present invention is applied, an example of an encoding flowchart for performing conversion will be shown. [Figure 22] As an embodiment to which the present invention is applied, an example of a decoding flowchart for performing conversion is shown. [Figure 23] As an embodiment to which the present invention is applied, an example of a detailed block diagram of the conversion unit 120 in the encoding device 100 is shown. [Figure 24] As an embodiment to which the present invention is applied, an example of a detailed block diagram of the inverse transform unit 230 in the decoding device 200 is shown. [Figure 25] 1 shows a flowchart for processing a video signal in an embodiment to which the present invention is applied. [Figure 26] 1 is a flowchart illustrating a method for converting a video signal according to an embodiment of the present invention; [Figure 27] 1 is a flowchart illustrating a method for converting a video signal according to an embodiment of the present invention; [Figure 28] 1 shows an example of a block diagram of an apparatus for processing a video signal as an embodiment to which the present invention is applied. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below together with the accompanying drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention can be practiced. The following detailed description includes specific details to provide a thorough understanding of the present invention. However, those skilled in the art will understand that the present invention can be practiced without such specific details.

[0022] In some cases, in order to avoid obscuring the concept of the present invention, well-known structures and devices may be omitted or shown in block diagram form, focusing on the core functions of each structure and device.

[0023] Furthermore, the terms used in this invention are currently common terms that are widely used whenever possible, but in certain cases, the applicant will use terms arbitrarily selected by the applicant to explain the invention. In such cases, the meanings of the terms will be clearly explained in the detailed description of the relevant part, so it is made clear that the terms should not be interpreted simply based on the names of the terms used in the description of this invention, but should be interpreted by understanding the meanings of the terms.

[0024] Specific terms used in the following description are provided to facilitate understanding of the present invention, and the use of such specific terms may be changed to other forms without departing from the technical spirit of the present invention. For example, in the case of a signal, data, sample, picture, frame, block, etc., they may be appropriately substituted and interpreted in each coding process.

[0025] Hereinafter, in this specification, the term "processing unit" refers to a unit in which encoding / decoding processes such as prediction, transformation, and / or quantization are performed. The term "processing unit" may also be interpreted to include a unit of a luma component and a unit of a chroma component. For example, the processing unit may correspond to a block, a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0026] In the following description, pixels or picture elements are commonly referred to as samples. Using a sample can mean using a pixel value or picture element value.

[0027] Furthermore, the processing units are not necessarily limited to square blocks, but may be configured in the form of polygons having three or more vertices.

[0028] In the following description, pixels or picture elements are commonly referred to as samples. Using a sample can mean using a pixel value or picture element value.

[0029] FIG. 1 shows an example of a video coding system as an embodiment to which the present invention is applied.

[0030] The video coding system may include a source device 10 and a receiving device 20. The source device 10 may transfer encoded video / image information or data to the receiving device 20 in a file or streaming format via a digital storage medium or a network.

[0031] The source device 10 may include a video source 11, an encoding device 12, and a transmitter 13. The receiving device 20 may include a receiver 21, a decoding device 22, and a renderer 23. The encoding device 10 may be referred to as a video / video encoding device, and the decoding device 20 may be referred to as a video / video decoding device. The transmitter 13 may be included in the encoding device 12. The receiver 21 may be included in the decoding device 22. The renderer 23 may include a display unit or may be configured as a separate device or an external component of the display unit.

[0032] A video source can acquire video / images through a video / image capture, synthesis, or generation process. A video source can include a video / image capture device and / or a video / image generation device. A video / image capture device can include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device can include, for example, a computer, a tablet, or a smartphone, and can (electronically) generate video / images. For example, a virtual video / image can be generated through a computer, in which case the video / image capture process can replace the process by which the associated data is generated.

[0033] The encoding device 12 can encode the input video / image. The encoding device 12 can perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0034] The transfer unit 13 can transfer the encoded video / image information or data output in the form of a bitstream to a receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transfer unit 13 can include elements for generating a media file in a predetermined file format and elements for transfer via a broadcast / communication network. The receiver 21 can extract the bitstream and transfer it to a decoding device 22.

[0035] The decoding device 22 can decode the video / image by performing a series of steps such as inverse quantization, inverse transform, and prediction corresponding to the operations of the encoding device 12.

[0036] A renderer 23 can render the decoded video / image, and the rendered video / image can be displayed on a display unit.

[0037] 2 is a schematic block diagram of an encoding device 100 for encoding a video / image signal as an embodiment to which the present invention is applied. The encoding device 100 in FIG. 2 may correspond to the encoding device 12 in FIG.

[0038] The image division unit 110 may divide an input image (or picture, frame) input to the encoding device 100 into one or more processing units. For example, the processing units may be called coding units (CUs). In this case, the coding units may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) based on a quad-tree binary-tree (QTBT) structure. For example, one coding unit may be divided into multiple coding units of deeper depths based on a quad-tree structure and / or a binary tree structure. In this case, for example, the quad-tree structure may be applied first, and then the binary tree structure may be applied. Alternatively, the binary tree structure may be applied first. The coding procedure according to the present invention may be performed based on the final coding unit that is not further divided. In this case, the largest coding unit may be used as the final coding unit immediately based on coding efficiency according to image characteristics, or the coding unit may be recursively divided into coding units of lower depths as needed, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include prediction, transformation, and restoration procedures, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be divided or partitioned from the final coding unit. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0039] The term unit is sometimes used interchangeably with terms such as block or area. In the general case, an MxN block can refer to a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally refer to a pixel or a pixel value, or only a pixel / pixel value of a luminance (luma) component, or only a pixel / pixel value of a chroma (chroma) component. A sample is used as a term to represent one picture (or image) as a pixel or pel.

[0040] The encoding apparatus 100 may subtract a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 180 or the intra prediction unit 185 from an input video signal (original block, original sample array) to generate a residual signal (residual signal, remaining block, remaining sample array), and the generated residual signal is transmitted to the conversion unit 120. In this case, as shown in the figure, a unit in the encoder 100 that subtracts a prediction signal (predicted block, prediction sample array) from an input video signal (original block, original sample array) may be referred to as a subtraction unit 115. The prediction unit may predict a target block (hereinafter referred to as a current block) and generate a predicted block including prediction samples of the current block. The prediction unit may determine whether intra prediction or inter prediction is applied for the current block or for each CU. The prediction unit may generate various information related to prediction, such as prediction mode information, as will be described later in the description of each prediction mode, and transmit the information to the entropy encoding unit 190. The prediction information can be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

[0041] The intra prediction unit 185 may predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located far away, depending on the prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the granularity of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the settings. The intra prediction unit 185 may also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0042] The inter prediction unit 180 may derive a predicted block of a current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between surrounding blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the surrounding blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring blocks may be the same or different. The temporal neighboring blocks may be referred to as collocated reference blocks, collocated (CU, colCU), etc., and the reference picture including the temporal neighboring blocks may be referred to as a collocated picture (colPic). For example, the inter predictor 180 may construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive a motion vector and / or a reference picture index for the current block. Inter prediction may be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter predictor 180 may use motion information of neighboring blocks as motion information for the current block. In the case of skip mode, unlike in merge mode, a residual signal may not be transmitted.In the case of motion vector prediction (MVP) mode, the motion vector of the current block can be indicated by signaling the motion vector difference using the motion vector of the surrounding block as the motion vector predictor.

[0043] The prediction signal generated via the inter prediction unit 180 or the intra prediction unit 185 is used to generate a reconstructed signal or a residual signal.

[0044] The transform unit 120 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of the following: a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, the GBT refers to a transform obtained from a graph representing inter-pixel relationship information. The CNT refers to a transform obtained based on a predicted signal generated using all previously reconstructed pixels. The transform process may be applied to square pixel blocks of the same size, or to non-square variable-size blocks.

[0045] The quantization unit 130 quantizes the transform coefficients and transmits them to the entropy encoding unit 190. The entropy encoding unit 190 encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. The information about the quantized transform coefficients may be referred to as residual information. The quantization unit 130 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and may generate information about the quantized transform coefficients based on the quantized transform coefficients in a one-dimensional vector form. The entropy encoding unit 190 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit 190 may encode information required for video / image restoration (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., video / image information) can be transmitted or stored in the form of a bitstream in NAL (network abstraction layer) units. The bitstream can be transmitted via a network or stored in a digital storage medium. Here, the network can include a broadcast network and / or a network, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitting unit (not shown) for transmitting the signal output from the entropy encoding unit 190 and / or a storing unit (not shown) for storing the signal can be configured as an element inside / outside the encoding device 100, or the transmitting unit can be a component of the entropy encoding unit 190.

[0046] The quantized transform coefficients output from the quantization unit 130 can be used to generate a prediction signal. For example, the quantized transform coefficients can be inversely quantized and inversely transformed via the inverse quantization unit 140 and the inverse transform unit 150 in a loop to reconstruct a residual signal. The adder 155 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit 180 or the intra prediction unit 185. When there is no residual for the current block, such as when skip mode is applied, the predicted block is used as the reconstructed block. The adder 155 can be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal is used for intra prediction of the next current block in the current picture, and may also be used for inter prediction of the next picture after filtering, as described below.

[0047] The filtering unit 160 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 160 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture and transfer the modified reconstructed picture to the decoded picture buffer 170. Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit 160 may generate various information related to filtering, as will be described later in the description of each filtering method, and transfer the information related to filtering to the entropy encoding unit 190. The entropy encoding unit 190 may encode the information related to filtering and output it in the form of a bitstream.

[0048] The modified reconstructed picture transferred to the decoded picture buffer 170 is used as a reference picture by the inter prediction unit 180. This allows the encoding device to avoid prediction mismatch between the encoding device 100 and the decoding device when inter prediction is applied, and also improves encoding efficiency.

[0049] The decoded picture buffer 170 can store the modified reconstructed picture for use as a reference picture from the inter predictor 180 .

[0050] 3 is a schematic block diagram of a decoding device for decoding a video signal as an embodiment to which the present invention is applied. The decoding device 200 in FIG. 3 may correspond to the decoding device 22 in FIG.

[0051] 3, the decoding device 200 may include an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an adder 235, a filtering unit 240, a decoded picture buffer (DPB) 250, an inter prediction unit 260, and an intra prediction unit 265. The inter prediction unit 260 and the intra prediction unit 265 may be collectively referred to as a prediction unit. That is, the prediction unit may include an inter prediction unit 180 and an intra prediction unit 185. The inverse quantization unit 220 and the inverse transform unit 230 may be collectively referred to as a residual processing unit. That is, the residual processing unit may include an inverse quantization unit 220 and an inverse transform unit 230. The entropy decoding unit 210, the inverse quantization unit 220, the inverse transform unit 230, the adder 235, the filtering unit 240, the inter prediction unit 260, and the intra prediction unit 265 may be configured as a single hardware component (e.g., a decoder or a processor) according to an embodiment. Additionally, the decoded picture buffer 250 may be implemented by a single hardware component (eg, a memory or digital storage medium) depending on the embodiment.

[0052] When a bitstream including video / image information is input, the decoding device 200 can reconstruct an image corresponding to the process by which the video / image information from the encoding device 100 of Fig. 2 was processed. For example, the decoding device 200 can perform decoding using the processing unit applied in the encoding device 100. Therefore, the processing unit for decoding can be, for example, a coding unit, and the coding unit can be divided into a coding tree unit or a maximum coding unit according to a quad tree structure and / or a binary tree structure. The reconstructed video signal decoded and output by the decoding device 200 can be played back through a playback device.

[0053] The decoding apparatus 200 may receive a signal output from the encoding apparatus 100 of FIG. 2 in the form of a bitstream, and the received signal may be decoded via an entropy decoding unit 210. For example, the entropy decoding unit 210 may derive information (e.g., video / image information) necessary for image restoration (or picture restoration) by farsizing (analyzing) the bitstream. For example, the entropy decoding unit 210 may decode information in the bitstream based on a coding method such as Exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements necessary for image restoration and quantized values ​​of transform coefficients related to residuals. In more detail, the CABAC entropy decoding method receives bins corresponding to each syntax element from the bitstream, determines a context model using information on the syntax element to be decoded and decoded information on neighboring and current blocks, or information on symbols / bins decoded in a previous stage, predicts the occurrence probability of the bin based on the determined context model, and performs arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. After determining the context model, the CABAC entropy decoding method may update the context model using information on the decoded symbol / bin for the context model of the next symbol / bin. Prediction information among the information decoded by the entropy decoding unit 210 is provided to the prediction unit (inter prediction unit 260 and intra prediction unit 265), and entropy-decoded residual values, i.e., quantized transform coefficients and related parameter information, from the entropy decoding unit 210 may be input to the inverse quantization unit 220. In addition, information regarding filtering among the information decoded by the entropy decoding unit 210 may be provided to the filtering unit 240. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding device 100 may be further configured as an internal / external element of the decoding device 200, or the receiving unit may be a component of the entropy decoding unit 210.

[0054] The inverse quantization unit 220 may inverse quantize the quantized transform coefficients to output transform coefficients. The inverse quantization unit 220 may rearrange the quantized transform coefficients into a two-dimensional block format. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding apparatus 100. The inverse quantization unit 220 may inverse quantize the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0055] The inverse transform unit 230 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0056] The prediction unit may predict a current block and generate a predicted block including prediction samples of the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoding unit 210, and may determine a specific intra / inter prediction mode.

[0057] The intra prediction unit 265 may predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located far away from the current block depending on the prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 265 may also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0058] The inter prediction unit 260 may derive a predicted block of the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between surrounding blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the surrounding blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. For example, the inter prediction unit 260 may construct a motion information candidate list based on the surrounding blocks and derive a motion vector and / or a reference picture index for the current block based on received candidate selection information. Inter prediction may be performed based on various prediction modes, and the prediction information may include information indicating the inter prediction mode of the current block.

[0059] The adder 235 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed, sample array) by adding the acquired residual signal to the predicted signal (predicted block, predicted sample array) output from the inter prediction unit 260 or intra prediction unit 265. As in the case where skip mode is applied, if there is no residual for the current block, the predicted block is used as the reconstructed block.

[0060] The adder 235 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of the next block to be processed in the current picture, and may also be used for inter prediction of the next picture after filtering, as described below.

[0061] The filtering unit 240 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 240 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture and transfer the modified reconstructed picture to the decoded picture buffer 250. Various filtering methods may include, for example, deblocking filtering, sample adaptive offset (SAO), adaptive loop filter (ALF), bilateral filter, etc.

[0062] The modified reconstructed picture transferred to the decoded picture buffer 250 is used as a reference picture by the inter predictor 260 .

[0063] In this specification, the embodiments described for the filtering unit 160, inter prediction unit 180 and intra prediction unit 185 of the encoding device 100 can also be applied identically or correspondingly to the filtering unit 240, inter prediction unit 260 and intra prediction unit 265 of each decoding device.

[0064] FIG. 4 is a structural diagram of a content streaming system according to an embodiment of the present invention.

[0065] The content streaming system to which the present invention is applied can broadly include an encoding server 410 , a streaming server 420 , a web server 430 , a media storage 440 , a user device 450 and a multimedia input device 460 .

[0066] The encoding server 410 compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server 420. As another example, if a multimedia input device 460 such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server 410 can be omitted.

[0067] The bitstream can be generated by an encoding method or a bitstream generating method to which the present invention is applied, and the streaming server 420 can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0068] The streaming server 420 transfers multimedia data to the user device 450 based on a user request via the web server 430, and the web server 430 acts as an intermediary informing the user of available services. When a user requests a desired service from the web server 430, the web server 430 transfers the request to the streaming server 420, which then transfers the multimedia data to the user. In this case, the content streaming system may include a separate control server, which controls commands and responses between devices in the content streaming system.

[0069] The streaming server 420 can receive content from the media storage 440 and / or the encoding server 410. For example, when receiving content from the encoding server 410, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server 420 can store the bitstream for a certain period of time.

[0070] Examples of user device 450 include mobile phones, smartphones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, and head-mounted displays (HMDs)), digital TVs, desktop computers, and digital signage.

[0071] Each server in the content streaming system can be operated as a distributed server, in which case data received by each server can be processed in a distributed manner.

[0072] Figure 5 is a diagram illustrating block division structures according to embodiments to which the present invention can be applied, where Figure 5a is a QT (QuadTree, QT), Figure 5b is a BT (Binary Tree, BT), Figure 5c is a TT (Ternary Tree, TT), and Figure 5d is an AT (Asymmetric Tree, AT).

[0073] In video coding, a block can be divided based on the QT. A subblock divided by the QT can be further divided recursively using the QT. A leaf block that is no longer divided by the QT can be divided by at least one of BT, TT, or AT. BT can have two types of division: horizontal BT (2NxN, 2NxN) and vertical BT (Nx2N, Nx2N). TT can have two types of division: horizontal TT (2Nx1 / 2N, 2NxN, 2Nx1 / 2N) and vertical TT (1 / 2Nx2N, Nx2N, 1 / 2Nx2N). AT can have four types of division: horizontal-up AT (2Nx1 / 2N, 2Nx3 / 2N), horizontal-down AT (2Nx3 / 2N, 2Nx1 / 2N), vertical-left AT (1 / 2Nx2N, 3 / 2Nx2N), vertical-right AT (3 / 2Nx2N, 1 / 2Nx2N). Each BT, TT, and AT can be further divided recursively using BT, TT, and AT.

[0074] Figure 5a shows an example of QT partitioning. Block A can be partitioned into four sub-blocks (A0, A1, A2, A3) by QT. Sub-block A1 can be partitioned again into four sub-blocks (B0, B1, B2, B3) by QT.

[0075] Figure 5b shows an example of BT division. Block B3, which is no longer divided by QT, can be divided into vertical BT (C0, C1) or horizontal BT (D0, D1). Like block C0, each sub-block can be further divided recursively, such as into horizontal BT (E0, E1) or vertical BT (F0, F1).

[0076] Figure 5c shows an example of TT division. Block B3, which is no longer divided by QT, can be divided into vertical TTs (C0, C1, C2) or horizontal TTs (D0, D1, D2). Like block C1, each sub-block can be further divided recursively, such as into horizontal TTs (E0, E1, E2) or vertical TTs (F0, F1, F2).

[0077] Figure 5d shows an example of AT division. Block B3, which is no longer divided by QT, can be divided into vertical AT(C0, C1) or horizontal AT(D0, D1). Like block C1, each sub-block can be further divided recursively, such as into horizontal AT(E0, E1) or vertical TT(F0, F1).

[0078] Meanwhile, BT, TT, and AT divisions can be used together for division. For example, sub-blocks divided by BT can be divided by TT or AT. Also, sub-blocks divided by TT can be divided by BT or AT. Sub-blocks divided by AT can be divided by BT or TT. For example, after horizontal BT division, each sub-block can be divided into vertical BT, or after vertical BT division, each sub-block can be divided into horizontal BT. In this case, the division order is different, but the final division form is the same.

[0079] Also, when a block is divided, the order of searching the block can be defined in various ways. Generally, searching from left to right and from top to bottom means the order of determining whether to divide an additional block for each divided sub-block, or the encoding order of each sub-block if the block is no longer divided, or the search order when a sub-block refers to information on other adjacent blocks.

[0080] Transforms can be performed for each processing unit (or transform block) divided according to the division structure shown in Figures 5a to 5d, and in particular, transformation matrices can be applied by dividing the processing unit into rows and columns. According to an embodiment of the present invention, different transform types are used depending on the length of the processing unit (or transform block) in the row or column direction.

[0081] Transforms are applied to residual blocks to decorrelate them as much as possible, concentrate coefficients in low frequencies, and create a zero tail at the beginning of the block. In JEM software, the transform part includes two main functions: core transform and secondary transform. Core transforms consist of the DCT (discrete cosine transform) and DST (discrete sine transform) family, which are applied to every row and column of the residual block. Secondary transforms can then be additionally applied to the top left corner of the core transform output. Similarly, inverse secondary transforms and inverse transforms of the core transform order can be applied. First, the secondary transform can be applied to the top left corner of the coefficient block. The core transform is then applied to the rows and columns of the secondary transform output. The core transform or inverse transform can also be referred to as the primary transform or inverse transform.

[0082] 6 and 7 show embodiments to which the present invention is applied. FIG. 6 shows a schematic block diagram of the transform and quantization unit (120 / 130) and the inverse quantization and inverse transform unit (140 / 150) in the encoding device 100 of FIG. 2, and FIG. 7 shows a schematic block diagram of the inverse quantization and inverse transform unit (220 / 230) in the decoding device 200.

[0083] 6, the transform and quantization unit (120 / 130) may include a primary transform unit 121, a secondary transform unit 122, and a quantization unit 130. The inverse quantization and inverse transform unit (140 / 150) may include an inverse quantization unit 140, an inverse secondary transform unit 151, and an inverse primary transform unit 152.

[0084] Looking carefully at FIG. 7, the inverse quantization and inverse transform unit (220 / 230) may include an inverse quantization unit 220, an inverse secondary transform unit 231, and an inverse primary transform unit 232.

[0085] In the present invention, when a transformation is performed, the transformation can be performed through multiple stages. For example, as shown in Figure 6, two stages of a primary transform and a secondary transform can be applied, or more transformation stages can be used depending on the algorithm. Here, the primary transform can be referred to as a core transform.

[0086] The linear transform unit 121 can apply a linear transform to the residual signal, where the linear transform can be predefined in a table from the encoder and / or decoder.

[0087] The secondary transform unit 122 can apply a secondary transform to the primary transformed signal, where the secondary transform can be predefined in a table from the encoder and / or decoder.

[0088] In one embodiment, a non-separable secondary transform (NSST) may be conditionally applied as the secondary transform. For example, the NSST may be applied only to intra-predicted blocks, and each prediction mode group may have its own applicable transform set.

[0089] Here, the prediction mode group may be set based on the symmetry of the prediction direction. For example, since the prediction mode 52 and the prediction mode 16 are symmetric based on the prediction mode 34 (diagonal direction), they form one group and the same transform set may be applied. In this case, when applying the transform of the prediction mode 52, the input data is transposed before being applied, because the transform set is the same as that of the prediction mode 16.

[0090] On the other hand, for the planar mode and DC mode, there is no directional symmetry, so each mode has a transformation set, which can consist of two transformations, while for the remaining directional modes, each transformation set can consist of three transformations.

[0091] The quantization unit 130 can quantize the quadratic transformed signal.

[0092] The inverse quantization and inverse transformation unit (140 / 150) performs the process described above in reverse, and a duplicated description will be omitted.

[0093] FIG. 7 shows a schematic block diagram of the inverse quantization and inverse transform unit (220 / 230) in the decoding device 200.

[0094] Referring to FIG. 7, the inverse quantization and inverse transform unit (220 / 230) may include an inverse quantization unit 220, an inverse secondary transform unit 231, and an inverse primary transform unit 232.

[0095] The inverse quantization unit 220 uses the quantization step size information to obtain transform coefficients from the entropy decoded signal.

[0096] The inverse quadratic transform unit 231 performs an inverse quadratic transform on the transform coefficients. Here, the inverse quadratic transform refers to the inverse transform of the quadratic transform described with reference to FIG.

[0097] The inverse primary transform unit 232 performs an inverse primary transform on the inverse quadratic transformed signal (or block) to obtain a residual signal. Here, the inverse primary transform refers to the inverse transform of the primary transform described in FIG. 6.

[0098] In addition to the DCT-2 and 4x4 DST-4 applied in HEVC, the adaptive (or explicit) multiple transform (AMT or EMT) technique is used for residual coding of inter- and intra-encoded blocks. Besides the HEVC transforms, a number of selected transforms from other DCT / DST families are used. The newly introduced transform matrices from JEM are DST-7, DCT-8, DST-1, and DCT-5. Table 1 below shows the selected DST / DCT basis functions.

[0099] [Table 1]

[0100] EMT can be applied to CUs with width and height less than or equal to 64, and whether EMT is applied can be controlled by a CU-level flag. If the CU-level flag is 0, DCT-2 is applied to the CU to encode the residual. For intra-CU luma coding blocks to which EMT is applied, two additional flags are signaled to identify the horizontal and vertical transforms used. Like HEVC, the residuals of JEM blocks can be coded in transform skip mode. For intra-residual coding, a mode-dependent transform candidate selection process is used, depending on the residual statistics of other intra-prediction modes. Three transform subsets are defined as shown in Table 2 below, and the transform subset is selected based on the intra-prediction mode as shown in Table 3.

[0101] [Table 2]

[0102] With the subset concept, a subset of transforms is first identified based on Table 2 by using the intra prediction modes of CUs with CU-level EMT_CU_flag set to 1. Then, for each horizontal (EMT_TU_horizontal_flag) and vertical (EMT_TU_vertical_flag) transform, one of the two candidate transforms in the identified subset of transforms is selected based on explicit signaling using flags based on Table 3.

[0103] [Table 3]

[0104] [Table 4]

[0105] Table 4 shows a transform configuration group to which AMT (adaptive multiple transform) is applied as an embodiment to which the present invention is applied.

[0106] Looking carefully at Table 4, transform configuration groups are determined based on prediction modes, and there can be a total of six groups (G0 to G5). G0 to G4 correspond to cases where intra prediction is applied, and G5 indicates a combination (or transform set, or transform combination set) of transforms applied to a residual block generated by inter prediction.

[0107] A combination of transformations can be made by a horizontal transform (or row transform) applied to the rows of the 2D block and a vertical transform (or column transform) applied to the columns.

[0108] Here, each group of transform settings can have four candidate transform combinations, which can be selected or determined using transform combination indices 0 to 3, and the transform combination indices can be encoded and transmitted from the encoder to the decoder.

[0109] In one embodiment, residual data (or residual signals) obtained through intra prediction may have different statistical characteristics depending on the intra prediction mode. Therefore, a different transform may be applied to each intra prediction mode instead of a general cosine transform, as shown in Table 4. In this specification, the transform type may be represented as, for example, DCT-Type 2, DCT-II, or DCT-2.

[0110] A careful look at Table 4 reveals cases where 35 intra prediction modes are used and cases where 67 intra prediction modes are used. Multiple transform combinations can be applied to each transform setting group, which is divided into columns for each intra prediction mode. For example, multiple transform combinations can be configured with four combinations (row-direction transform, column-direction transform). For example, in group 0, DST-7 and DCT-5 can be applied to both the row (horizontal) and column (vertical) directions, resulting in a total of four possible combinations.

[0111] Since a total of four transform kernel combinations can be applied to each intra prediction mode, a transform combination index for selecting one of them can be transmitted for each transform unit. In this specification, the transform combination index can be referred to as an AMT index and can be represented by amt_idx.

[0112] In addition to the transform kernels listed in Table 4, there may be cases where DCT-2 is optimal for both row and column directions due to the characteristics of the residual signal. Therefore, by defining an AMT flag for each coding unit, transforms can be applied adaptively. Here, if the AMT flag is 0, DCT-2 is applied to both row and column directions, and if the AMT flag is 1, one of the four combinations can be selected or determined using the AMT index.

[0113] In one embodiment, when the AMT flag is 0, if the number of transform coefficients in one transform unit is less than 3, the transform kernel in Table 4 is not applied, and DST-7 can be applied in both the row and column directions.

[0114] In one embodiment, the values ​​of the transform coefficients are parsed first, and if the number of transform coefficients is less than 3, the AMT index is not parsed and DST-7 is applied, thereby reducing the amount of additional information transmitted.

[0115] In one embodiment, AMT can be applied only if the width and height of the transform unit are both 32 or less.

[0116] In one embodiment, Table 4 can be established via off-line training.

[0117] In one embodiment, the AMT index can be defined as a single index that can simultaneously refer to a combination of horizontal and vertical transforms, or the AMT index can be defined as separate horizontal and vertical transform indexes.

[0118] FIG. 8 is a flowchart showing the process of performing AMT (adaptive multiple transform).

[0119] Although this specification primarily describes an embodiment of a separable transform in which transformations are applied separately in the horizontal and vertical directions, the combination of transformations can also be configured into a non-separable transform.

[0120] Alternatively, a combination of separable and non-separable transforms can be configured. In this case, if a non-separable transform is used, row / column or horizontal / vertical direction selection is not required, and the transform combinations in Table 4 can be used only when a separable transform is selected.

[0121] In addition, the method proposed in this specification can be applied regardless of whether it is a linear transform or a secondary transform. That is, there is no restriction that it must be applied to only one of the two, and it can be applied to both. Here, the linear transform may refer to a transform for initially transforming a residual block, and the secondary transform may refer to a transform for applying a transform to a block generated as a result of the linear transform.

[0122] First, the encoding device 100 may determine a transform group corresponding to the current block (S805). Here, the transform group may refer to the transform group in Table 4, but the present invention is not limited thereto and may be configured with other transform combinations.

[0123] The encoding device 100 may perform a transform on a combination of candidate transforms to be used in the transform group (S810). As a result of the transform execution, the encoding device 100 may determine or select a combination of transforms with the smallest rate distortion (RD) cost (S815). The encoding device 100 may encode an index of the transform combination corresponding to the selected transform combination (S820).

[0124] FIG. 9 is a flowchart showing the decoding process in which AMT is performed.

[0125] First, the decoding apparatus 200 may determine a transform group for a current block (S905). The decoding apparatus 200 may analyze a transform combination index, where the transform combination index may correspond to one of a plurality of transform combinations in the transform group (S910). The decoding apparatus 200 may derive a transform combination corresponding to the transform combination index (S915). Here, the transform combination may refer to the transform combinations described in Table 4, but the present invention is not limited thereto. That is, configurations using other transform combinations are also possible.

[0126] The decoding device 200 may perform an inverse transform on the current block based on the transform combination (S920). If the transform combination is composed of a row transform and a column transform, the row transform may be applied first, and then the column transform may be applied. However, the present invention is not limited thereto, and if the transform combination is composed of non-separable transforms, the non-separable transform may be applied directly.

[0127] Meanwhile, in another embodiment, the process of determining the transformation group and the process of analyzing the transformation combination index can be performed simultaneously.

[0128] According to an embodiment of the present invention, the aforementioned term "AMT" can be redefined as "MTS (multiple transform set or multiple transform selection)". The MTS-related syntax and semantics described below are defined in the versatile video coding (VVC) standard document JVET-K1001-v4.

[0129] In an embodiment of the present invention, two MTS candidates are used for the directional mode and four MTS candidates for the non-directional mode as follows:

[0130] A) Non-directional mode (DC, Planner)

[0131] When the MTS index is 0, DST-7 is used for the horizontal and vertical transforms.

[0132] When the MTS index is 1, the DST-7 is used for the vertical transform and the DCT-8 is used for the horizontal transform.

[0133] When the MTS index is 2, a DCT-8 is used for the vertical transform and a DST-7 is used for the horizontal transform.

[0134] When the MTS index is 3, the DCT-8 is used for the horizontal and vertical transforms.

[0135] B) Modes belonging to the horizontal group mode

[0136] When the MTS index is 0, DST-7 is used for the horizontal and vertical transforms.

[0137] When the MTS index is 1, a DCT-8 is used for the vertical transform and a DST-7 is used for the horizontal transform.

[0138] C) Modes belonging to the vertical group mode

[0139] When the MTS index is 0, DST-7 is used for horizontal and vertical transformation.

[0140] When the MTS index is 1, the DST-7 is used for the vertical transform and the DCT-8 is used for the horizontal transform.

[0141] Here (in VTM 2.0, where 67 modes are used), the horizontal group modes include intra prediction modes 2 through 34, and the vertical modes include intra prediction modes 35 through 66.

[0142] In another embodiment of the present invention, three MTS candidates are used for all intra-prediction modes.

[0143] When the MTS index is 0, DST-7 is used for the horizontal and vertical transforms.

[0144] When the MTS index is 1, the DST-7 is used for the vertical transform and the DCT-8 is used for the horizontal transform.

[0145] When the MTS index is 2, a DCT-8 is used for the vertical transform and a DST-7 is used for the horizontal transform.

[0146] In another embodiment of the present invention, two MTS candidates are used for directional prediction modes and three MTS candidates for non-directional prediction modes.

[0147] A) Non-directional mode (DC, Planner)

[0148] When the MTS index is 0, DST-7 is used for the horizontal and vertical transforms.

[0149] When the MTS index is 1, the DST-7 is used for the vertical transform and the DCT-8 is used for the horizontal transform.

[0150] When the MTS index is 2, a DCT-8 is used for the vertical transform and a DST-7 is used for the horizontal transform.

[0151] B) Prediction mode corresponding to horizontal group mode

[0152] When the MTS index is 0, DST-7 is used for the horizontal and vertical transforms.

[0153] When the MTS index is 1, a DCT-8 is used for the vertical transform and a DST-7 is used for the horizontal transform.

[0154] C) Prediction mode corresponding to vertical group mode

[0155] When the MTS index is 0, DST-7 is used for the horizontal and vertical transforms.

[0156] When the MTS index is 1, the DST-7 is used for the vertical transform and the DCT-8 is used for the horizontal transform.

[0157] In another embodiment of the present invention, one MTS candidate (e.g., DST-7) is used for all intra modes. In this case, encoding time can be reduced by up to 40% with minor coding loss. Furthermore, one flag is used to indicate between DCT-2 and DST-7.

[0158] FIG. 10 is a flowchart illustrating an MTS-based inverse transformation process according to an embodiment of the present invention.

[0159] The decoding device 200 to which the present invention is applied may acquire sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag (S1005). Here, sps_mts_intra_enabled_flag indicates whether cu_mts_flag is present in the residual coding syntax of the intra coding unit. For example, if sps_mts_intra_enabled_flag = 0, cu_mts_flag is not present in the residual coding syntax of the intra coding unit. If sps_mts_intra_enabled_flag = 1, cu_mts_flag is present in the residual coding syntax of the intra coding unit. And, sps_mts_inter_enabled_flag indicates whether cu_mts_flag is present in the residual coding syntax of the inter coding unit. For example, if sps_mts_inter_enabled_flag=0, cu_mts_flag is not present in the residual coding syntax of the inter coding unit, and if sps_mts_inter_enabled_flag=1, cu_mts_flag is present in the residual coding syntax of the inter coding unit.

[0160] The decoding apparatus 200 may acquire cu_mts_flag based on sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag (S1010). For example, when sps_mts_intra_enabled_flag = 1 or sps_mts_inter_enabled_flag = 1, the decoding apparatus 200 may acquire cu_mts_flag. Here, cu_mts_flag indicates whether MTS is applied to residual samples of a luma transform block. For example, when cu_mts_flag = 0, MTS is not applied to residual samples of a luma transform block, and when cu_mts_flag = 1, MTS is applied to residual samples of a luma transform block.

[0161] The decoding apparatus 200 may acquire mts_idx based on cu_mts_flag (S1015). For example, when cu_mts_flag = 1, the decoding apparatus 200 may acquire mts_idx, where mts_idx indicates which transform kernel is applied to the luma residual samples along the horizontal and / or vertical directions of the current transform block.

[0162] For example, for mts_idx, at least one of the embodiments described herein can be applied.

[0163] The decoding apparatus 200 may derive a transform kernel corresponding to mts_idx (S1020). For example, the transform kernel corresponding to mts_idx may be defined as a horizontal transform and a vertical transform.

[0164] For example, if MTS is applied to the current block (i.e., cu_mts_flag = 1), the decoding apparatus 200 may configure MTS candidates based on the intra prediction mode of the current block. In this case, the decoding flowchart of Figure 10 may further include a step of configuring MTS candidates. Then, the decoding apparatus 200 may determine an MTS candidate to be applied to the current block from the configured MTS candidates using mts_idx.

[0165] As another example, different transform kernels may be applied to the horizontal transform and the vertical transform, although the present invention is not limited thereto and the same transform kernel may be applied to the horizontal transform and the vertical transform.

[0166] Then, the decoding device 200 can perform an inverse transform based on the transform kernel (S1025).

[0167] In addition, in this document, MTS can also be expressed as AMT or EMT, and similarly, it can be expressed as mts_idx, AMT_idx, EMT_idx, AMT_TU_idx, EMT_TU_idx, etc., but the present invention is not limited to such expressions.

[0168] Furthermore, although the present invention will be described by distinguishing between cases where MTS is applied and cases where it is not based on the MTS flag, the present invention is not limited to such expression. For example, whether or not MTS is applied may mean the same as whether or not a transform type (or transform kernel) other than a predefined specific transform type (which may also be referred to as a base transform type, default transform type, etc.) is used. If MTS is applied, a transform type other than the base transform type (e.g., any one of multiple transform types or a combination of two or more transform types) may be used for the transform, and if MTS is not applied, the base transform type may be used for the transform. In one embodiment, the base transform type may be set (or defined) to DCT2.

[0169] As one example, an MTS flag syntax indicating whether MTS is applied to the current transform block and an MTS index syntax indicating the transform type applied to the current block if MTS is applied may be transmitted separately from the encoder to the decoder, and as another example, a syntax (e.g., an MTS index) including both whether MTS is applied to the current transform block and the transform type applied to the current block if MTS is applied may be transmitted from the encoder to the decoder. That is, in the latter embodiment, a syntax (or syntax element) indicating the transform type applied to the current transform block (or unit) within the entire transform type group (or transform type set) including the above-mentioned basic transform types may be transmitted from the encoder to the decoder.

[0170] Therefore, despite this expression, the syntax (MTS index) indicating the transform type applied to the current transform block can include information on whether or not to apply MTS. In other words, in the latter embodiment, since only the MTS index is signaled without the MTS flag, in this case it can be interpreted that the MTS includes DCT2, but in the present invention, the case where DCT2 is applied may be described as the case where MTS is not applied, and the technical scope of MTS is nevertheless not limited to this definition.

[0171] FIG. 11 is a block diagram of an apparatus for performing MTS-based decoding according to an embodiment of the present invention.

[0172] The decoding device 200 to which the present invention is applied may include a sequence parameter acquisition unit 1105 , an MTS flag acquisition unit 1110 , an MTS index acquisition unit 1115 , and a transformation kernel derivation unit 1120 .

[0173] The sequence parameter acquisition unit 1105 may acquire sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag. Here, sps_mts_intra_enabled_flag indicates whether cu_mts_flag is present in the residual coding syntax of an intra coding unit, and sps_mts_inter_enabled_flag indicates whether cu_mts_flag is present in the residual coding syntax of an inter coding unit. For a specific example, the description related to FIG. 10 may be applied.

[0174] The MTS flag acquirer 1110 may acquire cu_mts_flag based on sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag. For example, when sps_mts_intra_enabled_flag = 1 or sps_mts_inter_enabled_flag = 1, the MTS flag acquirer 1110 may acquire cu_mts_flag. Here, cu_mts_flag indicates whether MTS is applied to residual samples of a luma transform block. For a specific example, the description related to FIG. 10 may be applied.

[0175] The MTS index obtainer 1115 may obtain mts_idx based on cu_mts_flag. For example, when cu_mts_flag = 1, the MTS index obtainer 1115 may obtain mts_idx. Here, mts_idx indicates which transform kernel is applied to luma residual samples along the horizontal and / or vertical directions of the current transform block. For a specific example, the description of FIG. 10 may be applied.

[0176] The transform kernel deriving unit 1120 can derive a transform kernel corresponding to mts_idx, and the decoding device 200 can perform an inverse transform based on the derive transform kernel.

[0177] A mode-dependent non-separable secondary transform (MDNSST) is introduced. To maintain low complexity, MDNSST is applied only to low-frequency coefficients after the primary transform. A non-separable transform applied primarily to low-frequency coefficients can also be called a low-frequency non-separable transform (LFNST). If the width (W) and height (H) of a transform coefficient block are both equal to or greater than 8, an 8x8 non-separable secondary transform is applied to the upper-left 8x8 region of the transform coefficient block. Otherwise, if the width or height is less than 8, a 4x4 non-separable secondary transform is applied. The 4x4 non-separable secondary transform can be performed on the upper-left min(8,W) x min(8,H) region of the transform coefficient block. Here, min(A,B) is a function that outputs the smaller of A and B. WxH is the block size, W is the width, and H is the height.

[0178] For 4x4 and 8x8 block sizes, there can be a total of 35x3 non-separable quadratic transforms, where 35 is the number of transform sets specified by the intra prediction mode and 3 is the number of NSST candidates for each prediction mode. The mapping from intra prediction mode to transform sets can be defined as shown in Table 5 below. Also, according to an embodiment of the present invention, among the four non-separable transform sets, depending on the intra prediction mode:

[0179] [Table 5]

[0180] An NSST index (NSST idx) may be coded to indicate a transform kernel within a transform set. If NSST is not applied, an NSST index with a value of 0 may be signaled.

[0181] 12 and 13 are encoding / decoding flowcharts in which a secondary transform is applied as an embodiment to which the present invention is applied.

[0182] In JEM, the secondary transform (MDNSST) is not applied to blocks coded in transform skip mode. If the MDNSST index is signaled for a CU and is non-zero, MDNSST is not used for blocks of components coded in transform skip mode within the CU. The overall coding structure, including coefficient encoding and NSST index coding, is shown in Figures 12 and 13. A coded block flag (CBF) is encoded to determine whether to perform coefficient encoding or NSST coding. In Figures 12 and 13, the CBF flag can indicate a luma block cbf flag (cbf_luma flag) or a chroma block cbf flag (cbf_cb flag or cbf_cr flag). Transform coefficients are coded when the CBF flag is 1.

[0183] Referring to FIG. 12, the encoding apparatus 100 checks whether the CBF is 1 (S1205). If the CBF is 0, the encoding apparatus 100 does not encode the transform coefficients or the NSST indexes. If the CBF is 1, the encoding apparatus 100 encodes the transform coefficients (S1210). Thereafter, the encoding apparatus 100 determines whether to perform NSST index coding (S1215) and performs NSST index coding (S1220). If NSST index coding is not applied, the encoding apparatus 100 may end the transform procedure in a state where NSST is not applied and perform subsequent steps (e.g., quantization).

[0184] 13, the decoding apparatus 200 checks whether the CBF is 1 (S1305). If the CBF is 0, the decoding apparatus 200 does not perform transform coefficient decoding or NSST index decoding. If the CBF is 1, the decoding apparatus 200 performs transform coefficient decoding (S1310). Thereafter, the decoding apparatus 200 determines whether to perform NSST index coding (S1315) and analyzes the NSST index (S1320).

[0185] NSST is not applied to the entire block (TU in the case of HEVC) to which the primary transform has been applied, but can be applied to the upper left 8x8 region or 4x4 region. For example, if the block size is 8x8 or greater, 8x8 NSST can be applied, and if the block size is less than 8x8, 4x4 NSST can be applied. Also, if 8x8 NSST is applied, 4x4 NSST can be applied to each 4x4 block. Both 8x8 NSST and 4x4 NSST can be determined according to the configuration of the transform set described above. As they are non-separable transforms, 8x8 NSST has 64 input data and 64 output data, while 4x4 NSST has 16 inputs and 16 outputs.

[0186] 14 and 15 are embodiments to which the present invention is applied. FIG. 14 illustrates a diagram for explaining Givens rotation, and FIG. 15 illustrates the configuration of one round in a 4x4 NSST consisting of a Givens rotation layer and permutation.

[0187] Both the 8x8 NSST and the 4x4 NSST can be constructed by a hierarchical combination of Givens rotations. The matrix corresponding to one Givens rotation is the same as Equation 1, and the matrix multiplication can be represented graphically as Figure 14.

[0188]

number

[0189] In FIG. 14, tm and tn output by the Givens rotation can be calculated as in Equation 2.

[0190]

number

[0191] As shown in Figure 14, one Givens rotation rotates two data points, so 32 or 8 Givens rotations are required to process 64 data points (for 8x8 NSST) or 16 data points (for 4x4 NSST), respectively. Therefore, a bundle of 32 or 8 Givens rotations can form a Givens rotation layer. As shown in Figure 15, the output data of one Givens rotation layer is shuffled and transmitted to the input data of the next Givens rotation layer. As shown in Figure 15, the permutation pattern is regularly defined. In the case of 4x4 NSST, four Givens rotation layers and corresponding permutations form one round. 4x4 NSST is performed in two rounds, while 8x8 NSST is performed in four rounds. Different rounds use the same permutation pattern but apply different Givens rotation angles. Therefore, it is necessary to store the angle data of all Givens rotations that make up each transformation.

[0192] In the final stage, a final permutation is performed on the data output through the Givens rotation layer, and the permutation information is stored separately for each transformation. This permutation is performed at the end of the forward NSST, and the inverse permutation is applied first to the inverse NSST.

[0193] The reverse NSST rotates the Givens rotation layers and permutations applied in the forward NSST in the reverse order, and also takes a negative (-) value for each Givens rotation angle.

[0194] RST (Reduced secondary transform)

[0195] Figure 16 shows the operation of RST as an embodiment to which the present invention is applied.

[0196] When an orthogonal matrix representing a single transformation has an NxN form, RT (reduced transform) leaves only R out of N transformation basis vectors (R < N). The matrix of the forward RT for generating the transform coefficients can be defined as in Mathematical Formula 3.

[0197]

Equation

[0198] Since the matrix of the reverse RT is the transpose matrix of the forward RT matrix, if the application of the forward RT and the reverse RT is illustrated, it can be the same as FIGS. 16a and 16b.

[0199] The RT applied to the upper left 8x8 block of a block of transform coefficients to which a primary transform has been applied may be referred to as an 8x8 RST. In Equation 3, when the value of R is set to 16, the forward 8x8 RST has a 16x64 matrix form, and the inverse 8x8 RST has a 64x16 form. The configuration of the transform set shown in Table 5 may also be applied to the 8x8 RST. That is, as shown in Table 5, the 8x8 RST may be determined based on a transform set corresponding to the intra prediction mode. Since one transform set consists of two or three transforms depending on the intra prediction mode, any one of up to four transforms may be selected, including cases where no secondary transform is applied (one transform may correspond to an identity matrix). When the four transforms are assigned indices of 0, 1, 2, and 3, respectively, the transform to be applied may be specified by signaling a syntax element corresponding to the NSST index for each block of transform coefficients. For example, the 0th index can be assigned to the identity matrix, i.e., when no quadratic transformation is applied. In conclusion, for an 8x8 top-left block via the NSST index, an 8x8 NSST can be specified according to the JEM NSST, and an 8x8 RST can be specified according to the RST configuration.

[0200] FIG. 17 is a diagram illustrating a process of performing a backward scan from the 64th to the 17th blocks based on the backward scan order, as an embodiment to which the present invention is applied.

[0201] When the forward 8x8 RST as shown in Equation 3 is applied, 16 valid transform coefficients are generated, so 64 input data constituting an 8x8 area is reduced to 16 output data, and from the perspective of a two-dimensional area, only about 1 / 4 of the area is filled with valid transform coefficients. Therefore, by applying the forward 8x8 RST, the 16 output data obtained are filled in the area at the top left of Figure 17.

[0202] In Figure 17, the 4x4 region at the top left is the ROI (region of interest) region filled with valid transform coefficients, and the remaining regions are empty. The empty regions may be filled with a value of 0 by default. If valid non-zero transform coefficients are found outside the ROI region of Figure 17, it is certain that the 8x8 RST is not applied, and therefore the coding of the corresponding NSST index may be omitted. Conversely, if no non-zero transform coefficients are found outside the ROI region of Figure 17 (when the 8x8 RST is applied, and the region outside the ROI is filled with 0), the 8x8 RST may have been applied, and therefore the NSST index may be coded. Such conditional NSST index coding may be performed after the residual coding process, since it is necessary to check for the presence of non-zero transform coefficients.

[0203] FIG. 18 shows an example of an encoding flowchart using a single transform indicator as an embodiment to which the present invention is applied.

[0204] In an embodiment of the present invention, a single transform indicator (STI) is introduced. Instead of sequentially using two transforms (a primary transform and a secondary transform), a single transform can be applied when the single transform indicator is activated (STI coding == 1). Here, the single transform can be any type of transform. For example, the single transform can be a separable transform or a non-separable transform. The single transform can be a transform approximated from a non-separable transform. A single transform index (ST_idx in FIG. 18) can be signaled when the single transform indicator is activated. Here, the single transform index can indicate a transform to be applied from available transform candidates.

[0205] Referring to FIG. 18, the encoding apparatus 100 determines whether CBF is 1 (S1805). If CBF is 1, the encoding apparatus 100 determines whether STI coding is applied (S1810). If STI coding is applied, the encoding apparatus 100 encodes an STI index (STI_Idx) (S1845) and performs coding of transform coefficients (S1850). If STI coding is not applied, the encoding apparatus 100 encodes a flag (EMT_CU_Flag) indicating whether EMT (or MTS) is applied at the CU level (S1815). Thereafter, the encoding apparatus 100 performs coding of transform coefficients (S1820). Thereafter, the encoding apparatus 100 determines whether EMT is applied to a transform unit (TU) (S1825). If EMT is applied to the TU, the encoding apparatus 100 encodes an index (EMT_TU Idx) of the primary transform applied to the TU (S1830). Thereafter, the encoding apparatus 100 determines whether an NSST is applied (S1835). If an NSST is applied, the encoding apparatus 100 encodes an index (NSST_Idx) indicating the NSST to be applied (S1840).

[0206] In one example, when the single transform coding condition is satisfied / activated (e.g., STI_coding == 1), the single transform index (ST_Idx) is not signaled but can be implicitly derived. ST_idx can be implicitly determined based on the block size and intra prediction mode. Here, ST_idx can indicate the transform (or transform kernel) to be applied to the current transform block.

[0207] A single transformation indicator can be activated (STI_coding == 1) if one or more of the following conditions are met:

[0208] 1) The block size corresponds to a predetermined value, such as 4 or 8.

[0209] 2) Block width == Block height (square block)

[0210] 3) The intra prediction mode is one of the predetermined modes such as DC or Planar.

[0211] In another example, the STI coding flag may be signaled to indicate whether a single transform is applied. The STI coding flag may be signaled based on the STI coding value and CBF. For example, the STI coding flag may be signaled when the CBF is 1 and STI coding is activated. Furthermore, the STI coding flag may be conditionally signaled taking into account the block size, block shape (square block or non-square block), or intra prediction mode.

[0212] Since the acquired information is used during coefficient coding, ST_idx can be determined after coefficient coding. In one example, ST_idx can be implicitly determined based on the block size, intra prediction mode, and the number of non-zero coefficients. In another example, ST_idx can be conditionally encoded / decoded based on the block size and / or block shape and / or intra prediction mode, and / or the number of non-zero coefficients. In another example, ST_idx signaling can be omitted depending on the distribution of non-zero coefficients (i.e., the locations of non-zero coefficients). In particular, if a non-zero coefficient is found in an area other than the upper left 4x4 area, ST_idx signaling can be omitted.

[0213] FIG. 19 shows an example of an encoding flowchart using a unified transform indicator (UTI) as an embodiment to which the present invention is applied.

[0214] In the embodiment of the present invention, a unified conversion indicator is introduced, which includes a primary conversion indicator and a secondary conversion indicator.

[0215] Referring to FIG. 19, the encoding apparatus 100 determines whether CBF is 1 (S1905). If CBF is 1, the encoding apparatus 100 determines whether UTI coding is applied (S1910). If UTI coding is applied, the encoding apparatus 100 encodes a UTI index (UTI_Idx) (S1945) and performs coding of transform coefficients (S1950). If UTI coding is not applied, the encoding apparatus 100 encodes a flag (EMT_CU_Flag) indicating whether EMT (or MTS) is applied at the CU level (S1915). Thereafter, the encoding apparatus 100 encodes the transform coefficients (S1920). Thereafter, the encoding apparatus 100 determines whether EMT is applied to a transform unit (TU) (S1925). If EMT is applied to the TU, encoding apparatus 100 encodes an index (EMT_TU Idx) of the primary transform applied to the TU (S1930). Then, encoding apparatus 100 determines whether NSST is applied (S1935). If NSST is applied, encoding apparatus 100 encodes an index (NSST_Idx) indicating the NSST to be applied (S1940).

[0216] The UTI can be encoded for each predetermined unit (CTU or CU).

[0217] The UTI coding mode may depend on the following conditions:

[0218] 1) Block Size

[0219] 2) Block shape

[0220] 3) Intra Prediction Mode

[0221] How to derive / extract the core transform index from the UTI is predefined. How to derive / extract the secondary transform index from the UTI is predefined.

[0222] The syntax structure of the UTI is selectively used. The UTI may depend on the CU (or TU) size. For example, a smaller CU (TU) may have a relatively narrow range of UTI indices. In one example, if a predefined condition is met (e.g., the block size is smaller than a predefined threshold), the UTI may point to only the core transform index.

[0223] [Table 6]

[0224] In another example, if a secondary transform is not indicated to be used (e.g., if the secondary transform index == 0 or the secondary transform has already been determined), the UTI index can be treated as a core transform index. Similarly, if the core transform index is known, the UTI index can be treated as a secondary transform index. In particular, a predetermined core transform can be used, taking into account the intra prediction mode and block size.

[0225] 20a and 20b show another example of an encoding flowchart using UTI as an embodiment to which the present invention is applied.

[0226] In another example, the transform encoding structure uses UTI index coding as shown in Figures 20a and 20b, where the UTI index can be encoded before or after the coefficient encoding.

[0227] Referring to FIG. 20A, the encoding apparatus 100 checks whether the CBF is 1 (S2005). If the CBF is 1, the encoding apparatus 100 codes the UTI index (UTI_Idx) (S2010) and performs coding of the transform coefficients (S2015).

[0228] Referring to FIG. 20b, the encoding apparatus 100 checks whether the CBF is 1 (S2055). If the CBF is 1, the encoding apparatus 100 performs coding of the transform coefficients (S2060) and codes the UTI index (UTI_Idx) (S2065).

[0229] In another embodiment of the present invention, data hiding and implicit coding methods of transform indicators are introduced. Here, the transform indicator includes ST_idx, UTI_idx, EMT_CU_Flag, EMT_TU_Flag, NSST_idx, and a transform-related index used to indicate a transform kernel. The above-mentioned transform indicators are not signaled, and the information can be inserted into the coefficient encoding process (can be extracted during the coefficient coding process). The coefficient encoding process can include the following parts:

[0230] - Last x position (Last_position_x), Last y position (Last_position_y)

[0231] - Group flag

[0232] - Significance map

[0233] - Flag indicating whether it is greater than 1 (Greater_than_1_flag)

[0234] - Flag indicating whether it is greater than 2 (Greater_than_2_flag)

[0235] - Remaining level coding

[0236] - Sign coding

[0237] For example, the transformation indicator information can be inserted into one or more of the coefficient coding processes described above. The following can be considered together for inserting the transformation indicator information:

[0238] - Pattern of Sign coding

[0239] - The absolute value of the remaining level

[0240] The number of flags indicating whether they are greater than -1 (Greater_than_1_flag)

[0241] - The value of Last_position_X and Last_position_Y

[0242] The aforementioned data hiding methods can be considered conditional. For example, the data hiding method can depend on the number of non-zero coefficients.

[0243] In yet another example, NSST_idx and EMT_idx can be dependent. For example, when EMT_CU_flag is 0 (or 1), NSST_idx may not be 0. In this case, NSST_idx-1 can be signaled instead of NSST_idx.

[0244] In another embodiment of the present invention, mapping of NSST transform sets based on intra prediction modes is presented as shown in Table 7 below. As mentioned above, the following description will focus on NSST as an example of a non-separable transform, but other known terms for non-separable transforms (e.g., LFNST) may also be used. For example, NSST set and NSST index are used interchangeably with LFNST set and LFNST index. Furthermore, RST described in this document is used interchangeably with RST or LFNST as an example of a non-separable transform (e.g., LFNST) that uses a non-square transform matrix having a reduced input length and / or a reduced output length with a square non-separable transform matrix applied to at least a portion of a transform block (the remaining region excluding the upper-left 4x4 or 8x8 region or the lower-right 4x4 region of an 8x8 block).

[0245] [Table 7]

[0246] The NSST set number can be rearranged between 0 and 3 as shown in Table 8.

[0247] [Table 8]

[0248] In the NSST transformation set, four transformation sets are used (instead of 35) to reduce the memory space required.

[0249] Furthermore, for each transformation set, a different number of transformation kernels are used as follows:

[0250] Case A: Two available transform kernels are used for each transform set, and the NSST index ranges from 0 to 2. For example, if the NSST index is 0, a secondary transform (a secondary inverse transform based on the decoder) may not be applied. If the NSST index is 1 or 2, a secondary transform may be applied. A transform set may include two transform kernels, and an index of 1 or 2 may be mapped to the two transform kernels.

[0251] [Table 9]

[0252] Referring to Table 9, two transform kernels are used for each of the 0th to 3rd non-separable transform (NSST or LFNST) sets.

[0253] Case B: Use two available transformation kernels for transformation set 0, and one for each of the remaining transformation sets. The available NSST indices for transformation set 0 (DC, Planner) are 0 to 2. However, the NSST indices for other modes (transform sets 1, 2, and 3) are 0 to 1.

[0254] [Table 10]

[0255] Referring to Table 10, two non-separable transform kernels are set for the non-separable transform (NSST) set corresponding to index 0, and one non-separable transform kernel is set for each of the non-separable transform (NSST) sets corresponding to indexes 1, 2, and 3.

[0256] Case C: One transform kernel is used for each transform set, and the NSST index ranges from 0 to 1.

[0257] [Table 11]

[0258] FIG. 21 shows an example of an encoding flowchart for performing conversion as an embodiment to which the present invention is applied.

[0259] The encoding apparatus 100 performs a primary transform on the residual block (S2105). The primary transform may be referred to as a core transform. In an embodiment, the encoding apparatus 100 may perform the primary transform using the MTS. The encoding apparatus 100 may also transmit an MTS index indicating a specific MTS from among MTS candidates to the decoding apparatus 200. In this case, the MTS candidate may be configured based on the intra prediction mode of the current block.

[0260] The encoding apparatus 100 determines whether to apply a secondary transform (S2110). For example, the encoding apparatus 100 may determine whether to apply a secondary transform based on the linearly transformed residual transform coefficients. For example, the secondary transform may be NSST or RST.

[0261] The encoding apparatus 100 determines the secondary transform (S2115). At this time, the encoding apparatus 100 can determine the secondary transform based on the NSST (or RST) transform set specified according to the intra prediction mode.

[0262] Also, for example, prior to operation S2115, the encoding apparatus 100 may determine an area to which the secondary transformation is applied based on the size of the current block.

[0263] The encoding apparatus 100 performs the secondary transformation using the secondary transformation determined in step S2115 (S2120).

[0264] FIG. 22 shows an example of a decoding flowchart for performing conversion as an embodiment to which the present invention is applied.

[0265] The decoding device 200 determines whether to apply a secondary inverse transform (S2205). For example, the secondary inverse transform may be an NSST or an RST. As an example, the decoding device 200 may determine whether to apply a secondary inverse transform based on a secondary transform flag received from the encoding device 100.

[0266] The decoding apparatus 200 determines a secondary inverse transform (S2210). At this time, the decoding apparatus 200 may determine a secondary inverse transform to be applied to the current block based on the NSST (or RST) transform set designated according to the above-described intra prediction mode.

[0267] Also, for example, prior to operation S2210, the decoding apparatus 200 may determine an area to which the secondary inverse transform is applied based on the size of the current block.

[0268] The decoding apparatus 200 performs a secondary inverse transform on the dequantized residual block using the secondary inverse transform determined in step S2210 (S2215).

[0269] The decoding apparatus 200 performs a primary inverse transform on the secondary inverse transformed residual block (S2220). The primary inverse transform may be referred to as a core inverse transform. In one embodiment, the decoding apparatus 200 may perform the primary inverse transform using the MTS described above. Also, as an example, the decoding apparatus 200 may determine whether MTS is applied to the current block prior to step S2220. In this case, the decoding flowchart of FIG. 22 may further include a step of determining whether MTS is applied.

[0270] For example, when MTS is applied to the current block (i.e., cu_mts_flag = 1), the decoding apparatus 200 may configure MTS candidates based on the intra prediction mode of the current block. In this case, the decoding flowchart of Figure 22 may further include a step of configuring MTS candidates. Then, the decoding apparatus 200 may determine a linear inverse transform to be applied to the current block using mts_idx indicating a specific MTS among the configured MTS candidates.

[0271] FIG. 23 shows an example of a detailed block diagram of the conversion unit 120 in the encoding device 100 as an embodiment to which the present invention is applied.

[0272] The encoding device 100 to which the embodiment of the present invention is applied may include a primary transform unit 2310, a secondary transform application determining unit 2320, a secondary transform determining unit 2330, and a secondary transform unit 2340.

[0273] The primary transform unit 2310 may perform a primary transform on the residual block. The primary transform may be referred to as a core transform. In an embodiment, the primary transform unit 2310 may perform the primary transform using the MTS. The primary transform unit 2310 may also transmit an MTS index indicating a specific MTS from among MTS candidates to the decoding apparatus 200. In this case, the MTS candidates may be configured based on the intra prediction mode of the current block.

[0274] The secondary transform application determining unit 2320 may determine whether to apply a secondary transform. For example, the secondary transform application determining unit 2320 may determine whether to apply a secondary transform based on transform coefficients of the primary transformed residual block. For example, the secondary transform may be NSST or RST.

[0275] The secondary transform determination unit 2330 determines a secondary transform. In this case, as described above, the secondary transform determination unit 2330 may determine the secondary transform based on the NSST (or RST) transform set specified according to the intra prediction mode.

[0276] Also, for example, the secondary transformation determination unit 2330 may determine the area to which the secondary transformation is applied based on the size of the current block.

[0277] The secondary transform unit 2340 can perform the secondary transform using the determined secondary transform.

[0278] FIG. 24 shows an example of a detailed block diagram of the inverse transform unit 230 in the decoding device 200 as an embodiment to which the present invention is applied.

[0279] The decoding device 200 to which the present invention is applied includes a secondary inverse transform application determining unit 2410, a secondary inverse transform determining unit 2420, a secondary inverse transform unit 2430, and a primary inverse transform unit 2440.

[0280] The secondary inverse transform application determining unit 2410 may determine whether to apply a secondary inverse transform. For example, the secondary inverse transform may be NSST or RST. As one example, the secondary inverse transform application determining unit 2410 may determine whether to apply a secondary inverse transform based on a secondary transform flag received from the encoding apparatus 100. As another example, the secondary inverse transform application determining unit 2410 may determine whether to apply a secondary inverse transform based on transform coefficients of a residual block.

[0281] The secondary inverse transform determination unit 2420 may determine a secondary inverse transform to be applied to the current block based on an NSST (or RST) transform set designated according to the intra prediction mode.

[0282] Also, for example, the secondary inverse transform determination unit 2420 may determine an area to which the secondary inverse transform is applied based on the size of the current block.

[0283] Also, as an example, the secondary inverse transform unit 2430 may perform a secondary inverse transform on the dequantized residual block using the determined secondary inverse transform.

[0284] The primary inverse transform unit 2440 may perform a primary inverse transform on the secondary inverse transformed residual block. As an embodiment, the primary inverse transform unit 2440 may perform a primary transform using the MTS described above. As another example, the primary inverse transform unit 2440 may determine whether MTS is applied to the current block.

[0285] For example, if an MTS is applied to the current block (i.e., cu_mts_flag = 1), the primary inverse transform unit 2440 may configure MTS candidates based on the intra prediction mode of the current block. Then, the primary inverse transform unit 2440 may determine a primary transform to be applied to the current block using mts_idx, which indicates a specific MTS from among the configured MTS candidates.

[0286] 25 shows a flowchart for processing a video signal as an embodiment to which the present invention is applied. The flowchart in FIG. 25 can be executed by the decoding device 200 or the inverse transform unit 230.

[0287] First, the decoding device 200 may determine whether to apply an inverse non-separable transform of the current block based on the non-separable transform index and the width and height of the current block. For example, the decoding device 200 may determine to apply the non-separable transform if the non-separable transform index is not 0 and the width and height of the current block are each equal to or greater than 4. If the non-separable transform index is 0 or the width or height of the current block is less than 4, the decoding device 200 may omit the inverse non-separable transform and perform an inverse linear transform.

[0288] In operation S2505, the decoding apparatus 200 determines a non-separable transform set index indicating a non-separable transform set to be used for non-separable transform of the current block from among predefined non-separable transform sets based on the intra prediction mode of the current block. The non-separable transform set index may be set to be assigned to each of four transform sets set according to the range of the intra prediction mode, as shown in Table 7 or Table 8. That is, as shown in Table 7 or Table 8, if the intra prediction mode is 0 to 1, the non-separable transform set index may be determined to be a first index value; if the intra prediction mode is 2 to 12 or 56 to 66, the non-separable transform set index may be determined to be a second index value; if the intra prediction mode is 13 to 23 or 45 to 55, the non-separable transform set index may be determined to be a third index value; and if the intra prediction mode is 24 to 44, the non-separable transform set index may be determined to be a fourth index value.

[0289] Here, each of the predefined non-separable transform sets can include two transform kernels, as shown in Table 9. Alternatively, each of the predefined non-separable transform sets can include one or two transform kernels, as shown in Table 10 or Table 11.

[0290] In operation S2510, the decoding apparatus 200 determines a transform kernel indicated by the non-separable transform index of the current block from among the transform kernels included in the non-separable transform set indicated by the non-separable transform set index, using a non-separable transform matrix. For example, two non-separable transform kernels may be set for each index value of the non-separable transform set index, and the decoding apparatus 200 may determine a non-separable transform matrix from among the two transform matrix kernels corresponding to the non-separable transform set index, based on the transform kernel indicated by the non-separable transform index.

[0291] In operation S2515, the decoding apparatus 200 applies a non-separable transform matrix to an upper left region of the current block, which is determined according to the width and height of the current block. For example, if both the width and height of the current block are equal to or greater than 8, a non-separable transform is applied to an 8x8 region at the upper left of the current block. If the width or height of the current block is less than 8, a non-separable transform may be applied to a 4x4 region of the current block. The size of the non-separable transform may also be set to 8x8 or 4x4 depending on the region to which the non-separable transform is applied.

[0292] In addition, the decoding apparatus 200 may apply horizontal and vertical transforms to the current block to which the non-separable transform has been applied, where the horizontal and vertical transforms may be determined based on a prediction mode applied to the current block and an MTS index for selecting a transform matrix.

[0293] Hereinafter, a method for combining and applying a primary transform and a secondary transform will be described. That is, in an embodiment of the present invention, a method for efficiently designing transforms used in the primary transform and the secondary transform is proposed. Here, the methods proposed in FIGS. 1 to 25 may be applied, and related redundant descriptions will be omitted.

[0294] As described above, the primary transform refers to a transform that is first applied to a residual block based on the encoder. When a secondary transform is applied, the encoder performs a secondary transform on the primary transformed residual block. On the other hand, when a secondary transform is applied, a secondary inverse transform is performed on the decoder prior to the primary inverse transform. The decoder can derive the residual block by performing a primary inverse transform on the secondary inverse transformed transform coefficient block.

[0295] As described above, a non-separable transform may be used as the secondary transform, and may be applied only to low-frequency coefficients in a specific region on the upper left side to maintain low complexity. Such a secondary transform applied to low-frequency coefficients may be referred to as a non-separable secondary transform (NSST), low-frequency non-separable transform (LFNST), or reduced secondary transform (RST). The primary transform may also be referred to as a core transform.

[0296] In one embodiment of the present invention, the primary transform candidates used in the primary transform and the secondary transform kernels used in the secondary transform may be predefined in various combinations. In this specification, the primary transform candidates used in the primary transform may be referred to as MTS candidates, but are not limited to this name. As an example, the primary transform candidates may be a combination of transform kernels (or transform types) applied to the horizontal and vertical directions, respectively, and the transform kernels may be any one of DCT2, DST7, and / or DCT8. In other words, the primary transform candidates may be a combination of at least one of DCT2, DST7, and / or DCT8. A specific example will be described below.

[0297] -Combination A

[0298] In combination A, the primary transform candidates and secondary transform kernels are defined according to the intra prediction mode, as shown in Table 12 below.

[0299] [Table 12]

[0300] Referring to Table 12, as an example (Case 1), if the intra prediction mode has directionality, two primary transform candidates are used, and if it does not have directionality (e.g., DC or planar mode), four primary transform candidates are used. Here, the secondary transform candidates may include two transform kernels regardless of the directionality of the intra prediction mode. That is, as described above, multiple secondary transform kernel sets are predefined according to the intra prediction mode, and each of the predefined multiple secondary transform kernel sets includes two transform kernels.

[0301] In addition, as an example (Case 2), if the intra prediction mode has directionality, two primary transform candidates are used, and if it does not have directionality, four primary transform candidates are used. Here, the secondary transform candidates include one transform kernel if the intra prediction mode has directionality, and two transform kernels if it does not have directionality.

[0302] In addition, as an example (Case 3), if the intra prediction mode has directionality, two primary transform candidates are used, and if it does not have directionality, four primary transform candidates are used. Here, the secondary transform candidates may include one transform kernel regardless of the directionality of the intra prediction mode.

[0303] -Combination B

[0304] In combination B, the primary transform candidates and secondary transform kernels are defined according to the intra prediction mode, as shown in Table 13 below.

[0305] [Table 13]

[0306] Referring to Table 13, as an example (Case 1), three primary transform candidates are used regardless of the direction of the intra prediction mode. Here, the secondary transform candidates may include two transform kernels regardless of the direction of the intra prediction mode. That is, as described above, multiple secondary transform kernel sets are predefined according to the intra prediction mode, and each of the multiple predefined secondary transform kernel sets includes two transform kernels.

[0307] In addition, as an example (Case 2), three primary transform candidates are used regardless of the directionality of the intra prediction mode. Here, the secondary transform candidates include one transform kernel when the intra prediction mode has directionality, and two transform kernels when the intra prediction mode does not have directionality.

[0308] In addition, as an example (Case 3), three primary transform candidates are used regardless of the directionality of the intra prediction mode, and the secondary transform candidates may include one transform kernel regardless of the directionality of the intra prediction mode.

[0309] -Combination C

[0310] In combination C, the primary transform candidates and secondary transform kernels are defined according to the intra prediction mode, as shown in Table 14 below.

[0311] [Table 14]

[0312] Referring to Table 14, as an example (Case 1), if the intra prediction mode has directionality, two primary transform candidates are used, and if it does not have directionality (e.g., DC or planar mode), three primary transform candidates are used. Here, the secondary transform candidates may include two transform kernels regardless of the directionality of the intra prediction mode. That is, as described above, multiple secondary transform kernel sets are predefined according to the intra prediction mode, and each of the multiple predefined secondary transform kernel sets may include two transform kernels.

[0313] In addition, as an example (Case 2), if the intra prediction mode has directionality, two primary transform candidates are used, and if it does not have directionality, three primary transform candidates are used. Here, the secondary transform candidates include one transform kernel if the intra prediction mode has directionality, and two transform kernels if it does not have directionality.

[0314] In addition, as an example (Case 3), if the intra prediction mode has directionality, two primary transform candidates are used, and if it does not have directionality, three primary transform candidates are used. Here, the secondary transform candidates may include one transform kernel regardless of the directionality of the intra prediction mode.

[0315] The above explanation has focused on the case where multiple primary conversion candidates are used. Below, an example of a combination of primary conversion and secondary conversion when fixed primary conversion candidates are used will be explained.

[0316] -Combination D

[0317] In combination D, the primary transform candidates and secondary transform kernels are defined according to the intra prediction mode, as shown in Table 15 below.

[0318] [Table 15]

[0319] Referring to Table 15, in an embodiment, one primary transform candidate is fixedly used regardless of the intra prediction mode. For example, the fixed primary transform candidate may be a combination of at least one of DCT2, DCT7, and / or DCT8.

[0320] As an example (Case 1), one primary transform candidate is fixedly used regardless of the intra prediction mode, and the secondary transform candidate may include two transform kernels regardless of the direction of the intra prediction mode. That is, as described above, multiple secondary transform kernel sets are predefined according to the intra prediction mode, and each of the multiple predefined secondary transform kernel sets may include two transform kernels.

[0321] As another example (Case 2), one primary transform candidate is fixedly used regardless of the intra prediction mode, where the secondary transform candidate may include one transform kernel if the intra prediction mode has directionality, and may include two transform kernels if the intra prediction mode does not have directionality.

[0322] Also, as an example (Case 3), one primary transform candidate is fixedly used regardless of the intra prediction mode, where the secondary transform candidate may include one transform kernel regardless of the directionality of the intra prediction mode.

[0323] -Combination E

[0324] In combination E, the primary transform candidates and secondary transform kernels are defined according to the intra prediction mode, as shown in Table 16 below.

[0325] [Table 16]

[0326] Referring to Table 16, the secondary transform is defined only when DCT2 is applied as the primary transform. In other words, when MTS is not applied (i.e., when DCT2 is applied as the primary transform), the secondary transform can be applied. As described with reference to FIG. 10, this specification distinguishes between cases where MTS is applied and cases where it is not applied, but the present invention is not limited to such expression. For example, whether or not MTS is applied may mean whether or not a transform type (or transform kernel) other than a predefined specific transform type (which may also be referred to as a base transform type, default transform type, etc.) is used. If MTS is applied, a transform type other than the base transform type (e.g., any one or a combination of two or more of multiple transform types) may be used for the transform, and if MTS is not applied, the base transform type may be used for the transform. In one embodiment, the base transform type may be set (or defined) to DCT2.

[0327] For example, when DCT2 is applied to the primary transform, a secondary transform can be applied, where the secondary transform candidates include two transform kernels regardless of the direction of the intra prediction mode. That is, as described above, multiple secondary transform kernel sets are predefined according to the intra prediction mode, and each of the multiple predefined secondary transform kernel sets includes two transform kernels.

[0328] Also, as an example (Case 2), when DCT2 is applied to the primary transform, a secondary transform can be applied, where the secondary transform candidates may include one transform kernel if the intra prediction mode has directionality, and two transform kernels if it does not have directionality.

[0329] Also, as an example (Case 3), when DCT2 is applied to the primary transform, a secondary transform can be applied, where the secondary transform candidates can include one transform kernel regardless of the directionality of the intra prediction mode.

[0330] FIG. 26 is a flowchart illustrating a method for converting a video signal according to an embodiment of the present invention.

[0331] 26, for convenience of explanation, the decoder will be mainly described, but the present invention is not limited thereto, and the conversion method for a video signal according to this embodiment can be substantially equally applied to an encoder. The flowchart of FIG. 26 is performed by the decoding device 200 or the inverse conversion unit 230.

[0332] The decoding apparatus 200 parses a first syntax element indicating a primary transform kernel to be applied to a primary transform of a current block (S2601).

[0333] The decoding apparatus 200 determines whether a secondary transform can be applied to the current block based on the first syntax element (S2602).

[0334] If a secondary transform can be applied to the current block, the decoding apparatus 200 parses a second syntax element indicating a secondary transform kernel to be applied to the secondary transform of the current block (S2603).

[0335] The decoding apparatus 200 performs a secondary inverse transform on a specific region at the upper left side of the current block using the secondary transform kernel indicated by the second syntax element, thereby deriving a secondary inverse transformed block (S2604).

[0336] The decoding apparatus 200 derives a residual block of the current block by performing a primary inverse transform on the secondary inverse transformed block using the primary transform kernel indicated by the first syntax element (S2605).

[0337] As described above, step S2602 is performed by determining that a secondary transform can be applied to the current block if the first syntax element indicates a predefined first transform kernel, where the first transform kernel is defined as DCT2.

[0338] As described above, the decoding apparatus 200 may determine a secondary transform kernel set to be used for secondary transform of the current block from among predefined secondary transform kernel sets based on the intra prediction mode of the current block, and the second syntax element may indicate a secondary transform kernel to be applied to secondary transform of the current block from the determined secondary transform kernel set.

[0339] Also, as mentioned above, each of the predefined secondary transformation kernel sets may include two transformation kernels.

[0340] In one embodiment of the present invention, an example of a syntax structure in which an MTS (Multiple Transform Set) is used will be described.

[0341] As an example, Table 17 below shows an example of the syntax structure of a sequence parameter set.

[0342] [Table 17]

[0343] Referring to Table 17, whether MTS is enabled or disabled according to an embodiment of the present invention can be signaled via a sequence parameter set syntax. Here, sps_mts_intra_enabled_flag indicates whether an MTS flag or an MTS index is present in a lower level syntax (e.g., residual coding syntax, transform unit syntax) for an intra-coding unit. And, sps_mts_inter_enabled_flag indicates whether an MTS flag or an MTS index is present in a lower level syntax for an inter-coding unit.

[0344] As another example, Table 18 below shows an example of a transform unit syntax structure.

[0345] [Table 18]

[0346] Referring to Table 18, cu_mts_flag indicates whether MTS is applied to the residual samples of the luma transform block. For example, if cu_mts_flag=0, MTS is not applied to the residual samples of the luma transform block. If cu_mts_flag=1, MTS is applied to the residual samples of the luma transform block.

[0347] As described above, in the present invention, a case where MTS is applied and a case where it is not applied are described based on the MTS flag, but the present invention is not limited to this expression. For example, whether MTS is applied may mean whether a transform type (or transform kernel) other than a predefined specific transform type (which may be referred to as a base transform type, a default transform type, etc.) is used. If MTS is applied, a transform type other than the base transform type (e.g., any one of multiple transform types or a combination of two or more transform types) may be used for the transform, and if MTS is not applied, the base transform type may be used for the transform. In one embodiment, the base transform type may be set (or defined) to DCT2.

[0348] As one example, an MTS flag syntax indicating whether MTS is applied to the current transform block and an MTS index syntax indicating the transform type applied to the current block if MTS is applied may be transmitted separately from the encoder to the decoder, and as another example, a syntax (e.g., an MTS index) including whether MTS is applied to the current transform block and, if MTS is applied, all of the transform types applied to the current block may be transmitted from the encoder to the decoder. That is, in the latter embodiment, a syntax (or syntax element) indicating the transform type applied to the current transform block (or unit) within the entire transform type group (or transform type set) including the above-mentioned basic transform types may be transmitted from the encoder to the decoder.

[0349] Therefore, despite this expression, the syntax (MTS index) indicating the transform type applied to the current transform block may include information on whether MTS is applicable. In other words, in the latter embodiment, only the MTS index may be signaled without the MTS flag, and in this case, it may be interpreted that the MTS includes DCT2. However, in the present invention, the case where DCT2 is applied may be described as the case where MTS is not applied, and the technical scope of MTS is not limited to this definition.

[0350] As another example, Table 19 below shows an example of a residual unit syntax structure.

[0351] [Table 19-1]

[0352] [Table 19-2]

[0353] [Table 19-3]

[0354] [Table 19-4]

[0355] [Table 19-5]

[0356] Referring to Table 19, the transform_skip_flag and / or mts_idx syntax (or syntax element) can be signaled via the residual syntax. However, this is just an example and the present invention is not limited thereto. For example, the transform_skip_flag and / or mts_idx syntax can also be signaled via the transform unit syntax.

[0357] The following describes specific embodiments of a secondary transform matrix that can be used for the above-mentioned secondary transform. As described above, the secondary transform can be referred to as a non-separable secondary transform (NSST), a low frequency non-separable transform (LFNST), or a reduced secondary transform (RST).

[0358] As described above, in an embodiment of the present invention, four transform sets (or secondary transform sets) may be used to apply the secondary transforms for memory efficiency. In one embodiment, the encoder / decoder may assign indices 0, 1, 2, and 3 to the four transform sets, respectively.

[0359] Also, as described above, each transform set can include a predefined number of transform kernels. In one embodiment, four transform sets used for quadratic transforms can be predefined in the encoder and decoder, and each transform set can include one or two transform matrices (or transform types, transform kernels).

[0360] Table 20 below shows an example of a transformation that can be applied to an 8x8 region.

[0361] [Table 20-1]

[0362] [Table 20-2]

[0363] [Table 20-3]

[0364] [Table 20-4]

[0365] [Table 20-5]

[0366] [Table 20-6]

[0367] [Table 20-7]

[0368] Table 20 illustrates an example in which the coefficients of the transformation matrix are multiplied by a scaling value of 128. In Table 20, the first input [4] in the g_aiNsst8x8[4][2]

[16]

[64] array represents the number of transformation sets (each transformation set can be divided into indexes 0, 1, 2, and 3), the second input [2] represents the number of transformation matrices that make up each transformation set, and the third and fourth inputs

[16]

[64] represent the rows and columns of the 16x64 Reduced Secondary Transform (RST).

[0369] Table 20 assumes that a transform set includes two transform matrices, but if it includes one transform matrix, it can be set to use a specific order of transform matrices for each transform set in Table 20. For example, if a transform set includes one transform matrix, the encoder / decoder can use the transform matrix predefined in each transform set in Table 20, i.e., the first or second transform matrix.

[0370] When applying the RST of Table 20, the encoder / decoder can be configured (or defined or set) to output 16 transform coefficients, or can be configured to output only m transform coefficients by applying only an m×64 portion of a 16×64 matrix. For example, the encoder / decoder can be configured to output only eight transform coefficients using only an 8×64 matrix from the top by setting m=8. In this way, applying a reduced quadratic transform can reduce the amount of calculation by half. In one embodiment, the encoder / decoder can apply an 8×64 matrix to an 8×8 transform unit (TU) to reduce the amount of calculation in the worst case.

[0371] Table 21 below shows an example of a transformation that can be applied to a 4x4 region.

[0372] [Table 21-1]

[0373] [Table 21-2]

[0374] [Table 21-3]

[0375] Table 21 illustrates an example where the coefficients of the transformation matrix are multiplied by a scaling value of 128. In Table 21, in the g_aiNsst4x4[4][2]

[16]

[16] array, the "4" in the first input represents the number of transformation sets (each transformation set can be divided into indexes 0, 1, 2, and 3), the "2" in the second input represents the number of transformation matrices that make up each transformation set, and the "16][16" in the third and fourth inputs represent the rows and columns of the 16x16 Reduced Secondary Transform (RST).

[0376] Table 21 assumes that a transform set includes two transform matrices, but if it includes one transform matrix, it can be set to use a specific order of transform matrices for each transform set in Table 21. For example, if a transform set includes one transform matrix, the encoder / decoder can use the transform matrix predefined in each transform set in Table 21, i.e., the first or second transform matrix.

[0377] When applying the RST of Table 21, the encoder / decoder can be configured (or defined or set) to output 16 transform coefficients, or can be configured to output only m transform coefficients by applying only an m×16 portion of a 16×16 matrix. For example, the encoder / decoder can be configured to output only eight transform coefficients using only an 8×16 matrix from the top by setting m=8. In this way, applying a reduced quadratic transform can reduce the amount of calculation by half. In one embodiment, the encoder / decoder can apply an 8×64 matrix to an 8×8 transform unit (TU) to reduce the worst-case calculation amount.

[0378] In one embodiment, the secondary transform, the transformation matrix exemplified in Tables 20 and 21, can be applied to the upper left 4x4, 4x8, or 8x4 region (i.e., TU) according to a predefined condition, or can be applied only to the upper left 4x4 region. In the case of a 4x8 TU and an 8x4 TU, the encoder / decoder can divide the TU into two 4x4 regions and apply a specified transformation to each divided region. If the secondary transform is defined to be applied only to the 4x4 region, only the transformation defined in Table 21 can be applied (or used).

[0379] Meanwhile, in Tables 20 and 21, the coefficients of the transformation matrices are defined assuming that the scaling value is 128, but the present invention is not limited to this. For example, Tables 20 and 21 can be defined as shown in Tables 22 and 23 below, respectively, by setting the scaling value to 256.

[0380] [Table 22-1]

[0381] [Table 22-2]

[0382] [Table 22-3]

[0383] [Table 22-4]

[0384] [Table 22-5]

[0385] [Table 22-6]

[0386] [Table 22-7]

[0387] [Table 22-8]

[0388] [Table 23-1]

[0389] [Table 23-2]

[0390] [Table 23-3]

[0391] As described above, in an embodiment of the present invention, four transform sets (or secondary transform sets) may be used to apply the secondary transforms for memory efficiency. In one embodiment, the encoder / decoder may assign indices 0, 1, 2, and 3 to the four transform sets, respectively.

[0392] Also, as described above, each transform set can include a predefined number of transform kernels. In one embodiment, four transform sets used for the secondary transform can be predefined in the encoder and decoder, and each transform set can include one or two transform matrices (or transform types, transform kernels).

[0393] Below, various examples of various quadratic transformation sets and transformation matrices (or transformation types, transformation kernels) applicable to quadratic transformations will be shown. Although various transformation matrices different from those in Tables 20 to 23 may be defined in detail, for the sake of convenience, in this embodiment, examples will be given focusing on non-directional modes (e.g., DC mode, planar mode) along with a method for constructing a generalized quadratic transformation set.

[0394] First, specific examples of quadratic transformations that can be applied to a 4x4 region will be described. Among the following examples of quadratic transformation sets available for quadratic transformation, the first and fourth examples can be applied to an embodiment in which each transformation set is composed of two transformation matrices. The second and third examples can be applied to an embodiment in which each transformation set is composed of one transformation matrix.

[0395] In particular, the first example can be applied to Case 1 of the embodiment described in the above-mentioned combination D and Table 15, and can also be applied to Case 1 of the embodiment described in the above-mentioned combination A and Table 12, Case 1 of the embodiment described in the above-mentioned combination B and Table 13, Case 1 of the embodiment described in the above-mentioned combination C and Table 14, or Case 1 of the embodiment described in the above-mentioned combination E and Table 16.

[0396] In particular, the second exemplary transformation array (i.e., transformation set) can be applied to Case 3 of the embodiment described in combination D and Table 15 above, and further to Case 3 of the embodiment described in combination A and Table 12, Case 3 of the embodiment described in combination B and Table 13, Case 3 of the embodiment described in combination C and Table 14, or Case 3 of the embodiment described in combination E and Table 16.

[0397] Although the above-described combinations A, B, C, D, and E only deal with cases where the number of MTS candidates is three or less, it is also possible to configure the primary transform to apply all four MTS candidates to all intra prediction modes. The following first to fourth examples can also be used when all four MTS candidates are applied, and in particular, the transform arrangement of the fourth example can be more suitable when four MTS candidates are applied.

[0398] The fifth to seventh exemplary transform arrays below correspond to the case where 35 transform sets are applied. They can be applied when mapping the transform sets as shown in Table 24 below for each intra prediction mode.

[0399] [Table 24]

[0400] In Table 24, NSST set index represents a transformation set index. When the mapping method of Table 24 is applied, it can also be applied to the above-mentioned combinations A to E. That is, it can also be applied to the fifth to eighth examples in the same way as the above-mentioned method for each combination.

[0401] The fifth and eighth example transformation arrays can be applied to embodiments in which each transformation set consists of two transformation matrices, and the sixth and seventh example transformation arrays can be applied to embodiments in which each transformation set consists of one transformation matrix.

[0402] In particular, the fifth example can be applied to Case 1 of the embodiment described in the above-mentioned combination D and Table 15, and also to Case 1 of the embodiment described in the above-mentioned combination A and Table 12, to Case 1 of the embodiment described in the above-mentioned combination B and Table 13, to Case 1 of the embodiment described in the above-mentioned combination C and Table 14, or to Case 1 of the embodiment described in the above-mentioned combination E and Table 16.

[0403] In particular, the sixth and seventh exemplary transformation arrays (i.e., transformation sets) can be applied to Case 3 of the embodiment described in combination D and Table 15 above, and can also be applied to Case 3 of the embodiment described in combination A and Table 12, Case 3 of the embodiment described in combination B and Table 13, Case 3 of the embodiment described in combination C and Table 14, or Case 3 of the embodiment described in combination E and Table 16.

[0404] Although the above-described combinations A, B, C, D, and E only deal with cases where the number of MTS candidates is three or less, it is also possible to configure the primary transform to apply all four MTS candidates to all intra prediction modes. The following fifth to eighth examples can also be used when all four MTS candidates are applied, and in particular, the transform arrangement of the eighth example can be more suitable when four MTS candidates are applied.

[0405] Among the following first through eighth example transform arrays, all of the transform examples that can be applied to a 4x4 region correspond to transform matrices multiplied by a scaling value of 128. The following example transform arrays can be commonly expressed as a g_aiNsst4x4[N1][N2]

[16]

[16] array, where N1 represents the number of transform sets. Here, N1 can be 4 or 35 and can be divided into indexes 0, 1, ..., N1-1. N2 represents the number of transform matrices that make up each transform set (i.e., 1 or 2), and

[16]

[16] represents a 16x16 transform matrix.

[0406] Similarly, in the following example, when a transform set is composed of one transform, it can be set to use a specific order of transform matrices for each transform set. For example, when a transform set includes one transform matrix, the encoder / decoder can use the predefined transform matrix, i.e., the first or second transform matrix, in each transform set.

[0407] To reduce the amount of worst-case calculations, the encoder / decoder can apply an 8x16 matrix to a 4x4 TU. The following exemplary transformations that can be applied to 4x4 regions can be applied to 4x4 TUs, 4xM TUs, and Mx4 TUs (M>4). When applied to 4xM TUs and Mx4 TUs, the specified transformations can be applied to each 4x4 region separately, or only to the top-left 4x8 or 8x4 region. Alternatively, they can be applied only to the top-left 4x4 region.

[0408] In one embodiment, to reduce the worst case computational complexity, the following method may be applied.

[0409] For example, for a block having a width W and a height H, if W>=8 and H>=8, the encoder / decoder may apply a transform array (or transform matrix, transform kernel) that can be applied to an 8x8 region (e.g., a 16x64 matrix) to the upper left 8x8 region. If W=8 and H=8, the encoder / decoder may apply only the 8x64 portion of the 16x64 matrix. In this case, the input of the secondary transform may be generated with 8 transform coefficients, and the remaining coefficients in the region may be considered to be 0.

[0410] For example, for a block whose width is W and whose height is H, if one of W and H is less than 8 (i.e., 4), the encoder / decoder can apply a transform array that can be applied to a 4x4 region. If W=4 and H=4, the encoder / decoder can apply only an 8x16 portion of a 16x16 matrix. In this case, the input of the quadratic transform can be generated with 8 transform coefficients, and the remaining coefficients in the region can be considered to be 0.

[0411] In one embodiment, if (W, H)=(4, 8) or (8, 4), the encoder / decoder can apply the quadratic transform only to the top-left 4x4 region. If W or H is greater than 8, the encoder / decoder can apply the quadratic transform only to the top-left two 4x4 blocks. That is, the encoder / decoder can apply the specified transform matrix to only the top-left 4x8 or 8x4 region, divided into two 4x4 blocks.

[0412] First example

[0413] A first example can be defined as shown in Table 25 below. Four transformation sets can be defined, and each transformation set can be composed of two transformation matrices.

[0414] [Table 25]

[0415] Second Example

[0416] A second example can be defined as shown in Table 26 below. Four transformation sets can be defined, and each transformation set can consist of one transformation matrix.

[0417] [Table 26]

[0418] Third Example

[0419] A third example can be defined as shown in Table 27 below. Four transformation sets can be defined, and each transformation set can consist of one transformation matrix.

[0420] [Table 27]

[0421] Fourth Example

[0422] A fourth example can be defined as shown in Table 28 below. Four transformation sets can be defined, and each transformation set can be composed of two transformation matrices.

[0423] [Table 28]

[0424] Fifth Example

[0425] A fifth example can be defined as shown in Table 29 below. 35 transformation sets can be defined, and each transformation set can be composed of two transformation matrices.

[0426] [Table 29]

[0427] Sixth Example

[0428] A sixth example can be defined as shown in Table 30 below. 35 transformation sets can be defined, and each transformation set can consist of one transformation matrix.

[0429] [Table 30]

[0430] Seventh Example

[0431] The seventh example can be defined as shown in Table 31 below. 35 transformation sets can be defined, and each transformation set can consist of one transformation matrix.

[0432] [Table 31]

[0433] Eighth Example

[0434] The eighth example can be defined as shown in Table 32 below. 35 transformation sets can be defined, and each transformation set can be composed of two transformation matrices.

[0435] [Table 32]

[0436] Examples of quadratic transformations that can be applied to an 8x8 region are described below. Among the following examples of quadratic transformation sets available for quadratic transformation, examples 9 and 12 can be applied to embodiments in which each transformation set is composed of two transformation matrices. Examples 10 and 11 can be applied to embodiments in which each transformation set is composed of one transformation matrix.

[0437] In particular, the ninth example can be applied to Case 1 of the embodiment described in the above-mentioned combination D and Table 15, and also to Case 1 of the embodiment described in the above-mentioned combination A and Table 12, to Case 1 of the embodiment described in the above-mentioned combination B and Table 13, to Case 1 of the embodiment described in the above-mentioned combination C and Table 14, or to Case 1 of the embodiment described in the above-mentioned combination E and Table 16.

[0438] In particular, the tenth exemplary transformation array (i.e., transformation set) can be applied to Case 3 of the embodiment described in the above-mentioned combination D and Table 15, and further to Case 3 of the embodiment described in combination A and Table 12, Case 3 of the embodiment described in combination B and Table 13, Case 3 of the embodiment described in combination C and Table 14, or Case 3 of the embodiment described in combination E and Table 16.

[0439] Although the above-described combinations A, B, C, D, and E only deal with cases where the number of MTS candidates is three or less, it is also possible to configure the primary transform to apply all four MTS candidates to all intra prediction modes. The following ninth to twelfth examples can also be used when all four MTS candidates are applied, and in particular, the transform arrangement of the twelfth example can be more suitable when four MTS candidates are applied.

[0440] The following thirteenth to sixteenth exemplary transform arrays correspond to the case where 35 transform sets are applied, and can be applied when mapping the transform sets as shown in Table 24 for each intra prediction mode.

[0441] In Table 24, NSST set index represents a transformation set index. When the mapping method of Table 24 is applied, it can also be applied to the above-mentioned combinations A to E. That is, it can also be applied to the 13th to 16th examples in the same way as the above-mentioned method for each combination.

[0442] The thirteenth and sixteenth example transformation arrays may be applied to embodiments in which each transformation set consists of two transformation matrices, and the fourteenth and fifteenth example transformation arrays may be applied to embodiments in which each transformation set consists of one transformation matrix.

[0443] In particular, the thirteenth example can be applied to Case 1 of the embodiment described in the above-mentioned combination D and Table 15, and also to Case 1 of the embodiment described in the above-mentioned combination A and Table 12, to Case 1 of the embodiment described in the above-mentioned combination B and Table 13, to Case 1 of the embodiment described in the above-mentioned combination C and Table 14, or to Case 1 of the embodiment described in the above-mentioned combination E and Table 16.

[0444] In particular, the 14th and 15th exemplary transformation arrays (i.e., transformation sets) can be applied to Case 3 of the embodiment described in the above-mentioned combination D and Table 15, and can also be applied to Case 3 of the embodiment described in combination A and Table 12, Case 3 of the embodiment described in combination B and Table 13, Case 3 of the embodiment described in combination C and Table 14, or Case 3 of the embodiment described in combination E and Table 16.

[0445] Although the above-described combinations A, B, C, D, and E only deal with cases where the number of MTS candidates is three or less, it is also possible to configure the primary transform to apply all four MTS candidates to all intra prediction modes. The following thirteenth to sixteenth examples can also be used when all four MTS candidates are applied, and in particular, the eighth example transform arrangement can be further adapted to the case where four MTS candidates are applied.

[0446] Among the following eighth to sixteenth example transform arrays, all of the transform examples that can be applied to an 8x8 region correspond to transform matrices multiplied by a scaling value of 128. The following example transform arrays can be commonly represented as a g_aiNsst8x8[N1][N2]

[16]

[64] array, where N1 represents the number of transform sets. Here, N1 can be 4 or 35 and can be divided into indexes 0, 1, ..., N1-1. N2 represents the number of transform matrices that make up each transform set (i.e., 1 or 2), and

[16]

[64] represents a 16x64 Reduced Secondary Transform (RST).

[0447] Similarly, in the following example, when a transform set is composed of one transform, it can be set to use a specific order of transform matrices for each transform set. For example, when a transform set includes one transform matrix, the encoder / decoder can use the predefined transform matrix, i.e., the first or second transform matrix, in each transform set.

[0448] When the RST is applied, 16 transform coefficients are output, but if only the m×64 portion of the 16×64 matrix is ​​applied, only m transform coefficients can be output. For example, by setting m=8 and multiplying only the top 8×64 matrix to output only 8 transform coefficients, the amount of calculation can be reduced by half.

[0449] Ninth Example

[0450] The ninth example can be defined as shown in Table 33 below. Four transformation sets can be defined, and in this case, each transformation set can be composed of two transformation matrices.

[0451] [Table 33-1]

[0452] [Table 33-2]

[0453] [Table 33-3]

[0454] Tenth Example

[0455] The tenth example can be defined as shown in Table 34 below. Four transformation sets can be defined, and each transformation set can consist of one transformation matrix.

[0456] [Table 34]

[0457] Eleventh Example

[0458] An eleventh example can be defined as shown in Table 35 below. Four transformation sets can be defined, and each transformation set can consist of one transformation matrix.

[0459] [Table 35]

[0460] Twelfth Example

[0461] The twelfth example can be defined as shown in Table 36 below. Four transformation sets can be defined, and in this case, each transformation set can be composed of two transformation matrices.

[0462] [Table 36-1]

[0463] [Table 36-2]

[0464] 13th Example

[0465] The thirteenth example can be defined as shown in Table 37 below. Thirty-five transformation sets can be defined, and each transformation set can be composed of two transformation matrices.

[0466] [Table 37-1]

[0467] [Table 37-2]

[0468] Fourteenth Example

[0469] The fourteenth example can be defined as shown in Table 38 below. Thirty-five transformation sets can be defined, and each transformation set can consist of one transformation matrix.

[0470] [Table 38]

[0471] 15th Example

[0472] The 15th example can be defined as shown in Table 39 below. 35 transformation sets can be defined, and each transformation set can consist of one transformation matrix.

[0473] [Table 39]

[0474] Example 16

[0475] The 16th example can be defined as shown in Table 40 below. 35 transformation sets can be defined, and each transformation set can be composed of two transformation matrices.

[0476] [Table 40-1]

[0477] [Table 40-2]

[0478] FIG. 27 is a flowchart illustrating a method for converting a video signal according to an embodiment to which the present invention is applied.

[0479] 27, for convenience of explanation, the description will be focused on a decoder, but the present invention is not limited thereto, and the conversion method for a video signal according to this embodiment can be applied substantially similarly to an encoder. The flowchart of FIG. 27 can be performed by the decoding device 200 or the inverse conversion unit 230.

[0480] The decoding apparatus 200 determines a secondary transform set to be applied to a current block from among predefined secondary transform sets based on an intra-prediction mode of the current block (S2701).

[0481] The decoding device 200 obtains a first syntax element indicating a secondary transform matrix to be applied to the current block within the determined secondary transform set (S2702).

[0482] The decoding device 200 derives a quadratic inverse-transformed block by performing a quadratic inverse transform on the upper left corner region of the current block using the quadratic transform matrix specified by the first syntax element (S2703).

[0483] The decoding apparatus 200 derives a residual block of the current block by performing a primary inverse transform on the secondarily inverse transformed block using a primary transform matrix of the current block (S2704).

[0484] As mentioned above, each of the predefined quadratic transformation sets may include two quadratic transformation matrices.

[0485] As described above, step S2704 may further include determining the input length and output length of the inverse quadratic transform based on the width and height of the current block. As described above, if the height and width of the current block are each 4, the input length of the non-separable transform may be determined to be 8 and the output length may be determined to be 16.

[0486] As described above, the decoding apparatus 200 can parse a second syntax element indicating a linear transformation matrix to be applied to the linear transformation of the current block, and can determine whether a secondary transformation can be applied to the current block based on the second syntax element.

[0487] As mentioned above, the step of determining whether a secondary transform can be applied can be performed by determining that a secondary transform can be applied to the current block if the second syntax element indicates a predefined specific transform type.

[0488] As mentioned above, the predefined specific transform type can be defined as DCT2.

[0489] 28 shows an example of a block diagram of an apparatus for processing a video signal as an embodiment to which the present invention is applied. The video signal processing apparatus of FIG. 28 may correspond to the encoding apparatus of FIG. 1 or the decoding apparatus of FIG. 2.

[0490] The image processing device 2800 for processing an image signal includes a memory 2820 for storing an image signal, and a processor 2810 coupled to the memory for processing the image signal.

[0491] The processor 2810 according to an embodiment of the present invention may be configured with at least one processing circuit for processing a video signal, and may process the video signal by executing a command for encoding or decoding the video signal. That is, the processor 2810 may encode original video data or decode an encoded video signal by executing the encoding or decoding method described above.

[0492] Furthermore, a processing method according to the present invention can be produced in the form of a computer-executable program and stored in a computer-readable storage medium. Multimedia data having a data structure according to the present invention can also be stored in a computer-readable storage medium. The computer-readable storage medium includes any type of storage device or distributed storage device in which computer-readable data is stored. The computer-readable storage medium can include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable storage medium can also include media realized in the form of a carrier wave (e.g., transmission over the Internet). The bitstream generated by the encoding method can be stored in a computer-readable storage medium or transmitted via a wired or wireless communication network.

[0493] Furthermore, embodiments of the present invention may be realized as a computer program product with program code, which may be executed on a computer according to embodiments of the present invention. The program code may be stored on a computer-readable carrier.

[0494] As described above, the embodiments described in the present invention may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in each figure may be implemented and executed on a computer, processor, microprocessor, controller, or chip.

[0495] In addition, the decoder and encoder to which the present invention is applied may be included in real-time communication devices such as multimedia broadcasting transmitting / receiving devices, mobile communication terminals, home cinema video devices, digital cinema video devices, surveillance cameras, video interaction devices, video communications, etc., mobile streaming devices, storage media, camcorders, video-on-demand (VoD) service providing devices, over-the-top (OTT) video devices, internet streaming service providing devices, three-dimensional (3D) video devices, video phones, video devices, medical video devices, etc., and are used to process video signals or data signals. For example, over-the-top (OTT) video devices may include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, digital video recorders (DVRs), etc.

[0496] Furthermore, a processing method according to the present invention can be produced in the form of a computer-executable program and stored in a computer-readable storage medium. Multimedia data having a data structure according to the present invention can also be stored in a computer-readable storage medium. The computer-readable storage medium includes any type of storage device or distributed storage device in which computer-readable data is stored. The computer-readable storage medium can include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable storage medium can also include media realized in the form of a carrier wave (e.g., transmission over the Internet). The bitstream generated by the encoding method can be stored in a computer-readable storage medium or transmitted via a wired or wireless communication network.

[0497] Furthermore, embodiments of the present invention may be realized as a computer program product with program code, which may be executed on a computer according to embodiments of the present invention. The program code may be stored on a computer-readable carrier.

[0498] The above-described embodiments are combinations of elements and features of the present invention in a predetermined form. Each element or feature should be considered optional unless otherwise explicitly stated. Each element or feature may be implemented without being combined with other elements or features. It is also possible to combine some elements and / or features to form an embodiment of the present invention. The order of operations described in the embodiments of the present invention may be changed. Some elements or features of any embodiment may be included in other embodiments, or may be substituted with corresponding elements or features of other embodiments. It is obvious that claims that do not have an explicit reference relationship in the claims may be combined to form an embodiment, or may be included as new claims by amendment after filing.

[0499] Embodiments of the present invention may be implemented by various means, such as hardware, firmware, software, or a combination thereof. In a hardware implementation, an embodiment of the present invention may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0500] In the case of implementation by firmware or software, an embodiment of the present invention may be implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above. The software code may be stored in a memory and driven by a processor. The memory may be located inside or outside the processor and may transmit and receive data to and from the processor by various means known in the art.

[0501] It is obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential characteristics of the present invention. Therefore, the above detailed description should not be interpreted as limiting in all respects, but should be regarded as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention. [Industrial Applicability]

[0502] The above-described preferred embodiments of the present invention have been disclosed for illustrative purposes, and those skilled in the art will appreciate that various other embodiments may be improved, modified, substituted, or added within the technical spirit and scope of the present invention as disclosed in the appended claims.

Claims

1. A method for decoding a video signal, comprising: obtaining a first syntax element that identifies which quadratic transformation matrix in a quadratic transformation set is to be applied to a current block; performing an inverse quadratic transform on the current block based on the quadratic transform matrix specified by the first syntax element to generate an inverse quadratic transformed block; performing a linear inverse transform on the second-order inverse transformed block based on a linear transform matrix of the current block to generate a residual block of the current block; A method in which the input length of the quadratic inverse transform is determined as 8 based on the width and height of the current block being 4, and the input length of the quadratic inverse transform is determined as 8 based on the width and height of the current block being 8.

2. The method described in claim 1, wherein the quadratic transformation set includes two quadratic transformation matrices.

3. Parsing a second syntax element containing information about the linear transformation matrix to be applied to generate the residual block of the current block; The method of claim 1 , further comprising: determining whether the inverse secondary transform is applicable to the current block based on the second syntax element.

4. The method described in claim 3, wherein it is determined that the secondary inverse transform is applicable to the current block based on the second syntax element containing information regarding a predetermined specific transform type.

5. The method described in claim 4, wherein the predetermined specific transform type is defined as DCT2.

6. A method for encoding a video signal, comprising: obtaining a prediction block of the current block; generating a residual block of the current block based on the predicted block; performing a linear transformation on the residual block to obtain a linearly transformed block; performing a secondary transformation on the primary transformed block; The step of performing the secondary transformation includes: determining which quadratic transformation matrix within a quadratic transformation set is to be applied to the linearly transformed block; applying the determined quadratic transformation matrix to the linearly transformed block to obtain a quadratic transformed block; generating a first syntax element that identifies the determined quadratic transformation matrix within the quadratic transformation set; A method in which the output length of the quadratic transform is determined as 8 based on the width and height of the current block being 4, and the output length of the quadratic transform is determined as 8 based on the width and height of the current block being 8.

7. The method described in claim 6, wherein the quadratic transformation set includes two quadratic transformation matrices.

8. generating a second syntax element indicating a linear transformation matrix to be applied to obtain the linearly transformed block; The method of claim 6 , further comprising: determining whether to apply the secondary transform to the primary transformed block based on the second syntax element.

9. determining to apply the secondary transformation to the primary transformed block based on the second syntax element containing information about a predetermined specific transformation type; 9. The method of claim 8, wherein the predetermined specific transform type is defined as DCT2.

10. A method for transmitting data including a bitstream relating to video, comprising: obtaining the bitstream for the video; transmitting the data including the bitstream; The bitstream comprises: obtaining a prediction block of the current block; generating a residual block of the current block based on the predicted block; performing a linear transformation on the residual block to obtain a linearly transformed block; and performing a secondary transform on the primary transformed block, The step of performing the secondary transformation includes: determining which quadratic transformation matrix within a quadratic transformation set is to be applied to the linearly transformed block; applying the determined quadratic transformation matrix to the linearly transformed block to obtain a quadratic transformed block; generating a first syntax element that identifies the determined quadratic transformation matrix within the quadratic transformation set; A method in which the output length of the quadratic transform is determined as 8 based on the width and height of the current block being 4, and the output length of the quadratic transform is determined as 8 based on the width and height of the current block being 8.

Images