Method and apparatus for encoding / decoding video signals.

Abstract

The present invention provides a video signal processing method and apparatus that improves conversion efficiency. [Solution] A video signal decoding method includes the steps of: determining a quadratic transformation set to be applied to the current block from a predefined set of quadratic transformations based on the intra-prediction mode of the current block; obtaining a first syntax element that indicates the quadratic transformation matrix to be applied to the current block within the determined quadratic transformation set; deriving a quadratic inversely transformed block by performing a quadratic inversely transformed on the upper left region of the current block using the quadratic transformation matrix specified by the first syntax element; and deriving a residual block of the current block by performing a linear inversely transformed on the quadratic inversely transformed block using the linear transformation matrix of the current block.

Description

Technical Field

[0001] The present invention relates to a method and an apparatus for processing video signals, and more specifically, to a method and an apparatus for encoding / decoding video signals by performing conversion.

Background Art

[0002] Compression encoding (encoding) means a series of signal processing techniques for transferring digitized information via a communication line or storing it in a form suitable for a storage medium. Media such as video, video, and audio can be targets of compression encoding, and in particular, a technique for performing compression encoding on video is called video compression.

[0003] Next-generation video content will come to have characteristics such as high spatial resolution, high frame rate, and high dimensionality of scene representation. In order to process such content, it brings a huge increase in terms of memory storage, memory access rate, and processing power.

[0004] Therefore, it is necessary to design coding tools for more efficiently processing next-generation video content. In particular, video codec standards after the HEVC (High Efficiency Video Coding) standard require an efficient conversion technique for converting a video signal in the spatial domain to the frequency domain together with prediction techniques having higher accuracy.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The object of the present invention is to provide an image signal processing method and apparatus that applies a transformation suitable for the current block.

[0006] The technical problems that this invention aims to solve are not limited to those mentioned above, and any other technical problems not mentioned should be clearly understood by a person with ordinary skill in the art to which this invention belongs from the description below. [Means for solving the problem]

[0007] One aspect of the present invention relates to a method for decoding a video signal, which may include the steps of: determining a set of secondary transforms to be applied to the current block from a predefined set of secondary transforms based on the intra-prediction mode of the current block; obtaining a first syntax element that indicates a secondary transform matrix to be applied to the current block within the determined set of secondary transforms; deriving a quadratic inversely transformed block by performing a quadratic inversely transformed on the upper left region of the current block using the quadratic transform matrix identified by the first syntax element; and deriving a residual block of the current block by performing a linear inversely transformed on the quadratic inversely transformed block using the primary transform matrix of the current block.

[0008] Preferably, the predefined set of quadratic transformations can each include two quadratic transformation matrices.

[0009] Preferably, the step of deriving the quadratic inverse transform block may further include the step of determining the input and output lengths of the quadratic inverse transform based on the width and height of the current block.

[0010] Preferably, if the height and width of the current block are each 4, the input length of the non-separated transform can be determined to be 8 and the output length to be 16.

[0011] Preferably, the process may further include the steps of parsing a second syntax element that indicates a linear transformation matrix to be applied to the linear transformation of the current block, and determining whether a quadratic transformation can be applied to the current block based on the second syntax element.

[0012] Preferably, the step of determining whether the quadratic transformation can be applied can be performed by determining that the quadratic transformation can be applied to the current block if the second syntax element indicates a predefined specific transformation type.

[0013] Preferably, the predefined specific conversion type can be defined as DCT2.

[0014] Another aspect of the present invention relates to a device for decoding a video signal, comprising a memory for storing the video signal and a processor coupled to the memory, wherein the processor determines a set of secondary transforms to be applied to the current block from a predefined set of secondary transforms based on the intra-prediction mode of the current block, obtains a first syntax element indicating a secondary transform matrix to be applied to the current block within the determined set of secondary transforms, derives a quadratic inversely transformed block by performing a quadratic inversely transformed on the upper left region of the current block using the quadratic transform matrix specified by the first syntax element, and derives a residual block of the current block by performing a linear inversely transformed on the quadratic inversely transformed block using the primary transform matrix of the current block. [Effects of the Invention]

[0015] According to embodiments of the present invention, conversion efficiency can be improved by determining and applying a conversion suitable for the current block.

[0016] Furthermore, according to embodiments of the present invention, by efficiently designing the transformations used in primary and secondary transformations, the computational complexity can be improved and compression performance can be enhanced.

[0017] Furthermore, according to the embodiments of the present invention, the computational complexity can be significantly improved by restricting the transformation kernel of the linear transformation to which the quadratic transformation is applied.

[0018] The effects obtained by the present invention are not limited to those mentioned above, and any other effects not mentioned should be clearly understood by a person with ordinary skill in the art to which the present invention pertains from the following description. [Brief explanation of the drawing]

[0019] The accompanying drawings, included as part of the detailed description to aid in understanding the present invention, provide embodiments of the invention and illustrate the technical features of the invention together with the detailed description.

[0020] [Figure 1] An example of a video coding system is shown as an embodiment to which the present invention is applied. [Figure 2] As an embodiment to which the present invention is applied, a schematic block diagram of an encoding device in which video / image signals are encoded is shown. [Figure 3] A schematic block diagram of a decoding device that performs video signal decoding is shown as an embodiment to which the present invention is applied. [Figure 4] This is a structural diagram of a content streaming system, which is an embodiment to which the present invention is applied. [Figure 5a] This figure illustrates a block division structure using a QT (QuadTree, hereinafter referred to as "QT") as an embodiment to which the present invention can be applied. [Figure 5b]As an embodiment to which the present invention can be applied, it is a diagram for explaining a block division structure by BT (Binary Tree, hereinafter referred to as "BT"). [Figure 5c] As an embodiment to which the present invention can be applied, it is a diagram for explaining a block division structure by TT (Ternary Tree, hereinafter referred to as "TT"). [Figure 5d] As an embodiment to which the present invention can be applied, it is a diagram for explaining a block division structure by AT (Asymmetric Tree, hereinafter referred to as "AT"). [Figure 6] As an embodiment to which the present invention is applied, a schematic block diagram of a conversion and quantization unit and an inverse quantization and inverse conversion unit in an encoding device is shown. [Figure 7] As an embodiment to which the present invention is applied, a schematic block diagram of an inverse quantization and inverse conversion unit in a decoding device is shown. [Figure 8] It is a flowchart showing a process in which AMT (adaptive multiple transform) is executed. [Figure 9] It is a flowchart showing a decoding process in which AMT is executed. [Figure 10] Based on an embodiment of the present invention, it is a flowchart showing an inverse conversion process based on MTS. [Figure 11] Based on an embodiment of the present invention, it is a block diagram of a device that executes decoding based on MTS. [Figure 12] As an embodiment to which the present invention is applied, it is an encoding / decoding flowchart to which a secondary conversion is applied. [Figure 13] As an embodiment to which the present invention is applied, it is an encoding / decoding flowchart to which a secondary conversion is applied. [Figure 14] As an embodiment to which the present invention is applied, a diagram for explaining Givens rotation is shown. [Figure 15]As an embodiment to which the present invention is applied, a 1-round configuration of a 4x4 NSST (non-separable secondary transform) consisting of a Givens rotation layer and permutation is shown. [Figure 16] As an embodiment to which the present invention is applied, the operation of RST (reduced secondary transform) is shown. [Figure 17] This figure shows the process of performing reverse scans from the 64th to the 17th based on the reverse scan order, as an embodiment to which the present invention is applied. [Figure 18] An example of an encoding flowchart using a single transform indicator (STI) is shown as an embodiment to which the present invention is applied. [Figure 19] An example of an encoding flowchart using a unified transform indicator (UTI) is shown as an embodiment to which the present invention is applied. [Figure 20a] Another example of an encoding flowchart using a UTI is shown as an embodiment to which the present invention is applied. [Figure 20b] Another example of an encoding flowchart using a UTI is shown as an embodiment to which the present invention is applied. [Figure 21] An example of an encoding flowchart that performs a conversion is shown as an embodiment to which the present invention is applied. [Figure 22] An example of a decoding flowchart that performs a conversion is shown as an embodiment to which the present invention is applied. [Figure 23] An example of a detailed block diagram of the conversion unit 120 within the encoding device 100 is shown as an embodiment to which the present invention is applied. [Figure 24] An example of an embodiment to which the present invention is applied is shown, which is a detailed block diagram of the inverse conversion unit 230 in the decoding device 200. [Figure 25] A flowchart for processing a video signal is shown as an embodiment to which the present invention is applied. [Figure 26] This flowchart illustrates a method for converting video signals according to embodiments to which the present invention is applied. [Figure 27] This flowchart illustrates a video signal conversion method according to an embodiment to which the present invention is applied. [Figure 28] An example of a block diagram of an apparatus for processing video signals is shown as an embodiment to which the present invention is applied. [Modes for carrying out the invention]

[0021] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to illustrate exemplary embodiments of the present invention and is not intended to show only the only embodiments in which the present invention can be carried out. The following detailed description includes specific details to provide a complete understanding of the present invention. However, those skilled in the art will see that the present invention can be carried out without such specific details.

[0022] In some cases, known structures and devices may be omitted or shown in the form of block diagrams focusing on the core function of each structure and device, in order to avoid ambiguity of the concept of the present invention.

[0023] Furthermore, while the terminology used in this invention has been selected as widely used and common terms whenever possible, in certain cases, the applicant has chosen terminology at their discretion for explanation. In such cases, the meaning will be clearly described in the detailed explanation of that part, and it should be made clear that the terminology used in the description of this invention should not be interpreted simply by its name alone, but rather by understanding the meaning of that term.

[0024] Certain terms used in the following description are provided to aid in understanding the present invention, and the use of such specific terms can be modified in other ways without departing from the technical spirit of the invention. For example, terms such as signals, data, samples, pictures, frames, and blocks may be appropriately substituted and interpreted during each coding process.

[0025] Hereinafter, in this specification, “processing unit” means a unit on which an encoding / decoding process, such as prediction, transformation, and / or quantization, is performed. Furthermore, “processing unit” can be interpreted to include units of luminance (luma) components and units of chroma components. For example, a processing unit may correspond to a block, a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0026] In this specification, pixels or similar elements will be referred to as samples. Using samples can be interpreted as using pixel values ​​or similar elements.

[0027] Furthermore, processing units are not necessarily limited to square blocks; they may also be composed of polygons with three or more vertices.

[0028] In this specification, pixels or similar elements will be referred to as samples. Using samples can be interpreted as using pixel values ​​or similar elements.

[0029] Figure 1 shows an example of a video coding system as an embodiment to which the present invention is applied.

[0030] The video coding system may include a source device 10 and a receiving device 20. The source device 10 can transmit encoded video / image information or data to the receiving device 20 via a digital storage medium or network in file or streaming form.

[0031] The source device 10 may include a video source 11, an encoding device 12, and a transmitter 13. The receiving device 20 may include a receiver 21, a decoding device 22, and a renderer 23. The encoding device 10 may be called a video / image encoding device, and the decoding device 20 may be called a video / image decoding device. The transmitter 13 may be included in the encoding device 12. The receiver 21 may be included in the decoding device 22. The renderer 23 may include a display unit, or it may consist of another device or external component of the display unit.

[0032] A video source can acquire video / images through processes such as video / image capture, synthesis, or generation. A video source can include video / image capture devices and / or video / image generation devices. Video / image capture devices can include, for example, one or more cameras, or video / image archives containing previously captured video / images. Video / image generation devices can include, for example, computers, tablets, and smartphones, and can generate video / images (electronically). For example, virtual video / images can be generated through a computer, in which case the video / image capture process can be replaced by the process of generating the associated data.

[0033] The encoding device 12 can encode the input video / image. The encoding device 12 can perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0034] The transfer unit 13 can output the encoded video / image information or data in bitstream form and transmit it to the receiving unit of a receiving device via a digital storage medium or network in file or streaming form. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray®, HDD, SSD, etc. The transfer unit 13 may include elements for generating media files via a predetermined file format and may include elements for transmission via a broadcast / communication network. The receiver 21 can extract the bitstream and transmit it to the decoding device 22.

[0035] The decoding device 22 can decode video / images by performing a series of steps, such as inverse quantization, inverse transformation, and prediction, corresponding to the operation of the encoding device 12.

[0036] Renderer 23 can render the decoded video / image. The rendered video / image can be displayed via the display unit.

[0037] Figure 2 shows a schematic block diagram of an encoding device in which video / image signal encoding is performed, as an embodiment to which the present invention is applied. The encoding device 100 in Figure 2 can correspond to the encoding device 12 in Figure 1.

[0038] The video splitting unit 110 can split the input image (or picture, frame) input to the encoding device 100 into one or more processing units. For example, the processing units may be called coding units (CUs). In this case, coding units can be recursively split from a coding tree unit (CTU) or the largest coding unit (LCU) based on a QTBT (Quad-tree binary-tree) structure. For example, a single coding unit can be split into multiple coding units of deeper depth based on a quad-tree structure and / or a binary tree structure. In this case, for example, the quad-tree structure may be applied first and the binary tree structure later, or the binary tree structure may be applied first. Based on the final coding unit that cannot be further split, the coding procedure according to the present invention can be executed. In this case, based on coding efficiency according to the characteristics of the video, the largest coding unit can be immediately used as the final coding unit, or, if necessary, the coding unit can be recursively divided into lower-depth coding units, and the coding unit of optimal size can be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and reconstruction, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit can be divided or partitioned from the final coding unit described above, respectively. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit that derives transformation coefficients and / or a unit that derives a residual signal from transformation coefficients.

[0039] The term "unit" is sometimes used interchangeably with terms such as "block" or "area." Generally, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally refer to a pixel or a pixel value, or only to the luminance (luma) component pixel / pixel value, or only to the chroma component pixel / pixel value. A sample is used as a term corresponding to a single picture (or image) pixel or pel.

[0040] The encoding device 100 can generate a residual signal (remaining blocks, remaining sample array) by subtracting the predicted signal (predicted blocks, predicted sample array) output from the inter-prediction unit 180 or intra-prediction unit 185 from the input video signal (original blocks, original sample array), and the generated residual signal is transferred to the conversion unit 120. In this case, as shown in the figure, the unit that subtracts the predicted signal (predicted blocks, predicted sample array) from the input video signal (original blocks, original sample array) within the encoder 100 can be called the subtraction unit 115. The prediction unit can predict the block to be processed (hereinafter referred to as the current block) and generate a predicted block that includes the predicted samples of the current block. The prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block or CU. As will be described later in the description of each prediction mode, the prediction unit can generate various information related to prediction, such as prediction mode information, and transfer it to the entropy encoding unit 190. Prediction information can be encoded by the entropy encoding unit 190 and output in bitstream format.

[0041] The intra-prediction unit 185 can predict the current block by referring to a sample in the current picture. The referenced sample may be located in the vicinity (neighbor) of the current block or at a distance, depending on the prediction mode. The prediction mode in intra-prediction can include multiple non-directional modes and multiple directional modes. Non-directional modes may include, for example, DC mode and Planar mode. Directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is an example, and depending on the settings, more or fewer directional prediction modes may be used. The intra-prediction unit 185 can also determine the prediction mode to apply to the current block using the prediction modes applied to the surrounding blocks.

[0042] The interprediction unit 180 can guide the predicted block of the current block based on a reference block (reference sample array) identified by motion vectors on the reference picture. In this case, to reduce the amount of motion information transferred in interprediction mode, motion information can be predicted in blocks, subblocks, or samples based on the correlation of motion information between surrounding blocks and the current block. The motion information can include motion vectors and reference picture indices. The motion information can further include information on the interprediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, surrounding blocks can include spatial neighboring blocks that exist in the current picture and temporal neighboring blocks that exist in the reference picture. The reference picture containing the reference block and the reference picture containing the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, same location (CU colCU), etc., and the reference picture containing the temporal neighboring block may be called a collocated picture (colPic). For example, the interpretation unit 180 can construct a motion information candidate list based on surrounding blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Interpretation can be performed based on various prediction modes; for example, in skip mode and merge mode, the interpretation unit 180 can use the motion information of surrounding blocks as the motion information of the current block. In skip mode, unlike merge mode, the residual signal may not be transferred.In motion vector prediction (MVP) mode, the motion vector of the current block can be indicated by signaling the motion vector difference using the motion vectors of surrounding blocks in a motion vector predictor.

[0043] The prediction signal generated via the interpretation unit 180 or the intrapretation unit 185 is used to generate a restored signal or to generate a residual signal.

[0044] The transformation unit 120 can generate transformation coefficients by applying a transformation method to the residual signal. For example, the transformation method may include at least one of the following: DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT refers to a transformation obtained from a graph representing the relationship information between pixels. CNT refers to a transformation obtained by generating a prediction signal using all previously reconstructed pixels. The transformation process may also be applied to pixel blocks of the same size and square shape, or it may be applied to non-square, variable-size blocks.

[0045] The quantization unit 130 quantizes the conversion coefficients and transfers them to the entropy encoding unit 190, which can encode the quantized signal (information about the quantized conversion coefficients) and output it as a bitstream. Information about the quantized conversion coefficients can be called residual information. The quantization unit 130 can rearrange the quantized conversion coefficients in block form into a one-dimensional vector form based on the coefficient scan order, and can also generate information about the quantized conversion coefficients based on the quantized conversion coefficients in the form of a one-dimensional vector. The entropy encoding unit 190 can perform various encoding methods, such as exponential Golomb, CAVLC (context-adaptive variable length coding), and CABAC (context-adaptive binary arithmetic coding). In addition to the quantized conversion coefficients, the entropy encoding unit 190 can also encode information necessary for video / image restoration (such as the values ​​of syntax elements) together or separately. The encoded information (e.g., video / image information) can be transferred or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The bitstream can be transferred over a network or stored on a digital storage medium. Here, the network can include broadcast networks and / or other networks, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit 190 can be transmitted by a transmission unit (not shown) and / or stored by a storage unit (not shown) which are configured as internal / external elements of the encoding device 100, or the transfer unit may be a component of the entropy encoding unit 190.

[0046] The quantized conversion coefficients output from the quantization unit 130 can be used to generate a prediction signal. For example, the quantized conversion coefficients can be used to reconstruct the residual signal by applying inverse quantization and inverse transformation via the inverse quantization unit 140 and inverse transformation unit 150 in the loop. The adder 155 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter-prediction unit 180 or intra-prediction unit 185. If there is no residual for the block to be processed, as in the case where skip mode is applied, the predicted block is used as the reconstructed block. The adder 155 can be called the reconstruction unit or reconstructed block generation unit. The generated reconstructed signal is used for intra-prediction of the next block to be processed in the current picture, and may also be used for inter-prediction of the next picture after filtering, as described later.

[0047] The filtering unit 160 can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 160 can apply various filtering methods to the restored picture to generate a modified restored picture, and can transfer the modified restored picture to the decoded picture buffer 170. Various filtering methods can include, for example, ji-blocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter. The filtering unit 160 can generate various filtering information, as described later in the description of each filtering method, and transfer it to the entropy encoding unit 190. The filtering information can be encoded by the entropy encoding unit 190 and output in bitstream format.

[0048] The corrected restored picture transferred to the decoded picture buffer 170 is used as a reference picture by the interpretation unit 180. This allows the encoding device to avoid prediction mismatches between the encoding device 100 and the decoding device when interpretation is applied, and also improves encoding efficiency.

[0049] The decoded picture buffer 170 can store the corrected restored picture for use as a reference picture from the interpretation unit 180.

[0050] Figure 3 shows a schematic block diagram of a decoding device that decodes a video signal, as an embodiment to which the present invention is applied. The decoding device 200 in Figure 3 can correspond to the decoding device 22 in Figure 1.

[0051] Referring to Figure 3, the decoding device 200 can be configured to include an entropy decoding unit 210, an inverse quantization unit 220, an inverse transformation unit 230, an addition unit 235, a filtering unit 240, a decoded picture buffer (DPB) 250, an inter-prediction unit 260, and an intra-prediction unit 265. The inter-prediction unit 260 and the intra-prediction unit 265 can be collectively called the prediction unit. In other words, the prediction unit can include an inter-prediction unit 180 and an intra-prediction unit 185. The inverse quantization unit 220 and the inverse transformation unit 230 can be collectively called the residual processing unit. In other words, the residual processing unit can include an inverse quantization unit 220 and an inverse transformation unit 230. The aforementioned entropy decoding unit 210, inverse quantization unit 220, inverse transformation unit 230, addition unit 235, filtering unit 240, inter-prediction unit 260, and intra-prediction unit 265 can be configured by a single hardware component (e.g., a decoder or a processor) depending on the embodiment. Furthermore, the decoded picture buffer 250 can be implemented by a single hardware component (e.g., memory or a digital storage medium) depending on the embodiment.

[0052] When a bitstream containing video / image information is input, the decoding device 200 can reconstruct the image in accordance with the process by which the video / image information from the encoding device 100 in Figure 2 has been processed. For example, the decoding device 200 can decode using the processing units applied in the encoding device 100. Therefore, the decoding processing unit may be, for example, a coding unit, which can be divided along a quad-tree structure and / or binary tree structure from a coding tree unit or a maximum coding unit. The reconstructed video signal decoded and output via the decoding device 200 can then be reproduced through a playback device.

[0053] The decoding device 200 can receive the signal output from the encoding device 100 in Figure 2 in bitstream form, and the received signal can be decoded via the entropy decoding unit 210. For example, the entropy decoding unit 210 can farthe (analyze) the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). For example, the entropy decoding unit 210 can decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntactic elements necessary for image restoration and the quantized values ​​of conversion coefficients related to residuals. More specifically, the CABAC entropy decoding method receives the bins corresponding to each syntactic element from the bitstream, determines a context model using information about the syntactic element to be decoded, the surrounding and decoded information of the block to be decoded, or the symbol / bin information decoded in the previous step, predicts the probability of bin occurrence based on the determined context model, and performs arithmetic decoding of the bins to generate the symbols corresponding to the values ​​of each syntactic element. At this time, after determining the context model, the CABAC entropy decoding method can update the context model using the symbol / bin information decoded for the context model of the next symbol / bin. Of the information decoded by the entropy decoding unit 210, information related to predictions is provided by the prediction unit (inter-prediction unit 260 and intra-prediction unit 265), and the residual values ​​that have been entropy decoded from the entropy decoding unit 210, i.e., quantized conversion coefficients and related parameter information, can be input to the inverse quantization unit 220. Furthermore, information related to filtering from the information decoded by the entropy decoding unit 210 can be provided to the filtering unit 240. On the other hand, a receiving unit (not shown) that receives the signal output from the encoding device 100 may be further configured as an internal / external element of the decoding device 200, or the receiving unit may be a component of the entropy decoding unit 210.

[0054] The inverse quantization unit 220 can inverse quantize the quantized transformation coefficients and output the transformation coefficients. The inverse quantization unit 220 can rearrange the quantized transformation coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device 100. The inverse quantization unit 220 can perform inverse quantization of the quantized transformation coefficients using quantization parameters (e.g., quantization stage size information) to obtain the transformation coefficients.

[0055] The inverse conversion unit 230 inversely converts the conversion coefficients to obtain a resistive signal (residual block, resistive sample array).

[0056] The prediction unit can perform a prediction on the current block and generate a predicted block that includes the predicted sample of the current block. Based on the prediction information output from the entropy decoding unit 210, the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block, and can determine a specific intra / inter-prediction mode.

[0057] The intra-prediction unit 265 can predict the current block by referring to a sample in the current picture. Depending on the prediction mode, the referenced sample can be located in the vicinity (neighbor) of the current block or at a distance. The prediction mode in intra-prediction can include multiple non-directional modes and multiple directional modes. The intra-prediction unit 265 can also determine the prediction mode to be applied to the current block using the prediction mode applied to the surrounding block.

[0058] The interprediction unit 260 can derive the predicted block of the current block based on a reference block (reference sample array) identified by motion vectors on the reference picture. In this case, to reduce the amount of motion information transferred in interprediction mode, motion information can be predicted in blocks, subblocks, or samples based on the correlation of motion information between surrounding blocks and the current block. Motion information can include motion vectors and reference picture indices. Motion information can further include information on the interprediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In interprediction, surrounding blocks can include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the interprediction unit 260 can construct a motion information candidate list based on surrounding blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Interprediction can be performed based on various prediction modes, and the prediction information can include information indicating the interprediction mode of the current block.

[0059] The summing unit 235 can generate a restored signal (restored picture, restored block, restored, sample array) by adding the acquired residual signal to the predicted signal (predicted block, predicted sample array) output from the inter-prediction unit 260 or intra-prediction unit 265. If there is no residual for the block to be processed, as in the case where skip mode is applied, the predicted block is used as the restored block.

[0060] The addition unit 235 can be called the restoration unit or restoration block generation unit. The generated restoration signal is used for intra-prediction of the next block to be processed in the current picture, and can also be used for intra-prediction of the next picture after being filtered as described later.

[0061] The filtering unit 240 can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 240 can apply various filtering methods to the restored picture to generate a modified restored picture, and can transfer the modified restored picture to the decoded picture buffer 250. Various filtering methods can include, for example, ji-blocking filtering, sample adaptive offset (SAO), adaptive loop filter (ALF), and bilateral filter.

[0062] The corrected restored picture, transferred to the decoded picture buffer 250, is used as a reference picture by the interpretation unit 260.

[0063] In this specification, the embodiments described for the filtering unit 160, inter-prediction unit 180, and intra-prediction unit 185 of the encoding device 100 can be applied identically or in a corresponding manner to the filtering unit 240, inter-prediction unit 260, and intra-prediction unit 265 of the respective decoding devices.

[0064] Figure 4 is a structural diagram of a content streaming system as an embodiment to which the present invention is applied.

[0065] A content streaming system to which the present invention is applied may broadly include an encoding server 410, a streaming server 420, a web server 430, media storage 440, a user device 450, and a multimedia input device 460.

[0066] The encoding server 410 is responsible for compressing content input from multimedia input devices such as smartphones, cameras, and CAM coders into digital data to generate a bitstream, and then transferring this bitstream to the streaming server 420. In other cases, if the multimedia input device 460, such as a smartphone, camera, or CAM coder, directly generates the bitstream, the encoding server 410 can be omitted.

[0067] The bitstream can be generated by an encoding method or a method for generating a bitstream to which the present invention is applied, and the streaming server 420 can temporarily store the bitstream in the process of transferring or receiving the bitstream.

[0068] The streaming server 420 transfers multimedia data to the user device 450 based on a user request via the web server 430, and the web server 430 acts as an intermediary to inform the user of available services. When the user requests a desired service from the web server 430, the web server 430 transmits this to the streaming server 420, and the streaming server 420 transfers the multimedia data to the user. In this case, the content streaming system may include a separate control server, in which case the control server controls the commands / responses between each device in the content streaming system.

[0069] The streaming server 420 can receive content from the media storage 440 and / or the encoding server 410. For example, if it receives content from the encoding server 410, it can receive the content in real time. In this case, in order to provide a smooth streaming service, the streaming server 420 can store the bitstream for a certain period of time.

[0070] Examples of user devices 450 include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (such as smartwatches, smart glasses, and HMDs), digital TVs, desktop computers, and digital signage.

[0071] Each server in a content streaming system can be operated as a distributed server, in which case the data received by each server can be processed in a distributed manner.

[0072] Figure 5 illustrates embodiments to which the present invention can be applied, with Figure 5a illustrating a block partitioning structure using a Quad Tree (QT), Figure 5b illustrating a Binary Tree (BT), Figure 5c illustrating a Ternary Tree (TT), and Figure 5d illustrating an Asymmetric Tree (AT).

[0073] In video coding, a block can be divided based on QT. A subblock divided by QT can then be recursively divided further using QT. A leaf block that is no longer divided by QT can be divided by at least one of the following methods: BT, TT, or AT. BT can have two forms of division: horizontal BT (2NxN, 2NxN) and vertical BT (Nx2N, Nx2N). TT can have two forms of division: horizontal TT (2Nx1 / 2N, 2NxN, 2Nx1 / 2N) and vertical TT (1 / 2Nx2N, Nx2N, 1 / 2Nx2N). AT can have four forms of partitioning: horizontal-up AT (2Nx1 / 2N, 2Nx3 / 2N), horizontal-down AT (2Nx3 / 2N, 2Nx1 / 2N), vertical-left AT (1 / 2Nx2N, 3 / 2Nx2N), and vertical-right AT (3 / 2Nx2N, 1 / 2Nx2N). Each of BT, TT, and AT can be further recursively partitioned using BT, TT, and AT.

[0074] Figure 5a shows an example of QT partitioning. Block A can be partitioned into four subblocks (A0, A1, A2, A3) by QT. Subblock A1 can again be partitioned into four subblocks (B0, B1, B2, B3) by QT.

[0075] Figure 5b shows an example of BT partitioning. Block B3, which is no longer partitioned by QT, can be partitioned into vertical BT(C0, C1) or horizontal BT(D0, D1). Like block C0, each subblock can be recursively partitioned further into the form of horizontal BT(E0, E1) or vertical BT(F0, F1).

[0076] Figure 5c shows an example of TT partitioning. Block B3, which is no longer partitioned by QT, can be partitioned into a vertical TT (C0, C1, C2) or a horizontal TT (D0, D1, D2). Like block C1, each subblock can be recursively partitioned further into the form of a horizontal TT (E0, E1, E2) or a vertical TT (F0, F1, F2).

[0077] Figure 5d shows an example of AT partitioning. Block B3, which is no longer partitioned by QT, can be partitioned into vertical AT(C0, C1) or horizontal AT(D0, D1). Like block C1, each subblock can be recursively partitioned further into the form of horizontal AT(E0, E1) or vertical TT(F0, F1).

[0078] On the other hand, BT, TT, and AT partitions can be used together for partitioning. For example, a subblock partitioned by BT can be partitioned by TT or AT. Similarly, a subblock partitioned by TT can be partitioned by BT or AT. A subblock partitioned by AT can be partitioned by BT or TT. For example, after a horizontal BT partition, each subblock can be partitioned into vertical BTs, or after a vertical BT partition, each subblock can be partitioned into horizontal BTs. In this case, although the order of partitioning is different, the final partitioned form is the same.

[0079] Furthermore, once a block is divided, the order in which the block is searched can be defined in various ways. Generally, searching a block from left to right and from top to bottom can mean the order in which additional blocks are decided for each divided subblock, or, if the block is no longer divided, the coding order of each subblock, or the search order when a subblock refers to information from other adjacent blocks.

[0080] Transformations can be performed on each processing unit (or transformation block) divided by a partition structure as shown in Figures 5a to 5d, and in particular, the division can be separated by row and column directions, and a transformation matrix can be applied. According to an embodiment of the present invention, other transformation types are used depending on the row or column length of the processing unit (or transformation block).

[0081] The transformation is applied to the residual block, which is done to decorrelate the residual block as much as possible, concentrate the coefficients at low frequencies, and create a zero tail at the top of the block. In JEM software, the transformation part includes two main functions: core transform and secondary transform. The core transform consists of the DCT (discrete cosine transform) and DST (discrete sine transform) family of transforms, which are applied to all rows and columns of the residual block. Subsequently, the secondary transform can be applied to the top left corner of the output of the core transform. Similarly, the inverse quadratic transform and the inverse core transform can be applied in that order. First, the inverse quadratic transform can be applied to the top left corner of the coefficient block. Subsequently, the inverse core transform is applied to the rows and columns of the output of the inverse quadratic transform. The core transform or inverse transform can be referred to as the primary transform or inverse transform.

[0082] Figures 6 and 7 show embodiments to which the present invention is applied. Figure 6 shows a schematic block diagram of the conversion and quantization unit (120 / 130) and the inverse quantization and inverse conversion unit (140 / 150) within the encoding device 100 of Figure 2, and Figure 7 shows a schematic block diagram of the inverse quantization and inverse conversion unit (220 / 230) within the decoding device 200.

[0083] Careful examination of Figure 6 reveals that the transformation and quantization unit (120 / 130) can include a primary transform unit 121, a secondary transform unit 122, and a quantization unit 130. The inverse quantization and inverse transformation unit (140 / 150) can include an inverse quantization unit 140, an inverse secondary transform unit 151, and an inverse primary transform unit 152.

[0084] A careful examination of Figure 7 reveals that the inverse quantization and inverse transformation unit (220 / 230) can include an inverse quantization unit 220, an inverse secondary transform unit 231, and an inverse primary transform unit 232.

[0085] In this invention, when performing a transformation, the transformation can be carried out through multiple stages. For example, as shown in Figure 6, two stages, a primary transform and a secondary transform, can be applied, or more transformation stages may be used based on the algorithm. Here, the primary transform can be referred to as the core transform.

[0086] The primary conversion unit 121 can apply a primary conversion to the residual signal, where the primary conversion can be predefined in a table from the encoder and / or decoder.

[0087] The secondary conversion unit 122 can apply a secondary conversion to a primary-converted signal, where the secondary conversion can be predefined in a table from the encoder and / or decoder.

[0088] In one embodiment, a non-separable secondary transform (NSST) can be conditionally applied as the secondary transform. For example, the NSST may be applied only to in-screen prediction blocks, and there may be a set of applicable transforms for each prediction mode group.

[0089] Here, prediction mode groups can be set based on the symmetry of the prediction direction. For example, prediction mode 52 and prediction mode 16 are symmetrical based on prediction mode 34 (diagonal direction), so they can form a single group and the same transform set can be applied. In this case, when applying the transform for prediction mode 52, it is applied after transposing the input data, because the transform set is the same as that for prediction mode 16.

[0090] On the other hand, in Planar mode and DC mode, there is no directional symmetry, so each has its own set of transformations, and each transformation set can consist of two transformations. For the remaining directional modes, each transformation set can consist of three transformations.

[0091] The quantization unit 130 can perform quantization on the quadratic-converted signal.

[0092] The inverse quantization and inverse transformation section (140 / 150) is performed by reversing the process described earlier, and redundant explanations are omitted.

[0093] Figure 7 shows a schematic block diagram of the inverse quantization and inverse transformation section (220 / 230) within the decoding device 200.

[0094] Referring to Figure 7, the inverse quantization and inverse transform unit (220 / 230) can include an inverse quantization unit 220, an inverse secondary transform unit 231, and an inverse primary transform unit 232.

[0095] The inverse quantization unit 220 uses quantization stage size information to obtain conversion coefficients from the entropy-decoded (decoded) signal.

[0096] In the inverse quadratic transformation unit 231, an inverse quadratic transformation is performed on the transformation coefficients. Here, the inverse quadratic transformation is the inverse transformation of the quadratic transformation explained in Figure 6.

[0097] The inverse linear transformation unit 232 performs an inverse linear transformation on the inverse quadratic transformed signal (or block) to obtain a residual signal. Here, the inverse linear transformation is the inverse transformation of the linear transformation (primary transform) described in Figure 6.

[0098] In addition to DCT-2 and 4x4 DST-4 applied to HEVC, adaptive (or explicit) multiple transform (AMT or EMT) techniques are used for residual coding of inter- and intra-encoded blocks. A number of selected transforms from HEVC and other DCT / DST families are used. Newly introduced transform matrices from JEM are DST-7, DCT-8, DST-1, and DCT-5. Table 1 below shows the basis functions for the selected DST / DCTs.

[0099] [Table 1]

[0100] EMT can be applied to CUs smaller than 64 or with the same width and height, and whether EMT is applied can be controlled by a CU level flag. If the CU level flag is 0, DCT-2 is applied to the CU to encode the residual. For intra-residual coding blocks within a CU to which EMT is applied, two additional flags are signaled to identify horizontal and vertical transformations. Like HEVC, JEM blocks can be coded in transformation skip mode. For intra-residual coding, a mode-dependent transformation candidate selection process is used, along with other residual statistics for other intra-prediction modes. Three transformation subsets are defined as shown in Table 2 below, and the transformation subset is selected based on the intra-prediction mode as shown in Table 3.

[0101] [Table 2]

[0102] Along with the subset concept, a subset of transformations is initially identified based on Table 2 by using the intra-prediction mode for CUs where the CU-level EMT_CU_flag is 1. Thereafter, for each of the horizontal (EMT_TU_horizontal_flag) and vertical (EMT_TU_vertical_flag) transformations, one of two transformation candidates within the identified subset of transformations is selected based on explicit signaling using flags, as shown in Table 3.

[0103] [Table 3]

[0104] [Table 4]

[0105] Table 4 shows a transform configuration group to which AMT (adaptive multiple transform) is applied, as an embodiment to which the present invention is applied.

[0106] A careful examination of Table 4 reveals that the transform configuration groups are determined based on the prediction mode, and there can be a total of six groups (G0 to G5). G0 to G4 apply when intra-prediction is applied, while G5 represents the combination of transformations (or transformation sets, transformation combination sets) applied to residual blocks generated by inter-prediction.

[0107] A single transformation can be achieved by applying a horizontal transform (or row transform) to the rows of the 2D block and a vertical transform (or column transform) to the columns.

[0108] Here, every group of conversion settings can have four candidate conversion combinations. The four candidate conversion combinations can be selected or determined via conversion combination indices from 0 to 3, and the conversion combination indices can be encoded and transferred from the encoder to the decoder.

[0109] In one embodiment, the residual data (or residual signal) obtained via intra-prediction may have different statistical characteristics depending on the intra-prediction mode. Therefore, as shown in Table 4, a different transformation may be applied to each intra-prediction instead of the general cosine transform. In this specification, the transformation types may be denoted, for example, as DCT-Type 2, DCT-II, and DCT-2.

[0110] Table 4, upon closer examination, reveals two cases: one using 35 intra-prediction modes and another using 67 intra-prediction modes. Multiple transformation combinations can be applied to each transformation setting group, separated by the columns for each intra-prediction mode. For example, multiple transformation combinations can consist of four combinations (row-direction transformation, column-direction transformation). Specifically, in group 0, DST-7 and DCT-5 can be applied to both the row (horizontal) and column (vertical) directions, resulting in a total of four possible combinations.

[0111] For each intra-prediction mode, a total of four transformation kernel combinations can be applied, and an index of the transformation combination for selecting one of these can be transferred for each transformation unit. In this specification, the index of the transformation combination may be called the AMT index and may be represented by amt_idx.

[0112] Furthermore, in addition to the transformation kernels presented in Table 4, there may be cases where DCT-2 is optimal for all row and column directions due to the characteristics of the residual signal. Therefore, by defining an AMT flag for each coding unit, the transformation can be applied adaptively. Here, if the AMT flag is 0, DCT-2 is applied for all row and column directions, and if the AMT flag is 1, one of four combinations can be selected or determined using the AMT index.

[0113] In one embodiment, if the AMT flag is 0, and the number of conversion coefficients for one conversion unit is less than 3, the conversion kernel in Table 4 is not applied, and DST-7 can be applied to all rows and columns.

[0114] In one embodiment, the values ​​of the conversion coefficients are parsed (analyzed) first, and if the number of conversion coefficients is less than 3, the AMT index is not analyzed, and the amount of additional information transferred can be reduced by applying DST-7.

[0115] In one embodiment, the AMT can be applied only when the width and height of the conversion unit are both 32 or less.

[0116] In one embodiment, Table 4 can be set based on offline training.

[0117] In one embodiment, the AMT index can be defined as a single index that can simultaneously refer to a combination of horizontal and vertical transformations. Alternatively, the AMT index can be defined by separate horizontal and vertical transformation indices.

[0118] Figure 8 is a flowchart showing the process by which AMT (adaptive multiple transform) is performed.

[0119] In this specification, embodiments of separable transforms, in which the transformation is applied separately to the horizontal and vertical directions, are described in principle, but the combination of transformations can also be configured as non-separable transforms.

[0120] Alternatively, combinations of transformations can be constructed using a mixture of separable and inseparable transformations. In this case, when inseparable transformations are used, row / column-specific transformation selection and horizontal / vertical-specific selection become unnecessary, and the transformation combinations in Table 4 can only be used when separable transformations are selected.

[0121] Furthermore, the methods proposed herein can be applied regardless of whether the transformation is linear or quadratic. In other words, there is no constraint that it must be applied to only one of the two, but it can be applied to both. Here, a linear transformation can mean a transformation for initially transforming the residual block, and a quadratic transformation can mean a transformation for applying the transformation to the block produced as a result of the linear transformation.

[0122] First, the encoding device 100 can determine the conversion group corresponding to the current block (S805). Here, the conversion group may refer to the conversion groups in Table 4, but the present invention is not limited thereto and may consist of other combinations of conversions.

[0123] The encoding device 100 can perform transformations on candidate transformation combinations used within the transformation group (S810). Based on the transformation results, the encoding device 100 can determine or select the transformation combination with the minimum RD (rate distortion) cost (S815). The encoding device 100 can encode the index of the transformation combination corresponding to the selected transformation combination (S820).

[0124] Figure 9 is a flowchart showing the decoding process in which AMT is performed.

[0125] First, the decoding device 200 can determine the transformation group for the current block (S905). The decoding device 200 can analyze the index of the transformation combination, where the index of the transformation combination can correspond to any one of the multiple transformation combinations in the transformation group (S910). The decoding device 200 can derive the transformation combination corresponding to the index of the transformation combination (S915). Here, the transformation combination can mean the transformation combinations described in Table 4, but the present invention is not limited thereto. In other words, configurations using other transformation combinations are also possible.

[0126] The decoding device 200 can perform an inverse transformation on the current block based on the transformation combination (S920). If the transformation combination consists of a row transformation and a column transformation, the row transformation can be applied first, followed by the column transformation. However, the present invention is not limited thereto, and the transformations can be applied in reverse, or if it consists of non-separable transformations, the non-separable transformation can be applied directly.

[0127] On the other hand, in another embodiment, the process of determining the transformation group and the process of analyzing the index of the transformation combinations can be performed simultaneously.

[0128] In embodiments of the present invention, the aforementioned term "AMT" can be redefined as "MTS (multiple transform set or multiple transform selection)". The MTS-related syntax and semantics described below are defined in the VVC (versatile video coding) standard document JVET-K1001-v4.

[0129] In an embodiment of the present invention, two MTS candidates are used for the directional mode and four MTS candidates are used for the non-directional mode, as described below.

[0130] A) Non-directional mode (DC, Planner)

[0131] When the MTS index is 0, DST-7 is used for horizontal and vertical conversion.

[0132] When the MTS index is 1, DST-7 is used for vertical transformation and DCT-8 is used for horizontal transformation.

[0133] When the MTS index is 2, DCT-8 is used for vertical transformation and DST-7 is used for horizontal transformation.

[0134] When the MTS index is 3, DCT-8 is used for horizontal and vertical transformation.

[0135] B) Modes belonging to horizontal group mode

[0136] When the MTS index is 0, DST-7 is used for horizontal and vertical conversion.

[0137] When the MTS index is 1, DCT-8 is used for vertical transformation and DST-7 is used for horizontal transformation.

[0138] C) Modes belonging to vertical group mode

[0139] When the MTS index is 0, DST-7 is used for horizontal and vertical conversion.

[0140] When the MTS index is 1, DST-7 is used for vertical transformation and DCT-8 is used for horizontal transformation.

[0141] Here (in VTM 2.0, which uses 67 modes), the horizontal group mode includes intra-prediction modes 2 through 34, and the vertical mode includes intra-prediction modes 35 through 66.

[0142] In another embodiment of the present invention, three MTS candidates are used for all intra-prediction modes.

[0143] When the MTS index is 0, DST-7 is used for horizontal and vertical conversion.

[0144] When the MTS index is 1, DST-7 is used for vertical transformation and DCT-8 is used for horizontal transformation.

[0145] When the MTS index is 2, DCT-8 is used for vertical transformation and DST-7 is used for horizontal transformation.

[0146] In another embodiment of the present invention, two MTS candidates are used for the direction prediction mode and three MTS candidates are used for the non-direction prediction mode.

[0147] A) Non-directional mode (DC, Planner)

[0148] When the MTS index is 0, DST-7 is used for horizontal and vertical conversion.

[0149] When the MTS index is 1, DST-7 is used for vertical transformation and DCT-8 is used for horizontal transformation.

[0150] When the MTS index is 2, DCT-8 is used for vertical transformation and DST-7 is used for horizontal transformation.

[0151] B) Prediction mode for horizontal group mode

[0152] When the MTS index is 0, DST-7 is used for horizontal and vertical conversion.

[0153] When the MTS index is 1, DCT-8 is used for vertical transformation and DST-7 is used for horizontal transformation.

[0154] C) Prediction mode applicable to the vertical group mode

[0155] When the MTS index is 0, DST-7 is used for horizontal and vertical conversion.

[0156] When the MTS index is 1, DST-7 is used for vertical transformation and DCT-8 is used for horizontal transformation.

[0157] In another embodiment of the present invention, one MTS candidate (e.g., DST-7) is used for all intra-modes. In this case, encoding time can be reduced by up to 40%, along with minor coding loss. Furthermore, one flag is used to indicate between DCT-2 and DST-7.

[0158] Figure 10 is a flowchart showing the inverse transformation process based on MTS according to an embodiment of the present invention.

[0159] A decoding device 200 to which the present invention is applied can obtain sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag (S1005). Here, sps_mts_intra_enabled_flag indicates whether cu_mts_flag exists in the residual coding syntax of the intracoding unit. For example, if sps_mts_intra_enabled_flag = 0, then cu_mts_flag does not exist in the residual coding syntax of the intracoding unit, and if sps_mts_intra_enabled_flag = 1, then cu_mts_flag exists in the residual coding syntax of the intracoding unit. And sps_mts_inter_enabled_flag indicates whether cu_mts_flag exists in the residual coding syntax of the intercoding unit. For example, if sps_mts_inter_enabled_flag = 0, then cu_mts_flag does not exist in the recurrent coding syntax of the intercoding unit, and if sps_mts_inter_enabled_flag = 1, then cu_mts_flag exists in the recurrent coding syntax of the intercoding unit.

[0160] The decoding device 200 can obtain cu_mts_flag based on sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag (S1010). For example, when sps_mts_intra_enabled_flag = 1 or sps_mts_inter_enabled_flag = 1, the decoding device 200 can obtain cu_mts_flag. Here, cu_mts_flag indicates whether the MTS is applied to the residual samples of the Luma conversion block. For example, if cu_mts_flag = 0, the MTS is not applied to the residual samples of the Luma conversion block, and if cu_mts_flag = 1, the MTS is applied to the residual samples of the Luma conversion block.

[0161] The decoding device 200 can obtain mts_idx based on cu_mts_flag (S1015). For example, when cu_mts_flag = 1, the decoding device 200 can obtain mts_idx, where mts_idx indicates which translation kernel is currently applied to the lumaregid dual samples along the horizontal and / or vertical directions of the translation block.

[0162] For example, with respect to mts_idx, at least one of the embodiments described herein can be applied.

[0163] The decoding device 200 can induce a conversion kernel corresponding to mts_idx (S1020). For example, the conversion kernel corresponding to mts_idx can be defined by dividing it into horizontal conversion and vertical conversion.

[0164] For example, if MTS is currently applied to a block (i.e., cu_mts_flag = 1), the decoding device 200 can configure MTS candidates based on the intra-prediction mode of the current block. In this case, the decoding flowchart in Figure 10 may further include a step for configuring MTS candidates. The decoding device 200 can then use mts_idx to determine which MTS candidate is applied to the current block from among the configured MTS candidates.

[0165] As another example, horizontal and vertical transformations may be applied to each other using different transformation kernels. However, the present invention is not limited thereto, and the same transformation kernel may also be applied to the horizontal and vertical transformations.

[0166] The decoding device 200 can then perform the inverse conversion based on the conversion kernel (S1025).

[0167] Furthermore, in this document, MTS can also be expressed as AMT or EMT, and similarly, it can be expressed as mts_idx, AMT_idx, EMT_idx, AMT_TU_idx, EMT_TU_idx, etc., and the present invention is not limited to such expressions.

[0168] Furthermore, in this invention, we distinguish between cases where MTS is applied and cases where it is not based on the MTS flag, but the present invention is not limited to such expressions. For example, whether or not to apply MTS may have the same meaning as whether or not to use a different conversion type (or conversion kernel) other than a predefined specific conversion type (which may be called a basic conversion type, default conversion type, etc.). If MTS is applied, a different conversion type other than the basic conversion type (for example, a conversion type which is one or more of a combination of multiple conversion types) may be used for conversion, and if MTS is not applied, the basic conversion type may be used for conversion. In one embodiment, the basic conversion type may be set (or defined) in DCT2.

[0169] As an example, the encoder may separately send an MTS flag syntax indicating whether or not MTS is applied to the current conversion block, and an MTS index syntax indicating the conversion type applied to the current block if MTS is applied. As another example, the encoder may send a syntax (e.g., an MTS index) that includes both whether or not MTS is applied to the current conversion block and the conversion type applied to the current block if MTS is applied. In other words, in the latter embodiment, the encoder may send a syntax (or syntax element) indicating the conversion type applied to the current conversion block (or unit) within the overall conversion type group (or conversion type set) including the basic conversion types described above.

[0170] Therefore, despite that expression, the syntax indicating the type of transformation currently applied to the transformation block (MTS index) may include information about whether or not MTS is applied. In other words, in the latter embodiment, only the MTS index is signaled without an MTS flag, so in this case it can be interpreted that MTS includes DCT2, but in the present invention, applying DCT2 may be described as not applying MTS, and nevertheless the technical scope of MTS is not limited to that definition.

[0171] Figure 11 is a block diagram of an apparatus that performs decoding based on an MTS according to an embodiment of the present invention.

[0172] The decoding device 200 to which the present invention is applied may include a sequence parameter acquisition unit 1105, an MTS flag acquisition unit 1110, an MTS index acquisition unit 1115, and a conversion kernel induction unit 1120.

[0173] The sequence parameter acquisition unit 1105 can acquire sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag. Here, sps_mts_intra_enabled_flag indicates whether cu_mts_flag exists in the residual coding syntax of the intracoding unit, and sps_mts_inter_enabled_flag indicates whether cu_mts_flag exists in the residual coding syntax of the intercoding unit. A specific example can be applied to the explanation related to Figure 105.

[0174] The MTS flag acquisition unit 1110 can acquire cu_mts_flag based on sps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag. For example, when sps_mts_intra_enabled_flag = 1 or sps_mts_inter_enabled_flag = 1, the MTS flag acquisition unit 1110 can acquire cu_mts_flag. Here, cu_mts_flag indicates whether MTS is applied to the residual sample of the Luma transform block. A specific example can be applied to the explanation related to Figure 10.

[0175] The MTS index acquisition unit 1115 can acquire mts_idx based on cu_mts_flag. For example, when cu_mts_flag = 1, the MTS index acquisition unit 1115 can acquire mts_idx. Here, mts_idx indicates which translation kernel is applied to the Lumarezid dual samples along the horizontal and / or vertical directions of the current translation block. A specific example can be found in the explanation in Figure 10.

[0176] The conversion kernel induction unit 1120 can induce a conversion kernel corresponding to mts_idx. Then, the decoding device 200 can perform the inverse conversion based on the induced conversion kernel.

[0177] The mode-dependent non-separable secondary transform (MDNSST) is introduced. To maintain low complexity, the MDNSST is applied only to low-frequency coefficients after the linear transform. A non-separable transform applied primarily to low-frequency coefficients can also be called the LFNST (low-frequency non-separable transform). If both the width (W) and height (H) of the transform coefficient block are 8 or greater, an 8x8 non-separable secondary transform is applied to the upper left 8x8 area of ​​the transform coefficient block. Otherwise, if either the width or height is less than 8, a 4x4 non-separable secondary transform is applied, and the 4x4 non-separable secondary transform can be performed on the upper left min(8,W)xmin(8,H) of the transform coefficient block. Here, min(A,B) is a function that outputs the smaller of A and B. Also, WxH is the size of the block, where W is the width and H is the height.

[0178] For 4x4 and 8x8 block sizes, there can be a total of 35x3 non-separable quadratic transforms, where 35 is the number of transform sets identified by the intra-prediction mode and 3 is the number of NSST candidates for each prediction mode. The mapping from intra-prediction modes to transform sets can be defined as shown in Table 5 below. Furthermore, according to embodiments of the present invention, among the four non-separable transform sets, depending on the intra-prediction mode...

[0179] [Table 5]

[0180] Within the translation set, an NSST index (NSST idx) can be coded to indicate the translation kernel. If NSST is not applied, an NSST index with a value of 0 can be signaled.

[0181] Figures 12 and 13 show an encoding / decoding flowchart to which a quadratic transformation is applied, as an embodiment to which the present invention is applied.

[0182] In JEM, the quadratic transformation (MDNSST) is not applied to blocks coded in transformation-skip mode. If the MDNSST index is signaled to the CU and is not zero, MDNSST is not used for blocks of components coded in transformation-skip mode within the CU. The overall coding structure, including coefficient encoding and NSST index coding, is shown in Figures 12 and 13. The CBF (coded block flag) is encoded to determine whether coefficient encoding and NSST coding are performed. In Figures 12 and 13, the CBF flag can indicate a luma-block cbf flag (cbf_luma flag) or a chroma-block cbf flag (cbf_cb flag or cbf_cr flag). When the CBF flag is 1, the transformation coefficients are coded.

[0183] Referring to Figure 12, the encoding device 100 checks whether CBF is 1 (S1205). If CBF is 0, the encoding device 100 does not perform encoding of the conversion coefficients or encoding of the NSST index. If CBF is 1, the encoding device 100 performs encoding of the conversion coefficients (S1210). Thereafter, the encoding device 100 decides whether to perform NSST index coding (S1215) and performs NSST index coding (S1220). If NSST index coding is not applied, the encoding device 100 terminates the conversion procedure in a state where NSST is not applied and can perform subsequent stages (e.g., quantization).

[0184] Referring to Figure 13, the decoding device 200 checks whether CBF is 1 (S1305). If CBF is 0, the decoding device 200 does not decode the conversion coefficients or the NSST index. If CBF is 1, the decoding device 200 decodes the conversion coefficients (S1310). Subsequently, the decoding device 200 decides whether to code the NSST index (S1315) and analyzes the NSST index (S1320).

[0185] NSST is not applied to the entire block (TU in the case of HEVC) to which the linear transformation is applied, but rather to the upper left 8x8 or 4x4 region. For example, if the block size is 8x8 or larger, the 8x8 NSST is applied, and if it is less than 8x8, the 4x4 NSST is applied. Also, when the 8x8 NSST is applied, the 4x4 NSST can be applied to every 4x4 block. Both the 8x8 NSST and the 4x4 NSST can be determined according to the configuration of the transformation set described above. As they are both non-separable transformations, the 8x8 NSST has 64 input data and 64 output data, while the 4x4 NSST can have 16 inputs and 16 outputs.

[0186] Figures 14 and 15 show embodiments to which the present invention is applied. Figure 14 illustrates a Givens rotation, and Figure 15 shows the configuration of one round in a 4x4 NSST consisting of a Givens rotation layer and permutation.

[0187] Both 8x8 NSST and 4x4 NSST can be constructed as hierarchical combinations of Givens rotations. The matrix corresponding to one Givens rotation is the same as in mathematical formula 1, and the matrix product is represented graphically as in Figure 14.

[0188]

number

[0189] In Figure 14, the tm and tn outputs of the Givens rotation can be calculated as shown in mathematical equation 2.

[0190]

number

[0191] As shown in Figure 14, one Givens rotation rotates two data points, so 32 Givens rotations are required to process 64 data points (for 8x8 NSST) or 16 data points (for 4x4 NSST), respectively, or 8 Givens rotations. Therefore, a bundle of 32 or 8 Givens rotations can form a Givens rotation layer. As shown in Figure 15, the output data of one Givens rotation layer is transmitted to the input data of the next Givens rotation layer via shuffling. As shown in Figure 15, the patterns to be shuffled are defined regularly, and in the case of 4x4 NSST, four Givens rotation layers and their corresponding shuffles form one round. 4x4 NSST is performed in two rounds, and 8x8 NSST is performed in four rounds. Each round uses the same shuffling pattern, but the Givens rotation angles applied are different. Therefore, it is necessary to store the angle data of all Givens rotations that constitute each transformation.

[0192] In the final stage, the data output after passing through the Givens rotation layer undergoes one more substitution, and the information of this substitution is stored separately for each transformation. This substitution is performed at the end of the forward NSST, and in the inverse NSST, this inverse substitution is applied first.

[0193] Reverse NSST performs the Givens rotation layer applied in forward NSST in the reverse order of substitution, and rotates by taking a negative (-) value for each Givens rotation angle.

[0194] RST (Reduced secondary transform)

[0195] FIG. 16 shows the operation of RST as an embodiment to which the present invention is applied.

[0196] When an orthogonal matrix indicating a primary transform has a form of NxN, RT (reduced transform) leaves only R out of N transform basis vectors (R < N). The matrix of the forward RT for generating transform coefficients can be defined as in Mathematical Expression 3.

[0197]

Equation

[0198] Since the matrix of the reverse RT is the transpose matrix of the forward RT matrix, if the application of the forward RT and the reverse RT is illustrated, it can be the same as FIGS. 16a and 16b.

[0199] The RT applied to the top-left 8x8 block of the block of transformation coefficients to which a linear transformation has been applied can be called an 8x8 RST. In mathematical formula 3, when the value of R is set to 16, the forward 8x8 RST has the form of a 16x64 matrix, and the reverse 8x8 RST has the form of a 64x16 matrix. Furthermore, the transformation set configurations shown in Table 5 can also be applied to the 8x8 RST. In other words, the 8x8 RST can be determined based on the transformation set corresponding to the intra-prediction mode, as shown in Table 5. Since one transformation set consists of two or three transformations depending on the intra-prediction mode, one of up to four transformations can be selected, including the case where no quadratic transformation is applied (one transformation can correspond to the identity matrix). When the four transformations are assigned the indices 0, 1, 2, and 3 respectively, the transformation to be applied can be specified by signaling the syntax element corresponding to the NSST index for each block of transformation coefficients. For example, index 0 can be assigned to the identity matrix, i.e., when no quadratic transformation is applied. In conclusion, for an 8x8 top-left block via NSST indexing, according to JEM NSST, an 8x8 NSST can be specified, and according to RST configuration, an 8x8 RST can be specified.

[0200] Figure 17 shows the process of performing reverse scans from the 64th to the 17th based on the reverse scan order, as an embodiment to which the present invention is applied.

[0201] When a forward 8x8 RST like the one in mathematical formula 3 is applied, 16 effective transformation coefficients are generated. As a result, the 64 input data points that make up the 8x8 region are reduced to 16 output data points. From the perspective of a two-dimensional region, only about 1 / 4 of the region is filled with effective transformation coefficients. Therefore, by applying the forward 8x8 RST, the 16 output data points obtained fill the region at the top left of Figure 17.

[0202] In Figure 17, the 4x4 area in the upper left corner becomes the ROI (region of interest) area where valid conversion coefficients are satisfied, and the remaining area is left empty. The empty area can be filled with a value of 0 by default. If a non-zero valid conversion coefficient is found outside the ROI area in Figure 17, it is certain that the 8x8 RST will not be applied, and the corresponding coding may be omitted for the NSST index. Conversely, if no non-zero conversion coefficient is found outside the ROI area in Figure 17 (when the area outside the ROI is filled with 0 if the 8x8 RST is applied), the NSST index can be coded because the 8x8 RST may have been applied in the past. This conditional NSST index coding requires checking for the presence or absence of non-zero conversion coefficients, so it can be performed after the residual coding process.

[0203] Figure 18 shows an example of an encoding flowchart using a single transform indicator as an embodiment to which the present invention is applied.

[0204] In embodiments of the present invention, a single transform indicator (STI) is introduced. Instead of sequentially using two transforms (a linear transform and a quadratic transform), a single transform can be applied when the single transform indicator is activated (STI coding == 1). Here, the single transform can be any type of transform. For example, the single transform can be a separable transform or an inseparable transform. The single transform can be a transform approximated from an inseparable transform. A single transform index (ST_idx in Figure 18) can be signaled when the single transform indicator is activated. Here, the single transform index can indicate the transform to be applied from among the available transform candidates.

[0205] Referring to Figure 18, the encoding device 100 determines whether CBF is 1 (S1805). If CBF is 1, the encoding device 100 determines whether STI coding is applied (S1810). If STI coding is applied, the encoding device 100 encodes the STI index (STI_Idx) (S1845) and codes the conversion coefficients (S1850). If STI coding is not applied, the encoding device 100 encodes a flag (EMT_CU_Flag) indicating whether EMT (or MTS) is applied at the CU level (S1815). Thereafter, the encoding device 100 codes the conversion coefficients (S1820). Thereafter, the encoding device 100 determines whether EMT is applied to the conversion unit (TU) (S1825). If EMT is applied to the TU, the encoding device 100 encodes the index of the primary conversion applied to the TU (EMT_TU Idx) (S1830). Thereafter, the encoding device 100 determines whether NSST is applicable (S1835). If NSST is applicable, the encoding device 100 encodes an index (NSST_Idx) that indicates the applicable NSST (S1840).

[0206] In one example, when the conditions for single transformation coding are met / activated (e.g., STI_coding == 1), a single transformation index (ST_Idx) can be implicitly induced without signaling. ST_Idx can be implicitly determined based on the block size and intra-prediction mode, where ST_Idx can indicate the transformation (or transformation kernel) to be applied to the current transformation block.

[0207] A single translation indicator can be activated if one or more of the following conditions are met (STI_coding == 1):

[0208] 1) The block size corresponds to a predetermined value, such as 4 or 8.

[0209] 2) Block width == Block height (square block)

[0210] 3) One of the predetermined intra-prediction modes, such as DC or Planner.

[0211] In another example, an STI coding flag can be signaled to indicate whether a single transformation is applicable. The STI coding flag can be signaled based on the STI coding value and CBF. For example, the STI coding flag can be signaled when the CBF is 1 and STI coding is activated. Furthermore, the STI coding flag can be conditionally signaled considering the block size, block shape (square or non-square blocks), or intra-prediction mode.

[0212] During coefficient coding, the acquired information can be used to determine ST_idx after coefficient coding. In one example, ST_idx can be implicitly determined based on the block size, intra-prediction mode, and the number of non-zero coefficients. In another example, ST_idx can be conditionally encoded / decoded based on the block size and / or block shape and / or intra-prediction mode and / or the number of non-zero coefficients. In yet another example, ST_idx signaling can be omitted depending on the distribution of non-zero coefficients (i.e., the location of the non-zero coefficients). In particular, if non-zero coefficients are found in a region other than the upper-left 4x4 area, ST_idx signaling can be omitted.

[0213] Figure 19 shows an example of an encoding flowchart using a unified transform indicator (UTI) as an embodiment to which the present invention is applied.

[0214] In embodiments of the present invention, a unified conversion indicator is introduced. The UTI encompasses both a primary conversion indicator and a secondary conversion indicator.

[0215] Referring to Figure 19, the encoding device 100 determines whether CBF is 1 (S1905). If CBF is 1, the encoding device 100 determines whether UTI coding is applied (S1910). If UTI coding is applied, the encoding device 100 encodes the UTI index (UTI_Idx) (S1945) and codes the conversion coefficients (S1950). If UTI coding is not applied, the encoding device 100 encodes a flag (EMT_CU_Flag) indicating whether EMT (or MTS) is applied at the CU level (S1915). Thereafter, the encoding device 100 encodes the conversion coefficients (S1920). Thereafter, the encoding device 100 determines whether EMT is applied to the conversion unit (TU) (S1925). If EMT is applied to TU, the encoding device 100 encodes the index of the primary transformation applied to TU (EMT_TU Idx) (S1930). Thereafter, the encoding device 100 determines whether NSST is applied (S1935). If NSST is applied, the encoding device 100 encodes the index indicating the NSST to be applied (NSST_Idx) (S1940).

[0216] UTI can be encoded for each predetermined unit (CTU or CU).

[0217] The UTI coding mode can depend on the following conditions.

[0218] 1) Block size

[0219] 2) Block form

[0220] 3) Intra Predictive Mode

[0221] The method for deriving / extracting core transformation indices from UTIs is predefined. The method for deriving / extracting secondary transformation indices from UTIs is also predefined.

[0222] The syntactic structure of a UTI is used selectively. A UTI can depend on the CU (or TU) size. For example, a smaller CU (TU) may have a relatively narrower range of UTI indices. For instance, if a predefined condition (e.g., block size is smaller than a predefined threshold) is met, a UTI can only point to core translation indices.

[0223] [Table 6]

[0224] In another example, if it is not indicated that a quadratic transformation should be used (for example, if the quadratic transformation index is 0 or the quadratic transformation has already been determined), the UTI index can be treated as a core transformation index. Similarly, if the core transformation index is known, the UTI index can be treated as a quadratic transformation index. In particular, a predetermined core transformation can be used, taking into account the intra-prediction mode and block size.

[0225] Figures 20a and 20b show another example of an encoding flowchart using a UTI as an embodiment to which the present invention is applied.

[0226] In another example, the conversion encoding structure uses UTI index coding, as shown in Figures 20a and 20b. Here, the UTI index can be encoded before or after coefficient encoding.

[0227] Referring to Figure 20a, the encoding device 100 checks whether CBF is 1 (S2005). If CBF is 1, the encoding device 100 codes the UTI index (UTI_Idx) (S2010) and codes the conversion coefficients (S2015).

[0228] Referring to Figure 20b, the encoding device 100 checks whether CBF is 1 (S2055). If CBF is 1, the encoding device 100 performs coding of the conversion coefficients (S2060) and codes the UTI index (UTI_Idx) (S2065).

[0229] In another embodiment of the present invention, data hiding and implicit coding methods for transformation indicators are introduced. Here, the transformation indicator includes ST_idx, UTI_idx, EMT_CU_Flag, EMT_TU_Flag, NSST_idx, and transformation-related indices used to indicate the transformation kernel. The aforementioned transformation indicators are not signaled, and the information can be inserted into (or extracted from) the coefficient encoding process. The coefficient encoding process may include the following parts:

[0230] - Last x position (Last_position_x), Last y position (Last_position_y)

[0231] - Group flag

[0232] - Significance map

[0233] - A flag indicating whether the value is greater than 1 (Greater_than_1_flag)

[0234] - A flag indicating whether the value is greater than 2 (Greater_than_2_flag)

[0235] - Remaining level coding

[0236] - Sign coding

[0237] For example, conversion indicator information can be inserted into one or more of the coefficient coding processes described above. The following can be considered together when inserting conversion indicator information:

[0238] - Sign coding patterns

[0239] - The absolute value of the remaining level

[0240] The number of flags indicating whether the value is greater than -1 (The number of Greater_than_1_flag)

[0241] - The values ​​of the last x-position and last y-position.

[0242] The aforementioned data concealment methods can be considered conditionally. For example, the data concealment method may depend on the number of non-zero coefficients.

[0243] In another example, NSST_idx and EMT_idx can be dependent. For instance, when EMT_CU_flag is 0 (or 1), NSST_idx may not be 0. In this case, NSST_idx-1 can signal instead of NSST_idx.

[0244] In another embodiment of the present invention, the mapping of NSST transformation sets based on an intra-predictive mode is shown in Table 7 below. As previously stated, the following description focuses on NSST as an example of an inseparable transformation, but other known terms for inseparable transformations (e.g., LFNST) may be used. For example, NSST Set and NSST index may be replaced with LFNST Set and LFNST index. Also, RST as described in this document may be replaced with RST or LFNST as an example of an inseparable transformation (e.g., LFNST) that uses a non-square transformation matrix having a reduced input length and / or reduced output length with a square inseparable transformation matrix applied to at least a portion of the transformation block (the upper left 4x4, 8x8 region, or the remaining region of an 8x8 block excluding the right-lower 4x4 region).

[0245] [Table 7]

[0246] NSST set numbers can be rearranged between 0 and 3, as shown in Table 8.

[0247] [Table 8]

[0248] In NSST translation sets, four translation sets are used (instead of 35) to reduce the memory space required.

[0249] Furthermore, for each set of transformations, a different number of transformation kernels are used, as follows:

[0250] Case A: Two available transformation kernels are used for each transformation set, and the NSST index range is 0 to 2. For example, when the NSST index is 0, the second-order transformation (second-order inverse transformation based on the decoder) may not be applied. When the NSST index is 1 or 2, the second-order transformation can be applied. The transformation set can include two transformation kernels, and an index of 1 or 2 can be mapped to the two transformation kernels.

[0251]

Table 9

[0252] Referring to Table 9, two transformation kernels are used for each of the non-separable transformation (NSST or LFNST) sets from set 0 to set 3.

[0253] Case B: Two available transformation kernels are used for the 0th transformation set, and one transformation kernel is used for each of the remaining transformation sets. The available NSST index for the 0th transformation set (DC, planner) is 0 to 2. However, the NSST index for the other modes (transformation sets 1, 2, and 3) is 0 to 1.

[0254]

Table 10

[0255] Referring to Table 10, two non-separable transformation kernels are set for the non-separable transformation (NSST) set corresponding to the 0th index, and one non-separable transformation kernel is set for each of the non-separable transformation (NSST) sets corresponding to the 1st, 2nd, and 3rd indices.

[0256] Case C: One transformation kernel is used for each transformation set, and the NSST index range is 0 to 1.

[0257]

Table 11

[0258] FIG. 21 shows an example of an encoding flowchart for performing conversion as an embodiment to which the present invention is applied.

[0259] The encoding device 100 performs a primary conversion on the residual block (S2105). The primary conversion can be called a core conversion. As an embodiment, the encoding device 100 can perform the primary conversion using the MTS described above. Further, the encoding device 100 can transfer an MTS index indicating a specific MTS to the decoding device 200 from among the MTS candidates. At this time, the MTS candidates can be configured based on the intra prediction mode of the current block.

[0260] The encoding device 100 determines whether to apply a secondary conversion (S2110). As an example, the encoding device 100 can determine whether to apply a secondary conversion based on the primary-converted residual transform coefficients. For example, the secondary conversion can be NSST or RST.

[0261] The encoding device 100 determines the secondary conversion (S2115). At this time, the encoding device 100 can determine the secondary conversion based on the NSST (or RST) transform set specified according to the intra prediction mode.

[0262] Also, as an example, the encoding device 100 can determine the area to which the secondary conversion is applied based on the size of the current block prior to the S2115 stage.

[0263] The encoding device 100 performs the secondary conversion using the secondary conversion determined at the S2115 stage (S2120).

[0264] FIG. 22 shows an example of a decoding flowchart for performing conversion as an embodiment to which the present invention is applied.

[0265] The decoding device 200 decides whether to apply the inverse quadratic transform (S2205). For example, the inverse quadratic transform may be NSST or RST. As an example, the decoding device 200 may decide whether to apply the inverse quadratic transform based on the quadratic transform flag received from the encoding device 100.

[0266] The decoding device 200 determines the quadratic inverse transform (S2210). At this time, the decoding device 200 can determine the quadratic inverse transform to be applied to the current block based on the NSST (or RST) transform set specified according to the intra prediction mode described above.

[0267] As an example, prior to step S2210, the decoding device 200 can determine the region to which the quadratic inverse transform is applied based on the size of the current block.

[0268] The decoding device 200 performs a quadratic inverse transform on the inversely quantized resistive block using the quadratic inverse transform determined in step S2210 (S2215).

[0269] The decoding device 200 performs a first-order inverse transformation on the quadratic inversely transformed residual block (S2220). The first-order inverse transformation can be called a core inverse transformation. In one embodiment, the decoding device 200 can perform the first-order inverse transformation using the MTS described above. Also, as an example, the decoding device 200 can decide whether the MTS is applied to the current block prior to step S2220. In this case, the decoding flowchart in Figure 22 may further include a step of deciding whether the MTS is applied.

[0270] For example, if an MTS is applied to the current block (i.e., cu_mts_flag = 1), the decoding device 200 can construct MTS candidates based on the intra-prediction mode of the current block. In this case, the decoding flowchart in Figure 22 may further include a step for constructing MTS candidates. The decoding device 200 can then determine the linear inverse transform to be applied to the current block using mts_idx, which indicates a specific MTS from among the constructed MTS candidates.

[0271] Figure 23 shows an example of a detailed block diagram of a conversion unit 120 within an encoding device 100, as an embodiment to which the present invention is applied.

[0272] An encoding device 100 to which an embodiment of the present invention is applied may include a primary conversion unit 2310, a secondary conversion applicability determination unit 2320, a secondary conversion determination unit 2330, and a secondary conversion unit 2340.

[0273] The primary transformation unit 2310 can perform a primary transformation on a residual block. The primary transformation can be called a core transformation. In one embodiment, the primary transformation unit 2310 can perform a primary transformation using the aforementioned MTS. The primary transformation unit 2310 can also transfer an MTS index indicating a specific MTS from among the MTS candidates to the decoding device 200. At this time, the MTS candidates can be configured based on the intra-prediction mode of the current block.

[0274] The secondary transformation applicability determination unit 2320 can determine whether or not to apply a secondary transformation. For example, the secondary transformation applicability determination unit 2320 can determine whether or not to apply a secondary transformation based on the transformation coefficients of the primary transformed residual block. For example, the secondary transformation could be NSST or RST.

[0275] The second conversion determination unit 2330 determines the second conversion. At this time, as described above, the second conversion determination unit 2330 can determine the second conversion based on the NSST (or RST) conversion set specified according to the intra prediction mode.

[0276] Also, as an example, the second conversion determination unit 2330 can also determine the area to which the second conversion is applied based on the size of the current block.

[0277] The second conversion unit 2340 can execute the second conversion using the determined second conversion.

[0278] FIG. 24 shows an example of a detailed block diagram of the inverse conversion unit 230 in the decoding apparatus 200 as an embodiment to which the present invention is applied.

[0279] The decoding apparatus 200 to which the present invention is applied includes a second inverse conversion applicability determination unit 2410, a second inverse conversion determination unit 2420, a second inverse conversion unit 2430, and a first inverse conversion unit 2440.

[0280] The second inverse conversion applicability determination unit 2410 can determine whether to apply the second inverse conversion. For example, the second inverse conversion can be NSST or RST. As an example, the second inverse conversion applicability determination unit 2410 can determine whether to apply the second inverse conversion based on the second conversion flag received from the encoding apparatus 100. As another example, the second inverse conversion applicability determination unit 2410 can also determine whether to apply the second inverse conversion based on the transform coefficients of the residual block.

[0281] The second inverse conversion determination unit 2420 can determine the second inverse conversion. At this time, the second inverse conversion determination unit 2420 can determine the second inverse conversion applied to the current block based on the NSST (or RST) conversion set specified according to the intra prediction mode.

[0282] As an example, the quadratic inverse transform determination unit 2420 can determine the region to which the quadratic inverse transform is applied based on the size of the current block.

[0283] As an example, the second-order inverse transform unit 2430 can perform a second-order inverse transform on an inversely quantized resistive block using the determined second-order inverse transform.

[0284] The primary inverse transformer 2440 can perform a primary inverse transform on a secondary inverse transformed resistive block. In one embodiment, the primary inverse transformer 2440 can perform a primary transform using the MTS described above. Also, as an example, the primary inverse transformer 2440 can determine whether the MTS is applied to the current block.

[0285] For example, if an MTS is applied to the current block (i.e., cu_mts_flag = 1), the first-order inverse transform unit 2440 can construct MTS candidates based on the intra-prediction mode of the current block. Then, from among the constructed MTS candidates, the first-order inverse transform unit 2440 can determine the first-order transform to be applied to the current block using mts_idx which indicates a specific MTS.

[0286] Figure 25 shows a flowchart for processing a video signal as an embodiment to which the present invention is applied. The flowchart in Figure 25 can be executed by a decoding device 200 or an inverse conversion unit 230.

[0287] First, the decoding device 200 can determine whether to apply the inverse inseparable transformation to the current block based on the inseparable transformation index and the width and height of the current block. For example, the decoding device 200 can decide to apply the inseparable transformation if the inseparable transformation index is not 0 and the width and height of the current block are both 4 or greater. If the inseparable transformation index is 0, or if the width or height of the current block is less than 4, the decoding device 200 can skip the inverse inseparable transformation and perform the inverse linear transformation.

[0288] In step S2505, the decoding device 200 determines an inseparable transformation set index that indicates which inseparable transformation set to be used for the inseparable transformation of the current block from among the predefined inseparable transformation sets, based on the intra-prediction mode of the current block. The inseparable transformation set index can be set to be assigned to one of four transformation sets, which are set according to the range of the intra-prediction mode, as shown in Table 7 or Table 8. That is, as shown in Table 7 or Table 8, if the intra-prediction mode is 0 to 1, the inseparable transformation set index is determined to the first index value; if the intra-prediction mode is 2 to 12 or 56 to 66, the inseparable transformation set index is determined to the second index value; if the intra-prediction mode is 13 to 23 or 45 to 55, the inseparable transformation set index is determined to the third index value; and if the intra-prediction mode is 24 to 44, the inseparable transformation set index can be determined to the fourth index value.

[0289] Here, a predefined set of inseparable transforms can each contain two transform kernels, as shown in Table 9. Alternatively, a predefined set of inseparable transforms can contain one or two transform kernels, as shown in Table 10 or Table 11.

[0290] In step S2510, the decoding device 200 determines the transformation kernel indicated by the inseparable transformation index of the current block from among the transformation kernels included in the inseparable transformation set indicated by the inseparable transformation set index, using the inseparable transformation matrix. For example, two inseparable transformation kernels can be set for each index value of the inseparable transformation set index, and the decoding device 200 can determine the inseparable transformation matrix from among the two transformation matrix kernels corresponding to the inseparable transformation set index, based on the transformation kernel indicated by the inseparable transformation index.

[0291] At step S2515, the decoding device 200 applies an inseparable transformation matrix to the upper left region of the current block, which is determined according to the width and height of the current block. For example, if both the width and height of the current block are 8 or greater, an inseparable transformation is applied to the 8x8 region in the upper left of the current block. If either the width or height of the current block is less than 8, an inseparable transformation can be applied to a 4x4 region of the current block. The size of the inseparable transformation can also be set to 8x8 or 4x4, corresponding to the region to which the inseparable transformation is applied.

[0292] Furthermore, the decoding device 200 can apply horizontal and vertical transformations to the current block to which a non-separated transformation has been applied. Here, the horizontal and vertical transformations can be determined based on the prediction mode applied to the current block and the MTS index for selecting the transformation matrix.

[0293] The following describes a method for applying a combination of primary and secondary transforms. That is, in embodiments of the present invention, a method for efficiently designing the transforms used in primary and secondary transforms is proposed. Here, the methods proposed in Figures 1 to 25 may also be applied, and related, redundant explanations will be omitted.

[0294] As mentioned above, the linear transformation refers to the transformation applied first to the residual block with respect to the encoder. When a quadratic transformation is applied, the encoder performs the quadratic transformation on the linearly transformed residual block. On the other hand, when a quadratic transformation is applied, the inverse quadratic transformation is performed before the inverse linear transformation with respect to the decoder. The decoder can induce the residual block by performing the inverse linear transformation on the inverse quadratic transformation of the transformation coefficient block.

[0295] Furthermore, as mentioned above, a non-separable transform may be used as the quadratic transform, and in order to maintain low complexity, it can be applied only to low-frequency coefficients in a specific region on the upper left. Such a quadratic transform applied to low-frequency coefficients may be called NSST (Non-Separable Secondary Transform), LFNST (low frequency non-separable transform), or RST (reduced secondary transform). The aforementioned primary transform may also be called a core transform.

[0296] In one embodiment of the present invention, the primary transformation candidate used for primary transformation and the secondary transformation kernel used for secondary transformation may be predefined in various combinations. In this specification, the primary transformation candidate used for primary transformation may be referred to as an MTS candidate, but is not limited to that name. For example, the primary transformation candidate may be a combination of transformation kernels (or transformation types) applied to the horizontal and vertical directions, and the transformation kernel may be any one of DCT2, DST7, and / or DCT8. In other words, the primary transformation candidate may be at least one combination of DCT2, DST7, and / or DCT8. Specific examples are described below.

[0297] - Combination A

[0298] In combination A, the primary transformation candidate and the secondary transformation kernel are defined by the intra-prediction mode, as shown in Table 12 below.

[0299] [Table 12]

[0300] Referring to Table 12, as an example (Case 1), if the intra-prediction mode is directional, two linear transformation candidates are used, and if it is not directional (e.g., DC, planner mode), four linear transformation candidates are used. Here, the quadratic transformation candidates can include two transformation kernels regardless of the directionality of the intra-prediction mode. That is, as mentioned above, multiple sets of quadratic transformation kernels are predefined by the intra-prediction mode, and each of these predefined sets of quadratic transformation kernels includes two transformation kernels.

[0301] Furthermore, as an example (Case 2), if the intra-prediction mode is directional, two linear transformation candidates are used; if it is not directional, four linear transformation candidates are used. Here, the quadratic transformation candidates may include one transformation kernel if the intra-prediction mode is directional, and two transformation kernels if it is not directional.

[0302] Furthermore, as an example (Case 3), if the intra-prediction mode has directionality, two linear transformation candidates are used; if it does not have directionality, four linear transformation candidates are used. Here, the quadratic transformation candidate can include one transformation kernel regardless of the directionality of the intra-prediction mode.

[0303] - Combination B

[0304] In combination B, the primary transformation candidate and the secondary transformation kernel are defined by the intra-prediction mode, as shown in Table 13 below.

[0305] [Table 13]

[0306] Referring to Table 13, as an example (Case 1), three linear transformation candidates are used regardless of the direction of the intra-prediction mode. Here, the quadratic transformation candidates can include two transformation kernels regardless of the direction of the intra-prediction mode. That is, as mentioned above, multiple sets of quadratic transformation kernels are predefined by the intra-prediction mode, and each of these predefined sets of quadratic transformation kernels can include two transformation kernels.

[0307] As another example (Case 2), three linear transformation candidates are used regardless of the directionality of the intra-prediction mode. Here, the quadratic transformation candidates may include one transformation kernel if the intra-prediction mode is directional, and two transformation kernels if it is not directional.

[0308] As an example (Case 3), three linear transformation candidates are used regardless of the direction of the intra-prediction mode. In this case, the quadratic transformation candidate may include one transformation kernel regardless of the direction of the intra-prediction mode.

[0309] - Combination C

[0310] In combination C, the primary transformation candidate and the secondary transformation kernel are defined by the intra-prediction mode, as shown in Table 14 below.

[0311] [Table 14]

[0312] Referring to Table 14, as an example (Case 1), if the intra-prediction mode is directional, two linear transformation candidates are used, and if it is not directional (e.g., DC, planner mode), three linear transformation candidates are used. Here, the quadratic transformation candidates can include two transformation kernels regardless of the directionality of the intra-prediction mode. That is, as mentioned above, multiple sets of quadratic transformation kernels are predefined by the intra-prediction mode, and each of these predefined sets of quadratic transformation kernels can include two transformation kernels.

[0313] Furthermore, as an example (Case 2), if the intra-prediction mode is directional, two linear transformation candidates are used; if it is not directional, three linear transformation candidates are used. Here, the quadratic transformation candidate may include one transformation kernel if the intra-prediction mode is directional, and may include two transformation kernels if it is not directional.

[0314] Furthermore, as an example (Case 3), if the intra-prediction mode is directional, two linear transformation candidates are used; if it is not directional, three linear transformation candidates are used. Here, the quadratic transformation candidate can include one transformation kernel regardless of the directionality of the intra-prediction mode.

[0315] The above explanation focused on cases where multiple linear transformation candidates are used. Below, we will explain the combination of linear and quadratic transformations using a fixed linear transformation candidate as an example.

[0316] - Combination D

[0317] In combination D, the primary transformation candidate and the secondary transformation kernel are defined by the intra-prediction mode, as shown in Table 15 below.

[0318] [Table 15]

[0319] Referring to Table 15, in this embodiment, one linear transformation candidate is fixedly used regardless of the intra-prediction mode. For example, the fixed linear transformation candidate may be at least one combination of DCT2, DST7, and / or DCT8.

[0320] As an example (Case 1), one linear transformation candidate is fixedly used regardless of the intra-prediction mode, and here, the quadratic transformation candidate can include two transformation kernels regardless of the direction of the intra-prediction mode. That is, as mentioned above, multiple sets of quadratic transformation kernels are predefined by the intra-prediction mode, and each of the predefined sets of quadratic transformation kernels can include two transformation kernels.

[0321] As an example (Case 2), one linear transformation candidate is fixedly used regardless of the intra-prediction mode, where the quadratic transformation candidate may include one transformation kernel if the intra-prediction mode is directional, and two transformation kernels if it is not directional.

[0322] As an example (Case 3), one linear transformation candidate is fixedly used regardless of the intra-prediction mode, and the quadratic transformation candidate can include one transformation kernel regardless of the direction of the intra-prediction mode.

[0323] - Combination E

[0324] In combination E, the primary transformation candidate and the secondary transformation kernel are defined by the intra-prediction mode, as shown in Table 16 below.

[0325] [Table 16]

[0326] Referring to Table 16, a secondary transformation is defined only when DCT2 is applied as the primary transformation. In other words, if MTS is not applied (i.e., DCT2 is applied as the primary transformation), a secondary transformation is applicable. As explained in Figure 10, in this specification, cases where MTS is applied and cases where it is not are described separately, but the present invention is not limited to such expressions. For example, whether or not to apply MTS may mean the same thing as whether or not to use a different transformation type (or transformation kernel) other than a predefined specific transformation type (which may be called a basic transformation type, default transformation type, etc.). If MTS is applied, a different transformation type other than the basic transformation type (for example, a transformation type consisting of one or more combinations of multiple transformation types) may be used for the transformation, and if MTS is not applied, the basic transformation type may be used for the transformation. In one embodiment, the basic transformation type may be set (or defined) in DCT2.

[0327] As an example (Case 1), when DCT2 is applied to a linear transformation, a quadratic transformation is applicable, where the quadratic transformation candidate includes two transformation kernels regardless of the direction of the intra-prediction mode. That is, as mentioned above, multiple sets of quadratic transformation kernels are predefined by the intra-prediction mode, and each of these predefined sets of quadratic transformation kernels can include two transformation kernels.

[0328] Furthermore, as an example (Case 2), when DCT2 is applied to a linear transformation, a quadratic transformation can be applied. Here, the quadratic transformation candidate may include one transformation kernel if the intra-prediction mode is directional, and two transformation kernels if it is not directional.

[0329] Furthermore, as an example (Case 3), when DCT2 is applied to a linear transformation, a quadratic transformation can be applied, where the quadratic transformation candidate can include a single transformation kernel regardless of the direction of the intra-prediction mode.

[0330] Figure 26 is a flowchart illustrating a video signal conversion method according to an embodiment to which the present invention is applied.

[0331] Referring to Figure 26, the explanation will focus on the decoder for convenience, but the present invention is not limited to this, and the conversion method for video signals according to this embodiment can be applied substantially the same way to an encoder. The flowchart in Figure 26 is performed by the decoding device 200 or the inverse conversion unit 230.

[0332] The decoding device 200 parses a first syntax element that indicates the primary transform kernel to be applied to the primary transform of the current block (S2601).

[0333] The decoding device 200 determines whether a secondary transform can be applied to the current block based on the first syntax element (S2602).

[0334] If a secondary transformation can be applied to the current block, the decoding device 200 parses a second syntax element indicating a secondary transform kernel to be applied to the secondary transformation of the current block (S2603).

[0335] The decoding device 200 induces the block that has been quadratically inverted by performing a quadratically inverse transform on a specific region in the upper left of the current block using the quadratically transform kernel indicated by the second syntax element (S2604).

[0336] The decoding device 200 induces the residual block of the current block by performing a linear inverse transformation on the quadratic inversely transformed block using the linear transformation kernel indicated by the first syntax element (S2605).

[0337] As described above, step S2602 is performed by determining that a secondary transformation can be applied to the current block if the first syntax element indicates a predefined first transformation kernel. Here, the first transformation kernel is defined in DCT2.

[0338] Furthermore, as described above, the decoding device 200 can determine the set of secondary transform kernels to be used for the secondary transformation of the current block from a predefined set of secondary transform kernels based on the intra-prediction mode of the current block. The second syntax element can then indicate the secondary transform kernel to be applied to the secondary transformation of the current block within the determined set of secondary transform kernels.

[0339] Furthermore, as mentioned above, each of the predefined sets of secondary conversion kernels can contain two conversion kernels.

[0340] An example of a syntax structure in which an MTS (Multiple Transform Set) is used in one embodiment of the present invention will be described.

[0341] As an example, Table 17 below shows an example of the syntax structure of a sequence parameter set.

[0342] [Table 17]

[0343] Referring to Table 17, the availability of MTS according to the embodiment of the present invention can be signaled via sequence parameter set syntax. Here, sps_mts_intra_enabled_flag indicates whether an MTS flag or MTS index exists in the lower-level syntax for the intracoding unit (e.g., residual coding syntax, conversion unit syntax). And sps_mts_inter_enabled_flag indicates whether an MTS flag or MTS index exists in the lower-level syntax for the intercoding unit.

[0344] As another example, Table 18 below shows an example of a conversion unit syntax structure.

[0345] [Table 18]

[0346] Referring to Table 18, cu_mts_flag indicates whether MTS is applied to the residual samples of the Luma conversion block. For example, if cu_mts_flag=0, MTS is not applied to the residual samples of the Luma conversion block, and if cu_mts_flag=1, MTS is applied to the residual samples of the Luma conversion block.

[0347] As mentioned above, in the present invention, cases in which MTS is applied and cases in which it is not applied are described separately based on the MTS flag, but the present invention is not limited to such expressions. For example, whether or not MTS is applicable may have the same meaning as whether or not to use a conversion type (or conversion kernel) other than a predefined specific conversion type (which may be called a basic conversion type, default conversion type, etc.). If MTS is applied, a conversion type other than the basic conversion type (for example, one or more combinations of conversion types from among several conversion types) may be used for conversion, and if MTS is not applied, the basic conversion type may be used for conversion. In one embodiment, the basic conversion type may be set (or defined) in DCT2.

[0348] As an example, an MTS flag syntax indicating whether or not MTS is applied to the current conversion block, and an MTS index syntax indicating the conversion type applied to the current block if MTS is applied, can be individually sent from the encoder to the decoder. As another example, a syntax (e.g., an MTS index) including whether or not MTS is applied to the current conversion block and, if MTS is applied, all the conversion types applied to the current block can also be sent from the encoder to the decoder. That is, in the latter embodiment, a syntax (or syntax element) indicating the conversion type applied to the current conversion block (or unit) within the overall conversion type group (or conversion type set) including the basic conversion types described above can be sent from the encoder to the decoder.

[0349] Therefore, despite that expression, the syntax indicating the type of transformation currently applied to the transformation block (MTS index) may include information regarding MTS applicability. In other words, in the latter embodiment, only the MTS index may be signaled without an MTS flag, in which case it can be interpreted that the MTS includes DCT2, but in the present invention, the application of DCT2 can be described as not applying the MTS, and nevertheless, the technical scope of the MTS is not limited to that definition.

[0350] As another example, Table 19 below shows an example of a residual unit syntax structure.

[0351] [Table 19-1]

[0352] [Table 19-2]

[0353] [Table 19-3]

[0354] [Table 19-4]

[0355] [Table 19-5]

[0356] Referring to Table 19, the transform_skip_flag and / or mts_idx syntax (or syntax element) can be signaled via residual syntax. However, this is just one example, and the invention is not limited thereto. For example, the transform_skip_flag and / or mts_idx syntax can also be signaled via transform unit syntax.

[0357] The following describes specific embodiments of the secondary transform matrix that can be used for the aforementioned quadratic transformation. As mentioned above, the aforementioned quadratic transformation can be referred to as a non-separable secondary transform (NSST), LFNST (low-frequency non-separable transform), or RST (reduced secondary transform).

[0358] As described above, in embodiments of the present invention, four conversion sets (or quadratic conversion sets) may be used to improve memory efficiency when applying quadratic conversion. In one embodiment, the encoder / decoder may assign indices 0, 1, 2, and 3 to each of the four conversion sets.

[0359] Furthermore, as mentioned above, each transformation set may have a predefined number of transformation kernels. In one embodiment, four transformation sets used for quadratic transformations can be predefined in the encoder and decoder, and each transformation set may include one or two transformation matrices (or transformation types, transformation kernels).

[0360] Table 20 below shows an example of a transformation that can be applied to an 8x8 area.

[0361] [Table 20-1]

[0362] [Table 20-2]

[0363] [Table 20-3]

[0364] [Table 20-4]

[0365] [Table 20-5]

[0366] [Table 20-6]

[0367] [Table 20-7]

[0368] Refer to Table 20 for an example where the coefficients of the transformation matrix are multiplied by 128 as a scaling value. In Table 20, in the g_aiNsst8×8[4][2]

[16]

[64] array, the first input [4] represents the number of transformation sets (where each transformation set can be divided into indices 0, 1, 2, and 3), the second input [2] represents the number of transformation matrices that make up each transformation set, and the third and fourth inputs

[16]

[64] represent the rows and columns of a 16×64 RST (Reduced Secondary Transform).

[0369] Table 20 assumes that a transformation set contains two transformation matrices, but if it contains only one transformation matrix, it is possible to configure Table 20 to use transformation matrices in a specific order for each transformation set. For example, if a transformation set contains one transformation matrix, the encoder / decoder can use the first or second transformation matrix that is predefined within each transformation set in Table 20.

[0370] When applying the RST in Table 20, the encoder / decoder can be configured (or defined, set) to output 16 transformation coefficients, or it can be configured to output only m transformation coefficients by applying only the m×64 portion of the 16×64 matrix. For example, by setting m=8, the encoder / decoder can be configured to output only 8 transformation coefficients using only the 8×64 matrix from the top. In this way, the computational complexity can be halved by applying a reduced quadratic transformation. In one embodiment, the encoder / decoder can apply an 8×64 matrix to an 8×8 transform unit (TU) to reduce the computational complexity in the worst case.

[0371] Table 21 below shows an example of a transformation that can be applied to a 4x4 area.

[0372] [Table 21-1]

[0373] [Table 21-2]

[0374] [Table 21-3]

[0375] Refer to Table 21 for an example where the coefficients of the transformation matrix are multiplied by 128 as a scaling value. In Table 21, in the g_aiNsst4×4[4][2]

[16]

[16] array, the first input [4] represents the number of transformation sets (where each transformation set can be divided into indices 0, 1, 2, and 3), the second input [2] represents the number of transformation matrices that make up each transformation set, and the third and fourth inputs

[16]

[16] represent the rows and columns of a 16×16 RST (Reduced Secondary Transform).

[0376] Table 21 assumes that a transformation set contains two transformation matrices, but if it contains only one transformation matrix, it is possible to configure Table 21 to use transformation matrices in a specific order for each transformation set. For example, if a transformation set contains one transformation matrix, the encoder / decoder can use the first or second transformation matrix that is predefined within each transformation set in Table 21.

[0377] When applying the RST in Table 21, the encoder / decoder can be configured (or defined, set) to output 16 transformation coefficients, or it can be configured to output only m transformation coefficients by applying only the m×16 portion of the 16×16 matrix. For example, by setting m=8, the encoder / decoder can be configured to output only 8 transformation coefficients using only the 8×16 matrix from the top. In this way, the computational complexity can be halved by applying a reduced quadratic transformation. In one embodiment, the encoder / decoder can apply an 8×64 matrix to an 8×8 transform unit (TU) to reduce the computational complexity in the worst case.

[0378] In one embodiment, the quadratic transformation, as illustrated in Tables 20 and 21, can be applied to the upper left 4x4, 4x8, and 8x4 regions (i.e., TUs) according to predefined conditions, or it can be applied only to the upper left 4x4 region. In the case of the 4x8 TU and 8x4 TU, the encoder / decoder can divide it into two 4x4 regions and apply the specified transformation to each divided region. If the quadratic transformation is defined to be applied only to the 4x4 region, then only the transformations defined in Table 21 can be applied (or used).

[0379] On the other hand, while the coefficients of the transformation matrix were defined in Tables 20 and 21 assuming a scaling value of 128, the present invention is not limited thereto. For example, Tables 20 and 21 can also be defined as shown in Tables 22 and 23 below, respectively, by setting the scaling value to 256.

[0380] [Table 22-1]

[0381] [Table 22-2]

[0382] [Table 22-3]

[0383] [Table 22-4]

[0384] [Table 22-5]

[0385] [Table 22-6]

[0386] [Table 22-7]

[0387] [Table 22-8]

[0388] [Table 23-1]

[0389] [Table 23-2]

[0390] [Table 23-3]

[0391] As described above, in embodiments of the present invention, four conversion sets (or quadratic conversion sets) may be used to improve memory efficiency when applying quadratic conversion. In one embodiment, the encoder / decoder may assign indices 0, 1, 2, and 3 to each of the four conversion sets.

[0392] Furthermore, as mentioned above, each transformation set can contain a predefined number of transformation kernels. In one embodiment, four transformation sets used for quadratic transformations can be predefined in the encoder and decoder, and each transformation set can contain one or two transformation matrices (or transformation types, transformation kernels).

[0393] The following sections present various examples of different quadratic transformation sets and transformation matrices (or transformation types, transformation kernels) applicable to quadratic transformations. While various transformation matrices different from those in Tables 20 to 23 mentioned above can be defined in detail, in this embodiment, for the sake of explanation, we will focus on generalized quadratic transformation set configurations and non-directional modes (e.g., DC mode, planar mode) as examples.

[0394] First, we will describe specific examples of quadratic transformations that can be applied to a 4x4 region. Of the following examples of quadratic transformation sets available for quadratic transformations, the first and fourth examples can be applied to embodiments in which each transformation set consists of two transformation matrices. The transformation matrices of the second and third examples can be applied to embodiments in which each transformation set consists of one transformation matrix.

[0395] In particular, the first example can be applied to the above-mentioned combination D and Case 1 of the embodiment described in Table 15, and furthermore, it can be applied to the combination A and Case 1 of the embodiment described in Table 12, the combination B and Case 1 of the embodiment described in Table 13, the combination C and Case 1 of the embodiment described in Table 14, or the combination E and Case 1 of the embodiment described in Table 16.

[0396] Furthermore, the second example transformation sequence (i.e., transformation set) can be applied to combination D and Case 3 of the embodiment described in Table 15, and moreover, to combination A and Case 3 of the embodiment described in Table 12, combination B and Case 3 of the embodiment described in Table 13, combination C and Case 3 of the embodiment described in Table 14, or combination E and Case 3 of the embodiment described in Table 16.

[0397] The combinations A, B, C, D, and E mentioned above only deal with cases where the number of MTS candidates is three or less, but it is also possible to configure the system to apply all four MTS candidates to all intra-prediction modes in a linear transformation. The first through fourth examples below can all be used when applying four MTS candidates, and in particular, the transformation sequence in the fourth example can be further adapted when applying four MTS candidates.

[0398] The transformation sequences in the following examples, from the 5th to the 7th, apply when 35 transformation sets are applied. They can be applied when mapping the transformation sets to each intra-prediction mode as shown in Table 24.

[0399] [Table 24]

[0400] In Table 24, the NSST set index represents the transformed set index. The mapping method in Table 24 can also be applied to combinations A through E described above. That is, it can be applied to the fifth through eighth examples in the same way as the method described above for each combination.

[0401] The fifth and eighth exemplary transformation sequences can be applied to embodiments in which each transformation set consists of two transformation matrices, and the sixth and seventh exemplary transformation sequences can be applied to embodiments in which each transformation set consists of one transformation matrix.

[0402] In particular, the fifth example can be applied to the above-mentioned combination D and Case 1 of the embodiment described in Table 15, and furthermore, it can be applied to the combination A and Case 1 of the embodiment described in Table 12, the combination B and Case 1 of the embodiment described in Table 13, the combination C and Case 1 of the embodiment described in Table 14, or the combination E and Case 1 of the embodiment described in Table 16.

[0403] Furthermore, the sixth and seventh example transformation sequences (i.e., transformation sets) can be applied to combination D and Case 3 of the embodiment described in Table 15, and moreover, to combination A and Case 3 of the embodiment described in Table 12, combination B and Case 3 of the embodiment described in Table 13, combination C and Case 3 of the embodiment described in Table 14, or combination E and Case 3 of the embodiment described in Table 16.

[0404] The combinations A, B, C, D, and E mentioned above only deal with cases where the number of MTS candidates is three or less, but it is also possible to configure the system to apply all four MTS candidates to all intra-prediction modes in a linear transformation. The following examples 5 through 8 can all be used when applying four MTS candidates, and in particular, the transformation sequence in example 8 can be further adapted when applying four MTS candidates.

[0405] Of the following example transformation arrays from the first to the eighth, all transformation examples that can be applied to a 4x4 region correspond to transformation matrices multiplied by a scaling value of 128. The following example transformation arrays can commonly be represented by the g_aiNsst4x4[N1][N2]

[16]

[16] array, where N1 represents the number of transformation sets, which can be 4 or 35 and can be divided into indices 0, 1, ..., N1-1. N2 represents the number of transformation matrices that make up each transformation set (i.e., 1 or 2), and

[16]

[16] represents a 16x16 transformation matrix.

[0406] Similarly, as illustrated in the following examples, if a set of transformations consists of a single transform, it is possible to configure each transformation set to use transformation matrices in a specific order. For example, if a set of transformations contains a single transformation matrix, the encoder / decoder can use a predefined transformation matrix within each transformation set, i.e., the first or second transformation matrix.

[0407] To reduce the worst-case computational complexity, the encoder / decoder can apply an 8x16 matrix to a 4x4 TU. The transformations that can be applied to the 4x4 region in the example below can be applied to 4x4 TU, 4xM TU, and Mx4 TU (M>4). When applied to 4xM TU and Mx4 TU, the specified transformation can be applied to each of the 4x4 regions, or it can be applied only to the upper left 4x8 or 8x4 region. It can also be applied only to the upper left 4x4 region.

[0408] In one embodiment, the following method may be applied to reduce the computational complexity in the worst case.

[0409] For example, an encoder / decoder can apply a transformation array (or transformation matrix, transformation kernel) (e.g., a 16x64 matrix) to the top-left 8x8 region of a block with width W and height H, if W >= 8 and H >= 8. If W = 8 and H = 8, the encoder / decoder can apply only the 8x64 portion of the 16x64 matrix. In this case, the input for the quadratic transformation can be generated with 8 transformation coefficients, and the remaining coefficients in that region can be considered 0.

[0410] Furthermore, for example, for a block with width W and height H, if one of W or H is less than 8 (i.e., 4), the encoder / decoder can apply a transformation array that can be applied to a 4x4 area. If W=4 and H=4, the encoder / decoder can apply only the 8x16 portion of the 16x16 matrix. In this case, the input for the quadratic transformation can be generated with 8 transformation coefficients, and the remaining coefficients in that area can be considered 0.

[0411] In one embodiment, if (W, H) = (4, 8) or (8, 4), the encoder / decoder can apply the quadratic transformation only to the upper left 4x4 region. If W or H is greater than 8, the encoder / decoder applies the quadratic transformation only to the two upper left 4x4 blocks. That is, the encoder / decoder can apply the specified transformation matrix to two 4x4 blocks, at most up to the upper left 4x8 or 8x4 region.

[0412] First example

[0413] The first example can be defined as shown in Table 25 below. Four sets of transformations can be defined, in which case each set of transformations can consist of two transformation matrices.

[0414] [Table 25]

[0415] Second example

[0416] A second example can be defined as shown in Table 26 below. Four sets of transformations can be defined, in which case each set of transformations can consist of one transformation matrix.

[0417] [Table 26]

[0418] Third example

[0419] A third example can be defined as shown in Table 27 below. Four sets of transformations can be defined, in which case each set of transformations can consist of one transformation matrix.

[0420] [Table 27]

[0421] Fourth example

[0422] A fourth example can be defined as shown in Table 28 below. Four sets of transformations can be defined, in which case each set of transformations can consist of two transformation matrices.

[0423] [Table 28]

[0424] Fifth example

[0425] The fifth example can be defined as shown in Table 29 below. 35 sets of transformations can be defined, in which case each set of transformations can consist of two transformation matrices.

[0426] [Table 29]

[0427] Sixth example

[0428] The sixth example can be defined as shown in Table 30 below. 35 sets of transformations can be defined, and each set of transformations can consist of one transformation matrix.

[0429] [Table 30]

[0430] Seventh example

[0431] The seventh example can be defined as shown in Table 31 below. 35 sets of transformations can be defined, and each set of transformations can consist of one transformation matrix.

[0432] [Table 31]

[0433] Eighth example

[0434] The eighth example can be defined as shown in Table 32 below. 35 sets of transformations can be defined, in which case each set of transformations can consist of two transformation matrices.

[0435] [Table 32]

[0436] The following describes examples of quadratic transformations that can be applied to an 8x8 region. Of the following examples of quadratic transformation sets available for quadratic transformations, the 9th and 12th examples can be applied to embodiments in which each transformation set consists of two transformation matrices. The transformation matrices of the 10th and 11th examples can be applied to embodiments in which each transformation set consists of one transformation matrix.

[0437] In particular, the ninth example can be applied to the above-mentioned combination D and Case 1 of the embodiment described in Table 15, and furthermore, it can be applied to the combination A and Case 1 of the embodiment described in Table 12, the combination B and Case 1 of the embodiment described in Table 13, the combination C and Case 1 of the embodiment described in Table 14, or the combination E and Case 1 of the embodiment described in Table 16.

[0438] Furthermore, the transformation sequence (i.e., transformation set) of the tenth example can be applied to combination D and Case 3 of the embodiment described in Table 15, and moreover, it can be applied to combination A and Case 3 of the embodiment described in Table 12, combination B and Case 3 of the embodiment described in Table 13, combination C and Case 3 of the embodiment described in Table 14, or combination E and Case 3 of the embodiment described in Table 16.

[0439] The combinations A, B, C, D, and E mentioned above only deal with cases where the number of MTS candidates is three or less, but it is also possible to configure the system to apply all four MTS candidates to all intra-prediction modes in a linear transformation. The following examples 9 through 12 can all be used when applying four MTS candidates, and in particular, the transformation sequence in example 12 can be further adapted when applying four MTS candidates.

[0440] The transformation sequences in the following examples 13 through 16 apply when applying 35 transformation sets. They can be applied when mapping the transformation sets to each intra-prediction mode as shown in Table 24 above.

[0441] In Table 24, the NSST set index represents the transformed set index. The mapping method in Table 24 can also be applied to combinations A through E described above. That is, it can be applied to the 13th through 16th examples in the same way as the method described above for each combination.

[0442] The 13th and 16th exemplary transformation sequences can be applied to embodiments in which each transformation set consists of two transformation matrices, and the 14th and 15th exemplary transformation sequences can be applied to embodiments in which each transformation set consists of one transformation matrix.

[0443] In particular, the 13th example can be applied to the above-mentioned combination D and Case 1 of the embodiment described in Table 15, and furthermore, it can be applied to the combination A and Case 1 of the embodiment described in Table 12, the combination B and Case 1 of the embodiment described in Table 13, the combination C and Case 1 of the embodiment described in Table 14, or the combination E and Case 1 of the embodiment described in Table 16.

[0444] Furthermore, the transformation sequences (i.e., transformation sets) of the 14th and 15th examples can be applied to the above-mentioned combination D and Case 3 of the embodiment described in Table 15, and moreover, they can be applied to combination A and Case 3 of the embodiment described in Table 12, combination B and Case 3 of the embodiment described in Table 13, combination C and Case 3 of the embodiment described in Table 14, or combination E and Case 3 of the embodiment described in Table 16.

[0445] The combinations A, B, C, D, and E mentioned above only deal with cases where the number of MTS candidates is three or less, but it is also possible to configure the system to apply all four MTS candidates to all intra-prediction modes in a linear transformation. The following examples 13 through 16 can all be used when applying four MTS candidates, and in particular, the transformation sequence in example 8 can be further adapted when applying four MTS candidates.

[0446] Of the following example transformation arrays from the 8th to the 16th, all transformation examples that can be applied to an 8x8 region correspond to transformation matrices multiplied by a scaling value of 128. The following example transformation arrays can commonly be represented by the g_aiNsst8×8[N1][N2]

[16]

[64] array, where N1 represents the number of transformation sets, which can be 4 or 35 and divided into indices 0, 1, ..., N1-1. N2 represents the number of transformation matrices that make up each transformation set (i.e., 1 or 2), and

[16]

[64] represents the 16×64 Reduced Secondary Transform (RST).

[0447] Similarly, as illustrated in the following examples, if a set of transformations consists of a single transform, it is possible to configure each transformation set to use transformation matrices in a specific order. For example, if a set of transformations contains a single transformation matrix, the encoder / decoder can use a predefined transformation matrix within each transformation set, i.e., the first or second transformation matrix.

[0448] Applying the RST in question will output 16 transformation coefficients, but by applying only the m×64 portion of the 16×64 matrix, it is possible to configure it so that only m transformation coefficients are output. For example, by setting m=8 and applying only the 8×64 matrix from the top, the computational complexity can be halved.

[0449] Ninth example

[0450] The ninth example can be defined as shown in Table 33 below. Four sets of transformations can be defined, in which case each set of transformations can consist of two transformation matrices.

[0451] [Table 33-1]

[0452] [Table 33-2]

[0453] [Table 33-3]

[0454] Tenth example

[0455] The tenth example can be defined as shown in Table 34 below. Four sets of transformations can be defined, in which case each set of transformations can consist of one transformation matrix.

[0456] [Table 34]

[0457] Example 11

[0458] The eleventh example can be defined as shown in Table 35 below. Four sets of transformations can be defined, in which case each set of transformations can consist of one transformation matrix.

[0459] [Table 35]

[0460] Example 12

[0461] The twelfth example can be defined as shown in Table 36 below. Four sets of transformations can be defined, in which case each set of transformations can consist of two transformation matrices.

[0462] [Table 36-1]

[0463] [Table 36-2]

[0464] Example 13

[0465] The 13th example can be defined as shown in Table 37 below. 35 sets of transformations can be defined, in which case each set of transformations can consist of two transformation matrices.

[0466] [Table 37-1]

[0467] [Table 37-2]

[0468] 14th example

[0469] The 14th example can be defined as shown in Table 38 below. 35 sets of transformations can be defined, and each set of transformations can consist of one transformation matrix.

[0470] [Table 38]

[0471] Example 15

[0472] The 15th example can be defined as shown in Table 39 below. 35 sets of transformations can be defined, and each set of transformations can consist of one transformation matrix.

[0473] [Table 39]

[0474] Example 16

[0475] The 16th example can be defined as shown in Table 40 below. 35 sets of transformations can be defined, in which case each set of transformations can consist of two transformation matrices.

[0476] [Table 40-1]

[0477] [Table 40-2]

[0478] Figure 27 is a flowchart illustrating a video signal conversion method according to an embodiment to which the present invention is applied.

[0479] As shown in Figure 27, for the sake of explanation, the decoder will be the main focus of the explanation, but the present invention is not limited to this, and the method for converting video signals according to this embodiment can be substantially applied to encoders as well. The flowchart in Figure 27 can be performed by the decoding device 200 or the inverse conversion unit 230.

[0480] Based on the intra-prediction mode of the current block, the decoding device 200 determines which of the predefined secondary transform sets is to be applied to the current block (S2701).

[0481] The decoding device 200 obtains a first syntax element that indicates the secondary transform matrix to be applied to the current block within the determined set of secondary transforms (S2702).

[0482] The decoding device 200 uses the quadratic transformation matrix identified by the first syntax element to perform a quadratic inverse transformation on the upper left region of the current block, thereby obtaining a quadratic inversely transformed block (S2703).

[0483] The decoding device 200 derives the residual block of the current block by performing a linear inverse transformation on the quadratic inversely transformed block using the primary transform matrix of the current block (S2704).

[0484] As mentioned above, each of the predefined sets of quadratic transformations can contain two quadratic transformation matrices.

[0485] As described above, step S2704 may further include a step of determining the input and output lengths of the inverse quadratic transform based on the width and height of the current block. As described above, if the height and width of the current block are each 4, the input length of the non-separable transform can be determined to be 8 and the output length to be 16.

[0486] As described above, the decoding device 200 can parse a second syntax element that indicates the linear transformation matrix to be applied to the linear transformation of the current block. Based on the second syntax element, the decoding device 200 can then determine whether or not a quadratic transformation can be applied to the current block.

[0487] As described above, the step of determining whether the quadratic transformation can be applied can be performed by determining that the quadratic transformation can be applied to the current block if the second syntax element indicates a predefined specific transformation type.

[0488] As mentioned above, the aforementioned predefined specific conversion type can be defined as DCT2.

[0489] Figure 28 shows an example block diagram of an apparatus for processing video signals as an embodiment to which the present invention is applied. The video signal processing apparatus in Figure 28 can correspond to the encoding apparatus in Figure 1 or the decoding apparatus in Figure 2.

[0490] The video processing device 2800, which processes video signals, includes a memory 2820 for storing video signals and a processor 2810 that processes video signals while being coupled with the memory.

[0491] The processor 2810 according to an embodiment of the present invention can consist of at least one processing circuit for processing video signals, and can process video signals by executing commands for encoding or decoding video signals. In other words, the processor 2810 can encode original video data or decode encoded video signals by executing the encoding or decoding methods described above.

[0492] Furthermore, the processing methods to which the present invention is applied can be produced in the form of programs executed on a computer and stored on a computer-readable storage medium. Multimedia data having the data structure according to the present invention can also be stored on a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices on which computer-readable data is stored. The computer-readable recording medium can include, for example, Blu-ray discs (BDs), Universal Serial Bus (USB), ROMs, PROMs, EPROMs, EEPROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer-readable recording medium also includes media implemented in the form of carrier waves (e.g., transmission over the Internet). Furthermore, bitstreams generated by encoding methods can be stored on computer-readable recording media or transmitted over wireless communication networks.

[0493] Furthermore, embodiments of the present invention can be realized as a computer program product using program code, and the program code can be executed on a computer according to embodiments of the present invention. The program code can be stored on a computer-readable carrier.

[0494] As described above, the embodiments described in the present invention can be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in each figure can be implemented and executed on a computer, processor, microprocessor, controller, or chip.

[0495] Furthermore, decoders and encoders to which the present invention applies can include real-time communication devices such as multimedia broadcasting transceivers, mobile communication terminals, home cinema video equipment, digital cinema video equipment, surveillance cameras, video interaction devices, and video communication devices, as well as mobile streaming devices, storage media, CAM coders, video-on-demand (VoD) service providers, OTT video (Over the top video) devices, internet streaming service providers, 3D video devices, video phones, video equipment, and medical video equipment, and are used to process video signals or data signals. For example, OTT video (Over the top video) devices can include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, and DVRs (Digital Video Recorders).

[0496] Furthermore, the processing methods to which the present invention is applied can be produced in the form of programs executed on a computer and stored on a computer-readable storage medium. Multimedia data having the data structure according to the present invention can also be stored on a computer-readable storage medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices on which computer-readable data is stored. The computer-readable recording medium can include, for example, Blu-ray discs (BDs), Universal Serial Bus (USB), ROMs, PROMs, EPROMs, EEPROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer-readable recording medium also includes media implemented in the form of carrier waves (e.g., transmission over the Internet). Furthermore, bitstreams generated by encoding methods can be stored on computer-readable recording media or transmitted over wireless communication networks.

[0497] Furthermore, embodiments of the present invention can be realized as a computer program product using program code, and the program code can be executed on a computer according to embodiments of the present invention. The program code can be stored on a computer-readable carrier.

[0498] The embodiments described above are combinations of the components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless otherwise explicitly mentioned. Each component or feature can be implemented in a form that is not combined with other components or features. It is also possible to combine some components and / or features to constitute embodiments of the present invention. The order of operations described in the embodiments of the present invention can be changed. Some components or features of any embodiment may be included in other embodiments or may be replaced by components or features corresponding to other embodiments. It is obvious that claims that are not explicitly referenced in the claims may be combined to constitute embodiments or may be included as new claims by amendments after filing.

[0499] Embodiments of the present invention can be realized by various means, such as hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention can be realized by one or more ASICs (application-specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, etc.

[0500] In the case of implementation using firmware or software, one embodiment of the present invention can be implemented in the form of a module, procedure, function, etc., that performs the functions or operations described above. The software code can be stored in memory and driven by a processor. The memory can be located inside or outside the processor and can send and receive data with the processor by various already known means.

[0501] It will be obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential features of the invention. Therefore, the above-mentioned detailed description should not be constrained in any way restrictively, but should be considered illustrative. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention. [Industrial applicability]

[0502] The preferred embodiments of the present invention described above are disclosed for illustrative purposes only, and those skilled in the art can improve, modify, substitute, or add various other embodiments within the technical concept and technical scope of the present invention disclosed in the appended claims.

Claims

1. In a method for decoding a video signal, A step of obtaining a first syntax element that identifies which quadratic transformation matrix is ​​currently applied to the block within the set of quadratic transformations, The steps include: performing a quadratic inverse transform on the current block based on the quadratic transformation matrix identified by the first syntax element to generate a quadratic inverse transformed block; The step of generating a residual block of the current block by performing a linear transformation on the quadratically inversely transformed block based on the linear transformation matrix of the current block, A method in which, based on the fact that the width and height of the current block are 4, the input length of the inverse quadratic transform is determined to be 8, and based on the fact that the width and height of the current block are 8, the input length of the inverse quadratic transform is determined to be 8.

2. The method according to claim 1, wherein the quadratic transformation set includes two quadratic transformation matrices.

3. The steps include parsing a second syntax element which includes information about the linear transformation matrix applied to generate the residual block of the current block, The method according to claim 1, further comprising the step of determining whether the quadratic inverse transform is applicable to the current block based on the second syntax element.

4. The method according to claim 3, wherein it is determined that the quadratic inverse transform is applicable to the current block based on a second syntax element which includes information about a predetermined specific type of transform.

5. The method according to claim 4, wherein the predetermined specific conversion type is defined as DCT2.

6. In a method for encoding a video signal, The steps include obtaining the predicted block for the current block, The steps include generating a residual block of the current block based on the predicted block, The steps include performing a linear transformation on the residual block and obtaining the transformed block, The step includes performing a secondary transformation on the block that has been primary transformed, The step of performing the aforementioned quadratic transformation is: A step of determining which quadratic transformation matrix within the quadratic transformation set is applied to the linearly transformed block, The steps include applying the determined quadratic transformation matrix to the linearly transformed block to obtain a quadratic transformed block, The step of generating a first syntax element that identifies the determined quadratic transformation matrix within the set of quadratic transformations, A method in which, based on the width and height of the current block being 4, the output length of the quadratic transformation is determined to be 8, and based on the width and height of the current block being 8, the output length of the quadratic transformation is determined to be 8.

7. The method according to claim 6, wherein the quadratic transformation set includes two quadratic transformation matrices.

8. The steps include generating a second syntax element that indicates a linear transformation matrix to be applied to obtain the linearly transformed block, The method according to claim 6, further comprising the step of determining whether to apply the quadratic transformation to the linearly transformed block based on the second syntax element.

9. Based on the second syntax element which includes information about a predetermined specific transformation type, it is determined that the secondary transformation is applied to the primary transformed block. The method according to claim 8, wherein the predetermined specific conversion type is defined as DCT2.

10. In a method for transmitting data including a bitstream related to video, The steps include: acquiring the bitstream relating to the video; The step of transmitting the data, which includes the bitstream, The aforementioned bitstream is The steps include obtaining the predicted block for the current block, The steps include generating a residual block of the current block based on the predicted block, The steps include performing a linear transformation on the residual block and obtaining the transformed block, The block that has been linearly transformed is generated by performing a quadratic transformation on the block that has been linearly transformed, The step of performing the aforementioned quadratic transformation is: A step of determining which quadratic transformation matrix within the quadratic transformation set is applied to the linearly transformed block, The steps include applying the determined quadratic transformation matrix to the linearly transformed block to obtain a quadratic transformed block, The step of generating a first syntax element that identifies the determined quadratic transformation matrix within the set of quadratic transformations, A method in which, based on the width and height of the current block being 4, the output length of the quadratic transformation is determined to be 8, and based on the width and height of the current block being 8, the output length of the quadratic transformation is determined to be 8.

Images